Sample report

What you get for every task.

A sample from our first batch: real rollout and exploit-attempt results from our own demo shop, and the broken tasks we planted to test the checks. No task has shipped yet.

Book a call Request a sample set

First batch

One task, the planted controls, the whole batch.

The card is one task as a buyer would get it. The controls are the broken tasks we planted to test our own checks.

Sample First batch · 6 Oct 2026

Price-window pick plus two filters

nw-mid-price-plus-filters · PL · demo shop

W sklepie Northwind Supply Co. kup jeden produkt, który kosztuje więcej niż…

Pass rate: 0/6 95% interval 0–39%
Difficulty verdict: Unsolved by the agent. An expert's blind solve decides.
Exploit attempts: 12 tried · 0 got through
Expert review: Scheduled
Task hash: 517091…ab2313

Controls First batch · 6 Oct 2026

3 broken tasks planted, 3 caught

We planted tasks with known flaws in the batch to test the checks.

Loose grader: Its check ignored the filters, extra items, extra orders, the card and the address.Caught by the grader attacks. 6 of 10 scripted exploit attempts got credit, one of them an order without the filters.
Wrong reference answer: The stored answer named a cheaper product than the one the prompt asks for.Caught by a blind solve. A model solving from the prompt alone, without the answer, picked a different product, and a model review of the grader then flagged the task as broken.
Ambiguous prompt: “A few” filters and “something nice” have no single right answer.Caught by a blind solve. The model solving it blind flagged the prompt as ambiguous, and the task was marked suspect.

How each planted task was caught

Batch summary

23 tasks, 6 rollouts each

4 Too easy: dropped
6 In range: sent to expert review
13 Unsolved by the agent: an expert's blind solve decides

The in-range tasks, with their rollout and exploit-attempt results
Task	Pass rate	Exploit attempts	Expert review
nw-account-cart-edit-grinder-to-beans	4/6	13 tried · 0 through	Scheduled
nw-account-kettle-filters-budget-edit	3/6	12 tried · 0 through	Scheduled
nw-seed-best-under-budget	3/6	8 tried · 0 through	Scheduled
nw-seed-budget-bundle	3/6	12 tried · 0 through	Scheduled
nw-account-beans-and-filters-derived	2/6	13 tried · 0 through	Scheduled
nw-multistep-minimize-beans-filters	1/6	13 tried · 0 through	Scheduled

Run on 6 October 2026 against Northwind Supply Co., our own demo shop: 20 tasks generated by a model, 3 written by us. The exploit attempts ran on the same tasks on 8 October 2026.

Each task carries a provenance manifest (its checks, who solved and reviewed it, and the hashes of the task and of each recording).

Trust

Independent: we don't train models, so our reports have no stake in the result.

Applied Will is a Buena AI company. Buena AI, Inc. holds a SOC 2 Type II report (security, confidentiality, availability) with no exceptions, available under NDA. Task content never goes on-chain — only hashes. Experts solve blind, without the answer or each other's work, and accept a recording consent before their work is recorded.

Founded by Angel Padilla: ex-Scale AI, and a concrete estimator who has hand-labeled 74,750 sq ft of takeoffs.

SOC 2 Type II · Buena AI, Inc.

SecurityConfidentialityAvailability

Visit the trust center