Book a call

Task audits

Find out which of your RL tasks hold up.

Send tasks you already generated. We attack each grader, roll the tasks out against an agent and have experts solve them blind. You get a report with the evidence for every task. Today the automated checks run on web-shop tasks like our demo shop's; we'll tell you which apply to yours before you send them.

Book a 30-minute call See pricing

Free 10-task sample · pricing after the sample

The audit

What happens to each task.

Automated checks are triage: they tell experts where to look. A task is accepted when two blind solves and a sealed review agree, or when a senior expert settles a disagreement in writing.

  1. 01

    Automated

    Grader checks

    The grader must accept the reference answer and reject doing nothing. Probes test what else it lets through.

  2. 02

    Automated

    Rollouts

    A cheap agent attempts each task, so we know if it is too easy, in range or never solved.

  3. 03

    Automated

    Exploit attempts

    At least eight scripted attempts to earn reward without solving the task. One that gets credit rejects the task.

  4. 04

    Experts

    Two blind solves

    Two experts solve the task without the answer or each other's work, in a recorded browser.

  5. 05

    Experts

    Sealed review

    Two reviewers seal their verdicts before either is revealed. If the reviews disagree, or both accept but the solves don't match, a senior expert adjudicates.

What's in a report

Every task gets a verdict and a reason.

Per task

Pass rate
Share of cheap agent rollouts that solved the task.
Difficulty verdict
Too easy, in range or unsolved, read against the expert's blind solve.
Exploit attempts
What we tried against the grader, and what got through.
Accept or reject
The decision, with the reason.

Per batch

Acceptance rate
Share of the batch that was accepted.
Grader exploit rate
Share of graders we could game.
Cost per accepted task
What the batch cost, divided by the tasks that passed.
Delivered with
A cleaned task set in HUD or Harbor format, with a provenance manifest and the recording hashes for every accepted task.

Sample report

A task from our first batch.

Real results from our own demo shop, and the broken tasks we planted to test the checks. No task has shipped yet.

Sample First batch · 6 Oct 2026

Price-window pick plus two filters

nw-mid-price-plus-filters · PL · demo shop

W sklepie Northwind Supply Co. kup jeden produkt, który kosztuje więcej niż…

Pass rate
0/6 95% interval 0–39%
Difficulty verdict
Unsolved by the agent. An expert's blind solve decides.
Exploit attempts
12 tried · 0 got through
Expert review
Scheduled
Task hash
517091…ab2313
Controls First batch · 6 Oct 2026

3 broken tasks planted, 3 caught

We planted tasks with known flaws in the batch to test the checks.

Loose grader
Its check ignored the filters, extra items, extra orders, the card and the address.Caught by the grader attacks. 6 of 10 scripted exploit attempts got credit, one of them an order without the filters.
Wrong reference answer
The stored answer named a cheaper product than the one the prompt asks for.Caught by a blind solve. A model solving from the prompt alone, without the answer, picked a different product, and a model review of the grader then flagged the task as broken.
Ambiguous prompt
“A few” filters and “something nice” have no single right answer.Caught by a blind solve. The model solving it blind flagged the prompt as ambiguous, and the task was marked suspect.

See the whole batch

Pricing

Start with ten tasks, free.

Every task gets the same audit.

Looking for task sets?

From$200per task

Pilot sets of 25

Computer-use task sets, Polish first.

See task sets

Request

Send us ten tasks.

Book a 30-minute call. Or send us a note about your tasks, and we reply within one business day with how to send them.

Book a 30-minute call Or send a request