Research · Benchmark
ConcreteTakeoff-Bench v0.3: footing counts on real drawings.
Private benchmark · small sample
A private benchmark on real construction drawings from public bid sets, with the title blocks removed. We don't publish the drawings.
These results are the development split: n = 19 footing counts, a small sample. We developed the prompts on this split, so read these scores as upper bounds. The test split is held out and unpublished.
In short
ConcreteTakeoff-Bench v0.3 is a private benchmark on real drawings, title blocks removed. It scores footing counts per mark, exactly, against a concrete estimator's reviewed counts. On the development split (n = 19, a small sample), five systems read the same sheets three times each.
Claude Haiku 5.5 and Sonnet 5.5 scored 100%, at $0.020 and $0.16 per sheet; the prompts were developed on this split, so read these as upper bounds. Two text-only readers, our OCR baseline and TypeSafe's Jev, reached 79% for a fraction of a cent.
A labeling-tool glitch had mis-entered one count in our answer key. Every system but Qwen flagged it on every run, and expert review confirmed 8.
The open Qwen3-VL-8B is at 61%, 39 points behind: the gap fine-tuning on our data targets.
The results
Footing counts per mark, exactly right
Development split, n = 19. Mean of three runs; cost per sheet under each bar.
| System | Footing counts right | Cost per sheet |
|---|---|---|
| Claude Haiku 5.5 | 100% | $0.020 per sheet |
| Claude Sonnet 5.5 | 100% | $0.16 per sheet |
| Text/OCR | 79% | $0.00033 per sheet |
| Jev (TypeSafe) | 79% | $0.00056 per sheet: $0.00033 text/OCR + $0.00023 Jev |
| Qwen3-VL-8B | 61% | $0.014 per sheet |
Source: our private write-up of the run of 11 October 2026; its SHA-256 is under The data.
Claude Haiku 5.5, Claude Sonnet 5.5 and Qwen3-VL-8B read the sheet images. The two text-only readers cost a small fraction of that: the text/OCR baseline reads the sheet's text layer, with OCR on its raster parts, and Jev, TypeSafe's model, sorts each footing mark it finds in that text (a callout on the plan, a note, a detail tag or a schedule row) so code can count the callouts.
All results
| System | Counts right | 95% interval | Cost per sheet |
|---|---|---|---|
| Claude Haiku 5.5 | 100% | 83–100% | $0.020 |
| Claude Sonnet 5.5 | 100% | 83–100% | $0.16 |
| Text/OCR | 79% | 57–91% | $0.00033 |
| Jev (TypeSafe) | 79% | 57–91% | $0.00056* |
| Qwen3-VL-8B | 61% | 39–80% | $0.014 |
* Jev (TypeSafe): $0.00033 for our text/OCR step, which gives Jev the sheet's text, plus $0.00023 for the Jev calls. Run on 11 October 2026.
A correction to our answer key
One value in our answer key was wrong, from a labeling-tool bug. It was flagged by every system except Qwen3-VL-8B, on every run, and expert review confirmed 8. The scores on this page use the corrected key. The machines' answers never change the key; only the estimator's review does.
Method, briefly
- Input: every system gets the same de-identified crop of each sheet: the title block cut off, and lines that look like a client's name, an address or a contact blanked.
- The task: count each footing mark on the plan, not in the schedule. A count scores only if it is exact, against a concrete estimator's reviewed counts.
- Scoring: each run is scored on every item, and a system's score is the mean of its runs. The 95% interval is Wilson's, on the number of items.
What it doesn't show
- A small sample. One count moves a score by several points, and counts on the same sheet aren't independent, so the intervals are wide. Read this as a first measurement, not a ranking.
- Upper bounds. We developed the prompts on this split. The held-out test split isn't published.
- Footing counts only. No slab areas, lengths or volumes on this page.
- One harness. The prompts, the tiles and the text extraction are ours. A different setup could score differently.
The data
The drawings, the answer key and the per-item results stay private. This page is built from a public export of our write-up of the run: the split, the scores, their intervals and costs, the caveats and the correction. The write-up's SHA-256 is 8577123dacfcbddec6527a7d8bf8530a9bbb5aeb1206a5c5fdfb3577b935a88b.
v0.2 ran on synthetic sheets; its write-up stays up as it was.
Talk to us about a benchmark collaborationRequest construction takeoff environments