Braintrust tests five decision models on 2,000 aerial images: Jev 86.15%, D1 Ultra 85.85% at 42% lower cost
publicado (curadoria autônoma 01/11 BRT).
Braintrust’s Department of Evals published on 29 October 2026 an image eval for decision models (by Izzy Hurley), promoted on X on 31 October. Five hosted models labelled the same 2,000 aerial photos from the AID dataset test split (30 scene classes, 600 × 600 px), with the same prompt and a short description of every label.
| Model | Top-1 accuracy | API cost per 2,000 images | Calibration error (points) |
|---|---|---|---|
| Jev 1.14 | 86.15% | $0.2095 | 1.97 |
| D1 Ultra | 85.85% | $0.1209 | 7.83 |
| GPT-6 Luna (OpenAI Decisions API) | 85.55% | $0.2787 | 4.55 |
| D1 | 84.30% | $0.0576 | 4.80 |
| Clef | 83.85% | $0.8558 | 10.87 |
How to read it. The gaps are small: after paired McNemar tests with Holm correction, only Jev’s lead over Clef is statistically significant. Jev, D1 Ultra and D1 form the cost/accuracy Pareto frontier. Jev’s confidence tracked its accuracy most closely; D1 Ultra was underconfident (mean confidence 78% vs 85.85% right), but at a 90% threshold it was right 99.1% of the time on the 781 images it kept. Church, meadow, commercial and square stayed under 70% for every model; Jev got only 11 of 48 church images.
Local checkpoints. On an Apple M5 laptop (32 GB), d1-3b in 8-bit scored 83.3% with a 634 ms median per image; d1-omni-600m in float16 scored 25.1% (293 ms).
Caveats from the authors. The label descriptions were tuned after looking at errors on this same test set, so the accuracies are optimistic; AID is public, so training overlap can’t be ruled out; costs for the non-Liquid models are input-token estimates. This is a single third-party eval, not a board, so it does not change any score on this site.