Two independent tests put Liquid's d1-ultra first: Turkish reading and grammar, and finding Waldo
publicado (OK do Fabricio 19/10 BRT).
Two third-party tests published on 18 Oct put Liquid AI’s d1-ultra ahead of the other decision models they tried. Neither is a public board we score from, and both report their own runs. We relay the numbers as published.
Zeki Works: 2,198 Turkish reading and grammar questions
Zeki Works (results updated 17 Oct, answer key version 2) answered 2,198 native Turkish multiple-choice questions, not translations, with 24 decision models and several chat models for comparison. Every answer is public on the page.
- Accuracy: d1-ultra 77.1%, clearly first among decision models; GLiDE and Perplexity’s Decider v1.1 and v1.2 near 67%; Jev 1.14 59.2%.
- Calibration gap: smallest for GPT-6 Luna through OpenAI’s Decisions API, with Jev 1.14 close behind.
- Estimated saving: d1-ultra had the largest estimated saving when put in front of each of the 10 LLM setups Zeki Works tried.
- Reading vs grammar: decision models got up to 92.1% of reading questions right but at most 70.7% of grammar questions. The best chat setups, such as GPT-6 Astra with low reasoning, scored around 97–98%.
Zeki Works’ earlier test (11 Oct) looked at answer stability across 27 models.
WaldoBench: 66 Where’s Waldo scenes
WaldoBench (Andy Gao, last updated 18 Oct) gave image-input decision models 66 scenes from the IVSN Waldo set, split into tiles, plus a picture of Waldo. A search counts as found when the chosen tile touches Waldo.
| Model | Found | p50 per scene | Cost per scene |
|---|---|---|---|
| d1-ultra | 53 / 66 (80.3%) | 3.36 s | $0.0052 |
| Liquid d1 | 49 / 66 (74.2%) | 4.52 s | $0.0025 |
| Jev 1.14 (yes/no, batched) | 48 / 66 (72.7%) | 1.54 s | $0.0051 |
| Decider Flash v1.2 | 47 / 66 (71.2%) | 2.61 s | $0.0010 |
| Span-02, Mercury Decide, Decider v1.2 | 47 / 66 (71.2%) | 3.3–4.8 s | $0.0016–0.0021 |
| Clef / Clef Flash / Clef Omni | 46 / 45 / 44 | 6.0–13.6 s | $0.0043–0.0274 |
| GPT-6 Luna Decisions | 35 / 66 (53.0%) | 3.13 s | $0.0121 |
With 66 scenes, the 95% interval on d1-ultra’s found rate runs from about 71% to 89%, so most of the middle of the table overlaps.
Why it matters. These are small, narrow tests, but they are independent, and they agree with Liquid’s claim that d1-ultra is stronger than Jev 1.14 on harder questions. They also show the pattern on the public boards: Jev is rarely the most accurate, and often the fastest. No score changes; d1-ultra’s accuracy stays tied to JevBench and Artificial Analysis.