Hamel Husain and Antaripa Saha test six decision models as eval judges: tune the threshold, then check calibration
publicado (OK do Fabricio 20/10 BRT).
Hamel Husain and Antaripa Saha published “How To Use Decision Models for Evals” on 20 October 2026. It is an independent, small test of decision models as cheap judges for automated evals.
Setup. 100 human-labelled traces from an apartment-leasing agent; the task is to flag traces with an error, such as a failed booking or a missed handoff. They tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash and OpenAI Decisions (GPT-6 Luna), each with the same criteria.
Findings (their numbers, one dataset).
- Thresholds: many users keep the 0.50 default, which “can leave a lot of performance on the table”. With both precision and recall required to reach 80%, only Perplexity, d1-ultra and OpenAI Decisions qualified.
- Cost against quality: Perplexity was the cheapest qualifying option at 0.82 cents per 100 runs (F1 0.79); OpenAI Decisions reached F1 0.87 at 3.72 cents, about 4.5× the cost. OpenAI separated the classes best on this task.
- Calibration: Jev and d1-ultra were already well calibrated; Platt scaling moved Perplexity, d1 and Clef Flash closer to the diagonal (one Clef Flash group predicted about 8% failure where 60% failed) and made OpenAI’s probabilities slightly worse.
- Latency: mean response times from separate runs ranged from 1.56 s (d1-ultra) to 7.85 s (Perplexity); the authors say these do not establish comparative throughput.
How to read it. One broad task on 100 traces; the authors say results vary across tasks and recommend a separate judge per failure mode. It does not change any score. The practical point, tune the threshold on your own labels and check calibration before trusting a probability, applies to every model in the catalog.