Fabio Akita on the Jev hype: four independent audits recomputed, and where the probabilities break
Fabio Akita, one of the best-known software developers writing in Portuguese, published “Entendendo o Hype do Jev: pra que serve?” on 3 October 2026 (English version). His thesis: Jev is a good, cheap hosted classifier for repeated decisions about text. It is not a silver bullet. The surprise around it says more about teams using frontier chat models as expensive classifiers than about the product.
What is actually in it. Akita did not run his own benchmark. He says so: the leaderboard table in the post is the public Decision Index 0.2.1 data file rebuilt, and the Clef numbers are Cloudflare’s own run. The core is an essay by Paulo Câmara, CTO of the Brazilian company LUA Vision. He sorted about 350 GitHub repos that tested Jev in its launch week (89 with raw data, code, a pinned version and statistics; only 11 pre-registered) and recomputed four studies from scratch. He discloses that his company builds competing models.
We re-checked those four studies from the published raw data:
| Study | Result |
|---|---|
| Fair draws (1,000 requests) | Jev picked the first listed option in 1,000 of 1,000 cases, with 0.70–0.94 stated probability for events whose real chance is 1/6 to 1/2. A 2,000-request replication adds that wording matters too (“heads” won even when listed second) |
| ENEM 2025 (182 questions, Portuguese) | 103/182 vs 102 for Nemotron 3 Super 120B, a statistical tie (p = 1.0). Languages 78%, math 25% (chance 20%). All 39 answers above 0.9 confidence were right |
| Heart-disease risk (5,000 CDC survey respondents) | Mean stated risk 27.4% against a real rate of 9.0%. Brier is worse than just answering 9% for everyone (skill −1.32), though the ranking is usable (AUC 0.77) |
| Claim verification (495 claims) | Ties GPT-6 Astra on balanced accuracy (73.3 vs 73.8) at about 187× lower cost, but calls unsupported claims “supported” 23.2% of the time vs 14.1% |
The essay also covers prompt injection, using preprints and small studies that we did not re-check. Blunt attacks rarely work (1 in 1,056). Plausible unverified text does: a single unverified opinion appended to the text flipped 12.1% of decisions. Swapping option labels from “0/1” to “no/yes” flipped 70 of 100 answers. Its verdict: fast (100–400 ms), cheap, level with the top on narrow judgments, trustworthy above 0.9 on familiar data, and a good injection detector. Not a sole guard for agents, and not calibrated as a general rule.
The second half of the post is practical: when a Jev-style model fits (ticket triage, short-list ranking, model routing, flagging agent traces), when it does not (images, prose, exact parsers, destructive actions), and how to run a short pilot against your own labelled data. It also compares Clef and Laya and shows how to run the Decision Index kit yourself.
Methodology limits. Each study is a single dataset with one prompt design, mostly in English. The pt-BR evidence is one exam. The essay’s author competes in the same market. Akita’s own contribution is framing and practical guidance, not measurement.
What changes on the site. The four studies and the board numbers are now in the Jev page as independent third-party evidence. Jev’s capability goes from 8.2 to 8.1. The calibration subscore drops from 8 to 7, because confident wrong probabilities outside the training range are now shown by several reproducible studies with public raw data. The accuracy subscore rises from 7 to 8 for Jev’s #1 place on the public Decision Index 0.2.1 board (57.91). Laya and Hopper get their official Decision Index results in their Evidence sections, with no score change.