JevBench v1.6: a full re-measure, separate open-weights and API boards, and new leaders
Benchmark Heaven published JevBench v1.6.0 and v1.6.1 on 5 October 2026 and kept adding rows through board revision v1.7.13 (7 October). We read the live boards on 7 October; every number below comes from them.
What changed
- Full re-measure. Every system with a reproducible recipe was re-run overnight (4–5 Oct) on a new rotating draw: 1,200 sealed + 300 public items, 1,500 decisions per system. The board says v1.5 scores “are not comparable with v1.6-measured rows”. Systems not yet re-run keep a dated v1.5 score in a separate table.
- Two boards (v1.7.0, 6 Oct). /jev-models ranks open-weights systems that Benchmark Heaven runs on its own rented hardware. Jev 1.13.0 is shown there as an unranked reference (it defines the cost and latency caps). Hosted APIs moved to a new API leaderboard. Scores and method are the same on both boards.
- Headline = Capability. On the open-weights board the headline is Capability (the mean of Intelligence and Calibration) among “Jev-class” systems: at most 2× Jev’s cost and median latency. The official Composite is the harmonic mean of four axes (Intelligence, Calibration, Speed, Cost), with a soft gate below Intelligence 50.
Open-weights board (v1.6.1 scale, 7 Oct)
| # | System | Composite | Capability (rank) |
|---|---|---|---|
| ref | Jev 1.13.0 (reference, not ranked) | 71.5 | 77.1 |
| 1 | Quyet-1.0-Large | 71.4 | 81.7 (#1) |
| 2 | torchcast-decision-12b | 69.9 | 71.7 (#5) |
| 3 | Winnow-12B Q8 | 68.9 | 71.2 (#7) |
| 4 | deck-31B (frozen Gemma-4-31B, not a trained model) | 68.7 | 77.6 (#2) |
| 5 | Cygnet (frozen Gemma-4-12B, not a trained model) | 68.6 | 70.9 (#8) |
| 6 | Jev-Omni | 67.7 | 71.3 (#6) |
| 7 | Xor 26B-A4B (Juspay) | 67.4 | 72.4 (#4) |
| 9 | Surogate Rune 26B-A4B v3 | 65.0 | 73.5 (#3) |
| 10 | GEV-26B-Decide | 64.2 | 70.1 (#10) |
102 open-weights systems are ranked. Large 27B rows such as pplx-decider v1.1 (Intelligence 75.7, the highest open-weights value) land low on Composite because self-hosting them costs 5–7× Jev, which puts them outside the Jev-class caps.
API board (7 Oct)
| # | Offering | Composite | Capability |
|---|---|---|---|
| 1 | Sage 1.3.0 (Levanto Labs) | 74.0 | 78.6 |
| 2 | d1 (Liquid AI) | 73.0 | 74.4 |
| 3 | Jev 1.13.0 | 71.5 | 77.1 |
| 4 | wity-1 (reasoning auto) | 70.8 | 79.1 (outside Jev-class on latency) |
| 5 | OpenAI Decisions (gpt-6-luna) | 62.5 | 73.5 |
OpenAI Decisions answered a fresh sealed set (A5) plus the public items and is equated to the v1.6.1 scale. Its 95% interval is wide (51.6–64.6) and it is statistically tied with Instinct (62.2).
How we use it
We cite v1.6.1 numbers with the board edition and never next to v1.5 numbers. Pages that quoted v1.5.7 ranks now show both editions, labelled. A board result counts as third-party evidence; vendor numbers stay UnverifiedClaim.
Also on 6 October, the Hugging Face Decision Index 0.3 went live as a separate board with its own method. We cover it on the model pages where it applies.