Jeeves
Quadrant scores
See the full quadrantScored with the public quadrant rubric (maturity × capability; bubble = adoption). Revise as evidence lands.
- Maturity6.1/10
9B weights (bf16 + FP8) with full training code and train/dev/test data; own /v1/systemone server and a drop-in replacement for the TypeSafe Python SDK; CUDA and Apple Silicon (MPS) kernels. Community MLX port and an Ollaya package. PostHog research repo, no SLA. License: MIT on GitHub, Apache-2.0 on HF.
- Capability6.8/10
Noul / Choice / Score from a pointer head after a reasoning chain; one temperature fitted on dev, JevBench public ECE 0.037 (author). Thinking is on by default: median 3.3 s / p90 17.1 s on an H100 (FP8); about 0.3 s without thinking. JevBench public 0.935 vs Jev 0.866, self-run, off the board.
- Adoption14/100
About 400 GitHub stars and 20 forks in four days, external PRs merged, 201 Hugging Face downloads for the main weights plus FP8 and community MLX copies; aggregator coverage, no official PostHog launch post found. No production use reported.
Vendor claims
- JevBench public tiers (231 items): 0.935 vs Jev 0.866 and Kev-8B 0.715; hard tier 0.865 vs Jev 0.730; ECE 0.037 vs Jev 0.049 (Jev and Kev numbers restricted to the same public items; sealed judge tier excluded)[Vendor claim — not independently verified]
- Held-out / out-of-domain test split: 0.889 vs Jev 0.857 and Kev-9B 0.822 (Jev and Kev columns are the numbers Kev publishes, on different items from the same sources)[Vendor claim — not independently verified]
- On 325 dev questions, one H100 with FP8: full thinking 0.825 accuracy, median 3.3 s / p90 17.1 s; no thinking 0.775 at about 0.3 s[Vendor claim — not independently verified]
- Block-4 diffusion drafter: 176 chain tokens/s for one question (1.6× plain greedy), about 960 tokens/s total with eight questions batched[Vendor claim — not independently verified]
News covering this model
Read this first
- It thinks by default, so it is not fast by default. Full thinking runs at a 3.3 s median and 17.1 s p90 on an H100. Turn thinking off (
think: false, about 0.3 s) or cap it (max_think,nothink_threshold) when latency matters.- Every number is PostHog’s own run. The JevBench figures are on the 231 public items with PostHog’s script, not on the board; the Jev and Kev columns outside JevBench come from Kev’s published tables on different items.
- Two licenses on two pages. The GitHub repo says MIT; the Hugging Face weights are tagged Apache-2.0. Both are permissive, but the mismatch is unresolved.
Jeeves is an open decision model from PostHog, released on GitHub and Hugging Face on 29 September 2026. Its author is Nicholas P. Waltz (PostHog AI research); most of the inference engineering after release is by Robbie Coomber. The README credits Kev as the inspiration.
What sets it apart: Jeeves writes a reasoning chain for each question before it decides. After the chain closes, the question and its options are repeated and a pointer head scores each option. Most Jev-like models answer straight from the prompt.
Specs
| Attribute | Value |
|---|---|
| Base | Qwen/Qwen3.5-9B |
| Decision part | LoRA r16 on all projections + pointer head (scaled dot product between a <decide> query and each option’s </opt> key); one temperature fitted on dev |
| Training | SFT (2 epochs, 19,126 questions from 12 public datasets + synthetic policy data) → CISPO RL (stopped at step 402 of 624, 9,992 questions, 8 rollouts each) → calibration |
| Released weights | Fused 9B (bf16, 21 GB) and FP8 (11.5 GB); block-4 and block-8 diffusion drafters |
| Question types | Noul, Choice, Score in one request |
| Serving | python -m inference.serve → POST /v1/systemone; options think, max_think, nothink_threshold, return_reasoning |
| SDK | jeeves_sdk, a drop-in replacement for typesafe-sdk |
| Hardware | CUDA (bf16, FP8 on compute capability ≥ 8.9); Apple Silicon (MPS) with 48 GB or more |
| License | MIT (GitHub) / Apache-2.0 (Hugging Face tag) |
| Launch |
Evidence (UnverifiedClaim)
From the README, accuracy with thinking, greedy, 2,560-token cap:
| Benchmark | Kev-9B | Jev | Jeeves |
|---|---|---|---|
| Test overall (out-of-domain + held-out) | 0.822 | 0.857 | 0.889 |
| Transfer overall (MMLU-Pro, buried state) | 0.579 | 0.800 | 0.746 |
| JevBench overall (231 public items) | 0.715* | 0.866 | 0.935 |
| JevBench hard (111 public items) | 0.451* | 0.730 | 0.865 |
| MMLU / MMLU-Pro | 0.738 / 0.515 | 0.900 / 0.840 | 0.793 / 0.739 |
| JevBench ECE (public) | — | 0.049 | 0.037 |
* Kev-8B (Qwen3); no Kev-9B JevBench result is published.
Without thinking the same checkpoint scores 0.804 on PostHog’s test split, against 0.840 with it. On 325 dev questions (H100, FP8): full thinking 0.825 at 3.3 s median / 17.1 s p90; max_think 768 + nothink_threshold 0.9 gives 0.806 at 2.0 s / 5.6 s; no thinking 0.775 at about 0.3 s.
Fit / anti-fit
Fit: hard decisions where a reasoning fallback would otherwise be a full LLM call (multi-step policies, contrastive rules); teams that want to read the reasoning (return_reasoning); research on RL for decision heads, with code and data to reproduce.
Anti-fit: hot-path routing with tight latency budgets unless thinking is off; knowledge-heavy questions (trails Jev on MMLU / MMLU-Pro); small GPUs (21 GB of weights in bf16, plus caches).
Limits
- The reasoning chains have no language-consistency reward and are hard to read, per the README.
- Thinking tail latency is long (17 s p90 with full chains).
- Not on the JevBench or Decision Index boards.
Why it is in the catalog
Jeeves trains its own decision weights (a LoRA plus a pointer head, then RL with CISPO), publishes them with a typed API that returns a probability per option, and serves the System One request shape. It is listed as lora-adapter because the decision part is a LoRA and a small head on a stock Qwen3.5-9B, even though the release ships the fused weights. It is not a wrapper: the reasoning chain comes from the fine-tuned model, and the decision is read by a trained head, not by parsing text.