ENPT

JevAgentHarness: a decision model can take routine agent steps, but only behind a fitted confidence threshold

Last updated:

Source · Model card: Jev

publicado (OK do Fabricio 19/10 BRT).

“JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs” (Kunlun Zhu, Heng Ji and six co-authors from UIUC, Princeton and Google) went up on arXiv on 15 Oct. It asks a practical question: most calls an agent harness makes to its LLM end in a multiple-choice pick, such as which tool to call next. Can a decision model take those calls without hurting the agent?

The benchmark. AgentDecisionBench tests decision models from isolated questions up to control of a running agent, on public benchmarks and on DecisionGym, the authors’ customer-service environment.

What the authors report (their own runs, UnverifiedClaim):

  • The decision model with the best accuracy on isolated questions solved only 69.5% of live agent tasks.
  • Handing every tool choice to a decision model, with no confidence threshold, cut task success by 19.0 points.
  • JevAgentHarness fits a threshold for each kind of choice on the agent’s own logs. Above it the decision model answers; below it the LLM decides.
  • They trained AgentDecision-4B on the messy states an agent reaches after its own mistakes, using Jev’s answers as labels. On task types unlike its training data, agents solved 91.5% of tasks with it and 82.0% with Jev.
  • On tasks like the training data, agents using such 4B models were 81.1–89.0% cheaper and up to 2.5× faster, within 3 points of the full LLM’s success rate.
  • The repo’s proxy uses Jev 1.13 to label each coding-agent step as routine or hard and sends routine steps to a cheaper model. On 74 held-out Terminal-Bench 2.0 tasks, three unmodified harnesses stayed within 3 points of their strong-model success rate while spending 28–64% less and finishing 4–43% faster.

What is released. Code is on GitHub (MIT), the benchmark is on Hugging Face (CC BY 4.0), and there is a project page. The AgentDecision checkpoints (4B and 4B-v1 as LoRA adapters on Qwen3-4B, plus a 0.8B on StartLux-Decision-0.8B) are CC BY-NC 4.0 and are not on any public board, so they stay off the catalog for now.

Why it matters. It is the clearest evidence yet for a point vendors tend to skip: accuracy on single questions does not carry over to an agent that compounds its own errors, and the confidence score is what makes delegation safe. It sits next to the 13 Oct paper on Jev in agent workflows, which found Jev made agents cheaper and faster but not better.