LangChain tests Jev-as-a-Judge for agent evals on LangSmith
On 20 September 2026, LangChain published Can Jev Be a Better Agent Evaluator? (Daniel Shea & Seán Roche) — a separate piece from the earlier Building a Harness with Jev / langchain-typesafe middleware post already covered as langchain-harness-jev.
The question: can a System One model sit beside code-based evals and LLM-as-judge as a third path? The authors score fixed Deep Agents weather-agent traces in LangSmith with two signals (quality continuous score; does_pass binary) and compare Jev (langchain-typesafe==0.0.1a2) to GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 across 100 repeats per case, with a human oracle for agreement.
UnverifiedClaim / hygiene (LangChain experiment, not ModelSystem.One): Jev matched the oracle on all 500 binary repeats; mean per-case quality variance 0.0000149 (authors: Luna ~433×, Terra ~913×, Claude ~92× higher); mean latency ~0.44s and ~$0.00035/call in that run. The post itself calls the test narrow and early — low variance ≠ automatic correctness at scale.
Integration surface is the same TypeSafeClassifier / Choice·Score·Noul questions already used in the harness blog. Repro notes and a GitHub repo are linked from the LangChain post.
Primary source: langchain.com/blog/jev-agent-evals-langsmith.