Paper: decision models abstain more than LLMs when the right legal authority is missing

Source · Model card: pplx-decider v1.2 (27B + Flash 4B)

“Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority” (M. Mikail Demir, University at Albany, SUNY; October 2026) tests what models do when the correct legal holding is not among the options.

Setup. CaseHOLD’s 5,314 test items, plus an abstain condition: in 1,670 items the true holding is swapped for a similar one from another case and a sixth option, “none of these”, becomes correct. Four decision models (Jev 1.14, pplx-decider v1.2, d1-ultra and GPT-6 Luna Decisions, all through OpenRouter’s decisions interface) against six generative LLMs. Code, data and every model output are public.

Findings (author-run):

• On plain CaseHOLD, pplx-decider reaches 88.1%, level with Gemini 3.8 Flash (88.2%), at about 1/55 of the cost.
• When no option is correct, generative LLMs still pick a holding in 38–86% of items. Jev does it least (30.1%) but also abstains on 26.2% of items that do have a correct holding; pplx-decider rarely over-abstains (1.8%) but picks wrong more often (56.5%).
• Accepting Gemini’s answer only when pplx-decider agrees halves its wrong citations (18.4% → 9.4%), at the price of more items sent to review.
• Qwen3-8B fine-tuned only on Jev’s probabilities, with no labels, matches Jev’s accuracy (63.8% vs 63.9%) with a calibration error of 0.025.

One task, US case law, one prompt per model type; the numbers are the author’s, not a board.