ENPT

Stable is not correct: an independent test of 27 models finds pplx-decider v1.1 repeats the same wrong answer 728 times

Last updated:

Source · Model card: pplx-decider v1.1 27B

ZekiWorks (thread, 10 Oct) tested 27 models in 33 configurations on 2,198 reading and grammar questions from one exam-prep book in a language other than English. Results, code and recorded answers are public.

  • Stability: five runs of every question. Jev changed 314 answers, Liquid d1 217; pplx-decider v1.1 (“Decider v1.1”) and Open-Jev changed none, but Decider v1.1 gave the same wrong answer to 728 questions every time.
  • Confidence: at ≥0.99, Decider v1.1 passed 349 answers with 0 observed errors, GLiDE 121 with 0, Jev 312 with 1, OpenAI’s Decisions API (Luna) 368 with 3. Luna’s self-reported confidence passed 830 with 137 errors.
  • Accuracy: decision models 66.9% (Decider v1.1) and 67.9% (GLiDE), no detectable difference; GPT-6 Astra at low reasoning 97.6%. Decision models scored 83–89% on reading but 46–59% on grammar.

One book, one language, mostly grammar, so this does not generalise to other tasks. It is a useful reminder: a stable answer and a confident answer can still be wrong. No catalog scores change.