Paper: decision fine-tuning does not make a better reranker (Kev, Clef vs reranker fine-tunes)
publicado (OK do Fabricio 26/10 BRT).
“System One Models Are Not Better Rerankers” (Michael Dinzinger, Kanishka Ghosh Dastidar and Michael Granitzer; University of Passau and IT:U Linz; arXiv, submitted 23 Oct 2026) runs the comparison earlier Jev-as-reranker reports skipped: decision fine-tuning against plain reranker fine-tuning on the same backbone.
Setup: six Qwen backbones from 0.6B to 27B, each as the untuned model, a decision fine-tune (Kev on every size, Clef-Flash at 9B, Clef at 27B) and a reranker fine-tune (Qwen3-Reranker, Prism, and two the authors trained on MS MARCO). All rerank the same BM25 top-100 on TREC DL 2019–2023 and five BEIR collections.
What they report:
- Ranking: from 0.6B to 9B, the reranker fine-tune wins on every backbone by 5 to 29 points nDCG@10, also on the zero-shot collections. At 27B the two tie (Kev-27B 60.1, the authors’ reranker 59.6).
- Up to 9B, decision fine-tuning adds nothing over asking the untuned model the same question (−2.5 to +0.7 points).
- Calibration depends on the wording. Asked
Does document {doc} answer the query?, decision models from 4B up are better calibrated than published reranker fine-tunes; Jev (ECE 0.075) and Kev-27B (0.076) are the best calibrated systems in the study. With the looser S1Rank question, Jev’s ECE goes to 0.224 and the advantage disappears. - Context: Jev in joint mode averages 59.1 nDCG@10, level with 8–9B reranker fine-tunes. Laya (43.5) does not improve on BM25 (43.2).
The authors’ conclusion: “If you need a reranker, fine-tune a reranker.” Their limits: they compare released models (not RLCD, which is unpublished), train their own rerankers once with one recipe, and measure calibration only on TREC DL. Code and raw outputs are on GitHub. The thread that surfaced it: @kalyan_kpl (26 Oct).