Red Hat benchmarks decision models against classic guardrails: no clear win, but closer than the headline
Red Hat’s AI Safety team (Rob Geada, Mac Misiura, Shelton Cyril) published a benchmark on 2 October 2026 that puts Jev-style decision models next to the guardrails Red Hat already ships. Nine setups, two class-balanced English tasks (prompt injection; content safety / toxicity), all run through NeMo Guardrails with EvalHub: small pre-trained classifiers (the OpenShift AI 3.6 defaults), BART-large-mnli, three LLM judges (Shieldstral 3B, Nemotron-3.5 Content Safety 4B, Qwen3.6-35B), and three decision models (Laya, DiffusionGemma on vLLM’s /v1/systemone, and Jev 1.13 through TypeSafe’s API).
What the tables show.
| Setup | Prompt injection | Content safety |
|---|---|---|
| Pre-trained classifier (DeBERTa 200M / Granite Guardian HAP 125M) | 89.01%, 54 ms | 80.27%, 33 ms |
| Qwen3.6-35B (LLM judge) | 89.31%, 313 ms | 85.47%, 308 ms |
| Nemotron-3.5, custom policy | 84.84%, 240 ms | 85.07%, 242 ms |
| Jev 1.13 | 86.35%, 348 ms | 86.20%, 360 ms |
| DiffusionGemma (vLLM) | 87.72%, 562 ms | 85.53%, 499 ms |
| Laya | 85.44%, 119 ms | 57.87%, 118 ms (75.20% with a tuned policy) |
Accuracy and median round-trip latency, from the article.
Red Hat’s conclusion. Where labeled data exists, a small trained classifier is hard to beat on cost and latency: on prompt injection the 200M DeBERTa ends 0.30 points behind the 35B judge, at a sixth of the latency. Zero-shot classifiers are “a capable alternative to LLM-as-a-judge” when no trained classifier exists, but the authors “did not find that decision models produced faster, cheaper, or higher-quality answers” than LLM judges, and they single out Laya’s size as the exception.
Reading it fairly. The headline is stronger than the data. Jev ranks first on content safety and fourth on prompt injection, 3 points off the top; Jev and DiffusionGemma both beat the 125M content-safety classifier by 5 points or more, which the authors call “a promising signal”. The latency comparison is uneven by the article’s own account: Jev was called from the UK to US servers (at least ~56 ms of network), while the small classifiers ran locally on an M1. The risk policies were adapted from LLM-judge prompts, and the Laya re-prompt shows how much the wording moves decision models (+17.8 points for Laya, −3.7 for Jev on the same tuned policy). Two tasks, English only. The tables have small inconsistencies: the Shieldstral rows in tables 1 and 2 look swapped relative to the appendix; the DeBERTa and BART medians differ between the main tables and the appendix (54.1 vs 80.4 ms; 144.2 vs 180.1 ms); the text says “0.20 points” where the table gives 0.30, and “eight” guardrails where there are nine.
What changes here. Nothing in the scores. It is a useful independent data point for the Jev page: 86% on two guardrail tasks, with a 348–360 ms median round trip from Europe.