Clavia: Clairic extraction plus a Jev hide/keep decision beats OpenAI's Privacy Filter on legal PII
publicado (curadoria autônoma 01/11 BRT).
Arjun Kumar of Clavia posted on 31 October 2026 a small, reproducible comparison for detecting personal data before it leaks into RL environments. The code is public in clavia-labs/private-data-detection.
Setup. The test split of the Text Anonymization Benchmark (TAB): 127 European Court of Human Rights judgments with annotator marks. Three pipelines: OpenAI’s open-weight Privacy Filter (a token classifier, hosted on Baseten), Cloudflare’s Clairic extraction model alone, and Clairic followed by a Jev choice question per extracted span (“hide” or “keep”, with about 160 characters of context and the name of the person to protect).
| Pipeline | Precision | Recall, applicant names | Recall, case IDs | Cost per 1,000 judgments |
|---|---|---|---|---|
| OpenAI Privacy Filter | 93% | 77% | 2.2% | $0.26 |
| Clairic | 78% | 96% | 96% | $0.17 |
| Clairic + Jev | 91% | 96% | 96% | $0.86 |
Why it matters here. It is a clean example of the extract-then-decide pattern: the extraction model over-flags, and the decision model filters with a probability per span (in the example, case number p(hide) 0.93, the applicant 0.91, “Istanbul” 0.22). Precision recovers almost to the specialised model’s level while keeping Clairic’s recall.
Limits. One author, one legal-domain benchmark, a single run; Jev’s cost was measured on one judgment and scaled. The author’s claim that this generalises across domains is a hunch, not a result. No score on this site changes.