Before a RAG system writes a word, it decides: is there enough in the context to answer at all? Faithfulness metrics can't see half the ways that decision goes wrong — a refusal asserts nothing, so it's trivially “faithful,” even when refusing was the wrong call. AbstentionBench measures which tools can actually catch it.
Every case freezes a RAG trace — question, retrieved context, and the system's output — with gold labels for the decision. Hover a quadrant for a real case from the dataset.
The context contains the answer and the system gave it.
The answer was right there. The system said “I don't know.” Faithfulness metrics score this as fine.
Nothing in the context backs it — a confident, sourced-looking hallucination.
The context can't answer it, and the system said so.
Task per tool: given (question, context, answer), predict whether the answer/abstain decision was correct. The column that matters is catch rate on over-refusal.
“I can't answer that from the provided context.” — but the context contained the answer verbatim.
“Is every claim in the answer supported by the context?” A refusal makes no claims. There is nothing to check.
No unsupported claims found.
Eight archetypes, hand-authored, each kept only after unanimous three-reviewer adversarial label verification (2 of 32 drafts were rejected for label ambiguity — they're not in the dataset). One real example per archetype:
Clone the repo, point the harness at your eval tool, and see whether it can grade the decision — not just the words. PRs with new adapter results are welcome.