A 1.5B model that judges RAG groundedness as well as a frontier model — at $0 per call.

The base model is almost useless at this job. The fine-tune isn't.

Fine-tuned on 1,120 constructed cases (2 epochs, QLoRA on Qwen2.5-1.5B-Instruct, ~4.5 min on an RTX 5070 laptop). Judged on 140 held-out cases the model never saw. This before/after table is the release gate — evaluate.py exits non-zero if the fine-tune doesn't win.

Out of the box the base model hedges — recall 0.043 means it almost never confirms a genuinely-grounded answer — and it handles a refusal case correctly 0% of the time. The fine-tune turns the same 1.5B model into a reliable judge. Gate: PASS → ship.

Same judgment as a frontier model, zero marginal cost.

The point of distilling a judge isn't cheaper tokens — it's a judge that runs locally, in CI, on every commit, with no API dependency, no rate limits, and no data leaving the box. So the real question: does the small local judge actually match a frontier one?

No human annotation. No LLM labelling. Correct by construction.

The dataset builder generates the context first, then preserves or deliberately corrupts the answer — so the ground-truth label is known by construction, not guessed. Four balanced case types cover the ways a RAG answer can be right or wrong:

# reproduce the whole thing — deterministic
python src/build_dataset.py --n 1400 --seed 7 --out data
python src/train.py --base Qwen/Qwen2.5-1.5B-Instruct --epochs 2
python src/evaluate.py --adapter out/adapter   # base vs tuned + the gate

What this proves — and what it doesn't yet.

A benchmark is only worth as much as its honesty about scope. Here's the boundary, drawn plainly: