Proof, not slides.
Six open-source flagships and a merged-pending contribution to a real eval framework. Each one is live, inspectable, and backed by measured results.
shipgate
Point it at a config of cases + assertions and it computes a pass rate, then exits non-zero when quality drops below a threshold or regresses against a saved baseline. The deterministic checks run at zero API cost, so the gate fires on every commit.
groundcheck
A 1.5B open model, fine-tuned with QLoRA into a groundedness judge that runs off-box. The rule it shipped under: the fine-tune only ships if its own eval suite beats the base model. The gate, not vibes, made the call. (Corpus is synthetic — proves the pipeline end-to-end.)
mcp-vitals
Grades any MCP server A–F on whether an agent can actually use it — static, behavioral, and agent-usability layers. On the official filesystem server the most useful finding wasn’t a vulnerability; it was a tool name: an agent reached for read_file, which doesn’t exist.
abstentionbench
A meta-eval benchmark for the answer-or-abstain decision: 270 cases (real SQuAD 2.0 traps + an adversarially-verified hard tier) that measure whether eval tools can catch over-refusal — the failure faithfulness metrics structurally can't see. Honest leaderboard: only measured numbers.
agentic-rag-mcp
A multi-agent RAG server exposed over MCP: plan → retrieve → synthesize → self-critique. Every answer cites its exact source chunk or the agent refuses; groundedness, citation and refusal evals run in CI.
reelforge
An agentic short-form video engine on LangGraph + Claude: a supervisor orchestrates tool-calling sub-agents with deterministic gates — image-first, model-accurate, one topic to a finished plan.
AbstentionMetric
A community metric contributed to DeepEval that scores the answer-vs-abstain decision — the failure faithfulness metrics miss: confident answers with no support, and over-refusals. Mirrors their metrics, with docs and offline tests.