Case study 02
rag-verdict
Pytest for RAG agents: an open-source framework that tests what your agent does, not just how its answers score.
Python · pytest · LLM-as-judge · CI · Open source (MIT)
problem
RAG evaluation tools like RAGAS and DeepEval mostly average quality scores over answers. That misses the failures that actually embarrass you in production: the retrieval tool that silently never fired, the citation pointing at a document that does not exist, the agent that confidently answers questions its corpus knows nothing about. Scores don't catch behavior. Tests do.
approach
rag-verdict treats a RAG agent like code under test. Behavioral probes check whether tools fire when they should, whether every citation resolves to a real document, and whether the agent refuses out-of-corpus questions instead of improvising. A pluggable adapter (Python or HTTP) connects any RAG system in any language. An LLM-as-judge layer with structured output handles the fuzzy cases, and it degrades gracefully when no API key is present, so the deterministic checks still run in CI. Results come back as PASS, FAIL, or WEAK verdicts rather than a decimal nobody can act on.
outcome
An MIT-licensed framework that turns 'our agent seems fine' into a test suite with verdicts. CI-friendly, language-agnostic, and opinionated about what matters. It's the distilled lesson of every RAG system I've built: retrieve before you generate, cite what you retrieved, and test the behavior, not the vibes.