What is RAG Evaluation Starter?
RAG Evaluation Starter is an open-source Python tool for checking retrieval-augmented generation systems against a set of questions and expected answers. It measures retrieval, answer quality, and latency, and reports results for each question as well as summary metrics. Developers connect their own retriever and answer generator through two Python functions; the command-line demo runs a bundled mock system. It requires no SaaS account or mandatory API key, and is distributed under the MIT licence.
RAG Evaluation Starter Features:
Evaluation Metrics: Each question can be scored for Retrieval Recall@K, answer faithfulness, answer relevance, Exact Match, and Token F1.
Per-Question Verdicts: Metrics receive PASS, WARN, FAIL, SKIP, or ERROR verdicts where applicable, with configurable thresholds for interpreting scores.
Latency Recording: The evaluator records how long retrieval and generation take for each query, though latency is not scored.
Optional Embeddings: Answer relevance uses cosine similarity from a local embedding model, which downloads on first use, or a supplied embedder.
Custom System Hooks: Provide a retriever that returns text and source IDs and a generator that returns an answer to evaluate your own system.
Result Exports: Evaluation results can be displayed in a terminal table and saved as JSON or CSV.
RAG Evaluation Starter Benefits:
Find Retrieval Gaps: Comparing expected sources with retrieved chunks helps pinpoint when missing context, rather than answer generation, is the problem.
Separate Quality Issues: Faithfulness and relevance scores help distinguish unsupported answers from answers that stay grounded but miss the question.
Check Changes: Repeating evaluations against a consistent question set makes it easier to spot quality or latency regressions after system updates.
Start Without SaaS: A local workflow with no required account or API key lets teams begin evaluating without adopting a hosted service.
Set Relevant Standards: Editable thresholds let teams interpret scores according to the quality expectations of their own application.
Use Cases:
Support Bot Checks: Build a golden set from difficult or incorrectly answered support questions to check how a customer-facing RAG bot handles them.
Knowledge Base Updates: Rerun representative questions after documentation changes to see whether retrieval still finds the expected source material.
Retrieval Tuning: Use Recall@K results to assess whether changes to chunking or the embedding model improve source retrieval.
Hallucination Checks: Include questions whose answers are absent from the knowledge base and inspect faithfulness and answer scores.
Release Regression Tests: Evaluate the same question set before and after a system change to look for shifts in answer quality and latency.

