EncouRAGe - Local RAG Evaluation Cheatsheet
EncouRAGe is an open-source framework for evaluating retrieval-augmented generation (RAG) pipelines with an emphasis on running locally, fast, and reliably. Where many RAG evaluators call out to hosted judge models, EncouRAGe is designed to compute retrieval and generation metrics on your own hardware, keeping evaluation data private and results reproducible. It fits the 2026 pattern where RAG quality is treated as a measurable engineering property rather than a vibe.
Evaluation frameworks evolve quickly. Treat the commands below as the shape of the workflow and confirm exact flags against the current docs.
Installation
| Method | Command |
|---|
| pip | pip install encourage |
| From source | git clone the repo, then pip install -e . |
| Local judge models | pull models via your local runtime (e.g. Ollama/vLLM) |
| Verify | python -c "import encourage; print('ok')" |
Why Local Evaluation
| Benefit | Why it matters |
|---|
| Privacy | Your documents/queries never leave your machine |
| Reproducibility | Same model + seed → same scores |
| Cost | No per-eval API billing |
| Speed | Batch locally without rate limits |
What Gets Measured
RAG evaluation splits into the two halves of the pipeline.
| Half | Metrics |
|---|
| Retrieval | Context precision, context recall, hit rate, MRR/NDCG |
| Generation | Faithfulness (grounding), answer relevance, correctness |
| End-to-end | Whether the final answer is right and supported |
Core Workflow
# Conceptual flow: assemble a dataset, run your RAG, score it
import encourage
dataset = encourage.load_dataset("eval_questions.jsonl") # question, ground_truth, contexts
results = encourage.evaluate(
dataset=dataset,
metrics=["faithfulness", "answer_relevance",
"context_precision", "context_recall"],
judge_model="local/llama-3.1-8b-instruct", # runs locally
)
print(results.summary()) # per-metric scores
results.to_json("eval.json")
| Step | Purpose |
|---|
| Dataset | Questions + (optional) ground truth + retrieved contexts |
| Metrics | Which retrieval/generation dimensions to score |
| Judge model | The local LLM used for LLM-as-judge metrics |
| Results | Aggregate + per-example scores, exportable |
Building an Eval Dataset
| Field | Meaning |
|---|
question | The user query |
contexts | Chunks your retriever returned |
answer | Your pipeline’s generated answer |
ground_truth | Reference answer (for correctness/recall) |
Harvest real questions from production traces to make the eval representative.
Interpreting Results
| Symptom | Likely cause |
|---|
| Low context recall | Retriever misses relevant chunks (chunking/embedding) |
| Low context precision | Too much irrelevant context retrieved (add reranking) |
| Low faithfulness | Model hallucinates beyond the context |
| Low answer relevance | Answer drifts from the question (prompt/query issue) |
Fitting It Into CI
# Gate a RAG change on evaluation scores
python run_eval.py --dataset eval.jsonl --min-faithfulness 0.85 \
--min-context-recall 0.8 || exit 1
Run the same eval set on every retrieval/prompt/model change so regressions are caught before they ship.
EncouRAGe vs Other RAG Evaluators
| Aspect | EncouRAGe | RAGAS | TruLens | DeepEval |
|---|
| Emphasis | Local, fast, reproducible | Reference-free RAG metrics | Monitoring + OTel tracing | Unit-test mindset, CI |
| Judge | Local models | Configurable | Configurable | Configurable |
| Best for | Private, on-prem eval | Quick RAG scoring | Production monitoring | Pytest-style CI gates |
Pairs conceptually with RAGAS, TruLens, and DeepEval — pick by whether you need local privacy, quick metrics, monitoring, or CI gating.
Resources
