Inspect AI - LLM Evaluation Framework Cheatsheet
Inspect is an open-source framework for large language model evaluations, created by the UK AI Safety Institute. It gives evals a clear structure — a dataset of samples, a solver pipeline that produces answers, and a scorer that grades them — plus a first-class viewer for inspecting exactly what the model saw and did on every sample. It is designed for rigorous, reproducible evaluation, including agentic tasks and tool use, and is widely used for safety and capability testing.
Installation
| Method | Command |
|---|
| pip | pip install inspect-ai |
| uv | uv add inspect-ai |
| Model key | export OPENAI_API_KEY=... / ANTHROPIC_API_KEY=... |
| VS Code | Install the Inspect extension for the viewer |
| Verify | inspect --version |
The Three Building Blocks
| Component | Role |
|---|
| Dataset | Samples with input and target |
| Solver | Steps that turn input into model output |
| Scorer | Grades output against the target |
A Minimal Eval
from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message
@task
def security_qa():
return Task(
dataset=example_dataset("theory_of_mind"),
solver=[
system_message("You are a concise security expert."),
generate(),
],
scorer=model_graded_fact(),
)
inspect eval security_qa.py --model openai/gpt-4o
Running Evals
| Command | Description |
|---|
inspect eval task.py --model openai/gpt-4o | Run an eval |
inspect eval task.py --limit 20 | Only the first 20 samples |
inspect eval task.py --model-base-url URL | Custom/local endpoint |
inspect eval task.py -T param=value | Pass task parameters |
inspect eval-retry <log> | Retry failed samples |
inspect view | Open the log viewer |
Solvers
| Solver | Does |
|---|
generate() | Call the model |
system_message(...) | Prepend a system prompt |
chain_of_thought() | Add CoT prompting |
self_critique() | Model reviews its own answer |
use_tools([...]) | Enable tool calling |
basic_agent() | A ReAct-style agent loop |
Scorers
| Scorer | Grades by |
|---|
match() | Exact/substring match |
includes() | Target appears in output |
pattern(regex) | Regex extraction |
model_graded_qa() | LLM judge against a rubric |
model_graded_fact() | LLM judge for factual equivalence |
choice() | Multiple-choice answer |
from inspect_ai.tool import bash, python
from inspect_ai.solver import basic_agent
solver = basic_agent(tools=[bash(), python()], max_attempts=3)
| Capability | Note |
|---|
| Tool use | Built-in bash, python, web tools |
| Sandboxing | Run tools in Docker for isolation |
| Multi-turn | Agent loops with attempt limits |
| Custom tools | Decorate Python functions as tools |
The Viewer
| Shows | Why it matters |
|---|
| Every sample | Input, target, output, score |
| Full transcript | Each message, tool call, and result |
| Scoring rationale | Why the judge graded as it did |
| Metrics | Accuracy and custom metrics |
Being able to read the transcript of failures is what turns a score into an actionable finding.
Inspect vs Other Eval Frameworks
| Aspect | Inspect | DeepEval | Ragas |
|---|
| Origin | UK AI Safety Institute | Confident AI | Exploding Gradients |
| Focus | General + agentic evals | Unit-test style app evals | RAG-specific metrics |
| Viewer | Rich, first-class | Reports/dashboard | Scores |
| Best for | Rigorous capability/safety evals | CI test gating | RAG pipeline scoring |
Complements DeepEval for CI-style testing and Ragas for RAG-specific metrics.
Resources
