Giskard - ML & LLM Testing Framework Cheatsheet

Giskard - ML & LLM Testing Framework Cheatsheet

Giskard is an open-source testing framework for ML and LLM applications. Its signature feature is an automated scan that probes a model for vulnerabilities — hallucination, prompt injection, harmful content, robustness failures, bias, and data leakage — and produces a report you can convert directly into a reusable test suite. That scan-then-testify workflow is what distinguishes it: you do not have to know what to test for in advance.

Installation

Method	Command
pip	pip install giskard
LLM extras	pip install "giskard[llm]"
Model key	export OPENAI_API_KEY=... (used for LLM-based detectors)
Verify	python -c "import giskard; print(giskard.__version__)"

Wrapping an LLM App

import giskard
import pandas as pd

def predict(df: pd.DataFrame):
    return [my_llm_app(q) for q in df["question"]]

model = giskard.Model(
    model=predict,
    model_type="text_generation",
    name="Support Assistant",
    description="Answers customer questions using our docs",
    feature_names=["question"],
)

dataset = giskard.Dataset(
    pd.DataFrame({"question": ["How do I reset my password?"]}),
    target=None,
)

Field	Why it matters
description	Giskard uses it to generate relevant probes
model_type	text_generation, classification, regression
feature_names	Input columns

The description is not cosmetic — the scanner generates domain-specific adversarial inputs from it.

Scanning

report = giskard.scan(model, dataset)
report.to_html("scan_report.html")

Detector	Finds
Hallucination	Unsupported or incoherent claims
Prompt injection	Instruction override
Harmfulness	Unsafe content generation
Robustness	Sensitivity to trivial input changes
Sensitive disclosure	Leaking secrets/PII
Stereotypes	Bias in outputs
Performance bias	Uneven accuracy across slices (tabular/NLP)

# Scan only specific detector groups
report = giskard.scan(model, dataset, only=["hallucination", "jailbreak"])

Scan → Test Suite

suite = report.generate_test_suite("Support Assistant tests")
suite.run()

This is the key workflow: findings become regression tests, so a fixed issue stays fixed and can be gated in CI.

Custom Tests

from giskard import test, TestResult

@test(name="No refund promises")
def no_refund_promise(model, dataset):
    outputs = model.predict(dataset).prediction
    bad = [o for o in outputs if "guaranteed refund" in o.lower()]
    return TestResult(passed=len(bad) == 0, metric=len(bad))

suite.add_test(no_refund_promise).run()

RAG Evaluation (RAGET)

Giskard includes a RAG evaluation toolkit that generates a question set from your knowledge base and scores each pipeline component.

Component scored	Tells you
Generator	Is the LLM’s answer good given context?
Retriever	Did retrieval surface the right chunks?
Rewriter	Is query rewriting helping?
Router	Was the query routed correctly?
Knowledge base	Is the source material adequate?

Component-level scoring is more actionable than a single end-to-end number — it tells you which stage to fix.

Giskard vs Related Tools

Aspect	Giskard	DeepEval	garak
Core idea	Auto-scan → test suite	Unit-test style metrics	Model probe scanner
Scope	ML + LLM + RAG	LLM app evals	LLM model vulnerabilities
Report	Rich HTML scan report	CI output/dashboard	CLI report
Best for	Discovering unknown issues	Gating known metrics	Broad model probing

Pairs with DeepEval for metric gating and Ragas for RAG scoring; Giskard’s edge is finding issues you did not think to test.

Resources

• GitHub Repository
• Documentation
• RAGET (RAG evaluation)