Shipping an LLM application means shipping a system whose failure modes are not exceptions or stack traces but outputs — a chatbot that leaks its system prompt, an agent that can be argued into calling a tool it should not, a support assistant that confidently invents a refund policy. None of these throw an error. Traditional testing, which asserts that a function returns an expected value, cannot express most of them. The discipline that emerged to fill that gap is LLM red teaming: deliberately attacking your own model and application to find the inputs that produce unacceptable behavior, before someone else does.
By 2026 this has matured from ad-hoc prompt experiments into a tooling landscape with distinct layers. This guide maps that landscape — garak and PyRIT at the model layer, DeepTeam and promptfoo at the application layer, Agentic Security for black-box endpoint fuzzing, and Giskard and Inspect for scanning and rigorous evaluation. The through-line is that these tools are not competitors: they catch different classes of failure, and running only one leaves predictable gaps.
What you are actually testing for
Before the tools, it helps to name the failure classes, because they demand different tests. Jailbreaks and prompt injection are the headline: getting the model to ignore its instructions, either through direct manipulation ("ignore previous instructions") or indirectly through content it retrieves — a poisoned document that carries instructions the model then follows. Indirect injection is the more serious variant in RAG and agent systems, because the attacker never touches the prompt box.
Data leakage covers extraction of the system prompt, of PII the model saw in context, or of training data. Excessive agency is the failure class that grows most dangerous as agents gain tools: the model taking an action it should have refused, or chaining tools in a way that exceeds its intended authority. Harmful content is straightforward compliance with requests the application should refuse. And hallucination — confident fabrication — is often the highest-frequency business risk even though it is the least dramatic.
Each of these lives at a different layer. Jailbreak susceptibility is largely a property of the model. Excessive agency and prompt leakage are properties of the application — its system prompt, its tools, its guardrails. Hallucination is a property of the pipeline, especially retrieval quality. This layering is exactly why the tools split the way they do.
Model-layer scanners: garak and PyRIT
garak (from NVIDIA) is the closest thing to nmap for LLMs. It runs a large library of probes against a model — jailbreak families, prompt injection, toxicity, data leakage, encoding attacks — and reports which ones succeeded. You point it at a model (a HuggingFace model, an OpenAI endpoint, a local server) and it works through its catalog. Its value is breadth and low effort: you learn quickly which known attack families your model is susceptible to, without designing anything yourself.
PyRIT (from Microsoft) targets a harder problem: multi-turn and multi-modal attacks. Many real jailbreaks do not work in a single message; they work by establishing context over several turns and escalating gradually. PyRIT provides orchestration for that — attack strategies like crescendo (slow escalation) and TAP (tree-of-attacks with pruning) that adapt based on the model's responses. It is more of an SDK than a scanner: you compose orchestrators, targets, converters, and scorers. That makes it more work to start with and more powerful for research into attacks that a single-shot probe cannot find.
The shared limitation is that both primarily test the model, not your application. A model that resists garak's probes in isolation can still be trivially compromised inside your app, because your system prompt, your retrieval, and your tools create an attack surface the model-layer scan never saw.
Application-layer suites: DeepTeam and promptfoo
This is where DeepTeam and promptfoo fit. Both test your application as deployed, wrapping whatever your app actually is — prompt, retrieval, tools, guardrails — and attacking that whole assembly.
DeepTeam, from the DeepEval team, expresses red teaming as Python: you provide a model_callback that calls your app, declare which vulnerabilities to probe (PII leakage, excessive agency, bias, prompt leakage) and which attacks to use, and it generates and runs adversarial cases. Its attack enhancements are the interesting part — the same base attack rewritten as base64, in another language, as roleplay, or escalated across turns, which is how real attackers evade naive filters. Because it is code, it drops into a test suite and runs in CI.
promptfoo comes at it from evaluation: it is a config-driven CLI for testing LLM outputs that grew a substantial red-teaming capability, auto-generating adversarial prompts across dozens of attack plugins and mapping findings to compliance frameworks. Its strength is CI integration and the fact that the same tool covers both quality evals and security tests, so one harness serves both purposes.
For most teams shipping an LLM product, this layer matters more than the model layer, because it tests the thing you actually deployed. The catch is that it requires you to define what "unacceptable" means for your application — the tools generate the attacks, but you supply the judgment about which outputs are failures.
Black-box and scanning approaches
Two other shapes are worth knowing. Agentic Security treats your endpoint as a black box and fuzzes it agentically — generating probes, observing responses, adapting — with the useful addition of API stress testing. That last part is underrated: a meaningful share of LLM deployments fail on rate limits, token exhaustion, and resource abuse before they fail on content safety, and most red-teaming tools ignore that entirely.
Giskard inverts the usual workflow. Instead of requiring you to specify what to test, it scans the model — using your description of what the app does to generate domain-relevant probes — and produces a report of detected vulnerabilities, which it can then convert into a reusable test suite. That scan-then-testify pattern is valuable precisely because the hardest part of red teaming is knowing what to look for. Giskard finds issues you did not think to test, then locks them in as regression tests.
Finally, Inspect from the UK AI Safety Institute sits slightly apart: it is a rigorous evaluation framework rather than an attack tool, but its structure (datasets, solvers, scorers) and its excellent transcript viewer make it the right choice when you need defensible, reproducible measurement — including of agentic behavior — rather than a vulnerability list.
Building a layered practice
The practical conclusion is that these tools compose. A reasonable 2026 practice looks like this.
When you select or upgrade a base model, run a model-layer scanner — garak for breadth, PyRIT if multi-turn robustness matters to your risk profile. This informs model choice and tells you what the foundation resists on its own.
During development, run a scan-style tool like Giskard against your actual application to discover failure classes you had not anticipated, and convert its findings into tests.
In CI, on every prompt, model, or tool change, run an application-layer suite — DeepTeam or promptfoo — as a gate. This is the highest-value automation, because prompts and tool definitions are the control surface of an LLM app and they change constantly. A prompt edit that seems harmless can remove the sentence that was preventing a jailbreak.
Before release, add black-box fuzzing against the deployed endpoint, including stress testing, to catch deployment-level issues that code-level tests miss.
And keep humans in it. Every automated tool tests known attack families. Novel attacks — the ones specific to your domain, your data, and your tools — come from a person who understands the business logic thinking adversarially. Automation raises the floor; it does not replace the ceiling.
Indirect injection: the attack that breaks the model
One attack class deserves separate treatment because it defeats the intuition most teams start with. Direct prompt injection — a user typing "ignore your instructions" — is what everyone tests first, and it is the easier half. Indirect prompt injection is when the malicious instructions arrive through content the system retrieves: a document in your knowledge base, a web page the agent browses, an email it summarizes, a code comment it reads. The attacker never interacts with your prompt box at all.
This matters enormously for RAG systems and agents, because their entire value proposition is consuming external content. A support assistant that answers from your documentation will faithfully follow instructions embedded in a document if someone can get a document into the corpus — through a public wiki, a customer-submitted ticket, or a scraped page. An agent that reads a GitHub issue can be instructed by that issue. The trust boundary people imagine ("users are untrusted, our data is trusted") does not hold once any part of the corpus is influenceable.
Testing for this is harder than testing direct injection, because it requires simulating poisoned content, not poisoned prompts — you need to place adversarial text where retrieval will find it and verify the model does not act on it. Application-layer tools handle this better than model-layer scanners, since the retrieval pipeline is part of what they exercise, but it often requires you to construct the scenario deliberately.
The mitigations are architectural rather than prompt-based. Treat all retrieved content as untrusted input, the same way you treat user input. Do not let retrieved text reach a position where it can be interpreted as instructions if you can structurally avoid it. Constrain what tools an agent may call while processing untrusted content, and require confirmation for consequential actions. And keep provenance: knowing which document produced a bad answer is what turns an incident into a fix.
What red teaming does not fix
A caution worth stating plainly: finding a vulnerability is not the same as fixing it, and LLM vulnerabilities are frequently not fully fixable. You cannot patch a model the way you patch a buffer overflow. The realistic responses are mitigation rather than elimination: tighter system prompts, input and output guardrails like LLM Guard, constrained tool permissions, human approval for consequential actions, and monitoring for anomalous behavior.
This changes what a red team report means. In traditional security, a finding implies a fix. In LLM security, a finding often implies a risk decision: this attack succeeds at some rate, here is the mitigation, here is the residual risk, is that acceptable for this use case? Teams that expect red teaming to produce a clean bill of health will be perpetually disappointed. Teams that use it to quantify and consciously accept risk get real value.
The corollary is that architecture beats prompt engineering for the failures that matter most. An agent that cannot be tricked into transferring money because it structurally lacks that permission is safer than one relying on the model to refuse. Excessive agency is best addressed by giving less agency. Red teaming's most useful output is often the realization that a capability should not have been exposed in the first place.
The bottom line
LLM red teaming became a real discipline because LLM failures are outputs rather than errors, and ordinary testing cannot express them. The 2026 tooling splits by layer, and that split is the key to using it well: garak and PyRIT probe the model, DeepTeam and promptfoo attack the application you actually deployed, Agentic Security fuzzes the endpoint including its resource limits, and Giskard discovers issues you did not know to look for. Run more than one, gate CI on the application layer where prompts change most, keep human adversarial thinking in the loop, and treat findings as risk decisions rather than bugs awaiting a patch — then fix what you can in the architecture rather than in the prompt.
References and Resources
Tools
Background and analysis
- Best LLM Vulnerability Scanners 2026: Garak, PyRIT, Promptfoo
- LLM Red Teaming Guide 2026: Tools, Attacks & Methodology — AppSec Santa
- LLM Red Teaming Tools Compared — QAwerk
- awesome-ai-security-tools
Related 1337skills cheatsheets