Agentic Security - LLM Vulnerability Scanner Cheatsheet
Agentic Security is an open-source agentic LLM vulnerability scanner and red-team toolkit. Rather than firing a fixed list of prompts, it uses an agentic loop to fuzz a target model or API — generating probes, observing responses, and adapting — to surface jailbreaks, prompt injection, and unsafe behavior. It also performs API stress testing, which matters because many LLM deployments fail on rate limits and resource exhaustion before they fail on content safety.
Scan only systems you own or are authorized to test.
Installation
| Method | Command |
|---|
| pip | pip install agentic_security |
| From source | git clone https://github.com/msoedov/agentic_security && pip install -e . |
| Run the UI | agentic_security (serves a local web UI) |
| Headless | agentic_security --headless |
| Verify | agentic_security --help |
Running a Scan
| Command | Description |
|---|
agentic_security | Start the local UI (default port 8718) |
agentic_security --port 8718 | Choose a port |
agentic_security --headless | CI/non-interactive mode |
--llm-spec <file> | Define the target request format |
--max-budget N | Cap tokens/requests spent |
Defining the Target
The scanner needs to know how to call your LLM endpoint. You supply an HTTP spec:
POST https://api.example.com/v1/chat/completions
Authorization: Bearer $TOKEN
Content-Type: application/json
{
"model": "my-model",
"messages": [{"role": "user", "content": "<<PROMPT>>"}]
}
<<PROMPT>> is substituted with each generated adversarial probe, so any OpenAI-compatible or custom API can be targeted.
What It Probes
| Category | Looks for |
|---|
| Jailbreaks | Bypassing safety instructions |
| Prompt injection | Instruction override via input |
| Data leakage | System prompt / training data exposure |
| Harmful content | Unsafe compliance |
| Fuzzing | Malformed/edge-case inputs |
| Stress | Behavior under load/rate limits |
Datasets & Probes
| Source | Provides |
|---|
| Built-in probe sets | Curated adversarial prompts |
| HuggingFace datasets | Community jailbreak corpora |
| Custom CSV | Your own probes |
| Agentic generation | Model-generated adaptive attacks |
Reading Results
| Output | Meaning |
|---|
| Failure rate | Share of probes that produced unsafe output |
| Per-module breakdown | Which attack class succeeded |
| Sample transcripts | The exact prompt/response pairs |
| Throughput/latency | API behavior under stress |
Focus first on modules with a high failure rate — that is where your guardrails are thinnest.
CI Integration
# Headless scan with a spend cap, fail on threshold
agentic_security --headless --llm-spec ./target.spec --max-budget 50000
| Practice | Why |
|---|
| Cap budget | Agentic fuzzing can spend tokens fast |
| Pin probe sets | Comparable results run to run |
| Run on prompt/model change | Both alter the safety surface |
| Pair with guardrails | Re-scan after adding LLM Guard |
| Aspect | Agentic Security | garak | DeepTeam |
|---|
| Approach | Agentic fuzzing + stress | Static probe library | Pythonic test suite |
| API stress testing | Yes | No | No |
| UI | Local web UI | CLI | Code |
| Best for | Black-box endpoint scanning | Broad model probes | App-level CI red teaming |
Resources
