Prompt Injection Prevention: How to Defend LLM and AI Agent Applications

Prompt injection prevention is the set of controls that stop an attacker's text from steering a large language model into actions or disclosures the developer never intended. It is the hardest problem in LLM security because of one structural fact: the model reads its own instructions, the user's input, retrieved documents, and the output of any tools it calls as a single stream of text, and it cannot reliably tell which part is a trusted instruction and which is attacker-controlled data. This guide, organized around the Pwned Labs AI Attack Taxonomy, explains why the two defenses most teams reach for first, a firm system prompt and a content filter, are not boundaries, where the common prompt-level techniques help and where they stop, then sets out the architectural controls that do hold, each mapped to the family of attack it closes, how to detect injection when prevention slips, and how to learn the techniques hands-on. Prompt injection is LLM01, the top risk in the OWASP Top 10 for LLM Applications.

Why prompt injection is hard to prevent

A language model is given instructions, usually in a hidden system prompt, that shape how it behaves. Prompt injection is any input that gets the model to follow an attacker's instructions instead of those rules. It resists prevention because the model has no built-in separation between code and data: the system prompt, the user message, a retrieved document, and a tool's output all arrive as one flat sequence of tokens, and the role labels between them are themselves just more text. The model is pattern-matching over the whole stream, so a sentence written to look like a system instruction competes on equal terms with the real one. That single fact drives every control below. The separation the model cannot enforce has to be enforced by the application around it.

Injection comes in two forms, and the second is the one that matters most for prevention. In direct injection the attacker types the malicious instruction straight into the input. In indirect injection the attacker plants it in content the model will later read as trusted context, such as a document in a knowledge base, a web page the model summarizes, a support ticket, a code comment, or an email in an inbox the assistant triages. Indirect injection is the dangerous case because the attacker never has to speak to the model directly, and the malicious text arrives wearing the system's own clothes. For a deeper treatment of that variant, see our explainer on indirect prompt injection.

Two defenses that feel safe and are not

The system prompt is a preference, not a control

Instructions such as "only discuss this user's own account" or "never reveal configuration values" live in the same token stream the model is reasoning over. They express intent, and a cooperative model usually honors them, but nothing enforces them. A request framed convincingly enough, or one that arrives through trusted content rather than the chat box, can talk the model out of them. The practical corollary is blunt: never place a secret in the system prompt. Credentials, signed storage URLs, internal endpoints, and API keys all become readable, because a well-framed request can make the model repeat what a direct request would be refused. The fix is not a better-worded instruction, it is to keep the secret out of the model's context entirely, which is also why System Prompt Leakage is its own entry, LLM07, in the OWASP list.

Content filters are a speed bump, not a wall

A content filter that sits in front of or behind the model blocks input and output matching known-bad patterns, and it will stop the blunt attacks, which is worth having. But it matches patterns, so it catches the request that looks like an attack and misses the request that reads like an ordinary document, a training example, or a routine internal note. A filter also matches a specific written form, so the same sensitive value expressed in a slightly different shape can pass straight through. Use filters as one layer among several, and never let their presence stand in for a real boundary. A filtered blunt attack is not the same thing as a safe application.

Prompt-level defenses: useful layers, not boundaries

Most published advice on preventing prompt injection concentrates on the prompt itself, and those techniques are worth applying as layers in a defense-in-depth stack. They raise the cost of an attack and catch the clumsy attempts. What none of them can do is draw a boundary an adversary cannot cross, because they all operate inside the same untrusted text stream the model is trying to interpret. Apply them, and budget for them failing.

• Parameterization and structured prompts: keep developer instructions and user input in clearly separate fields rather than concatenated into one blob, the same instinct behind parameterized SQL. It helps, but a chatbot's input is open-ended, so the separation stays a convention the model can be argued across.
• Delimiters and spotlighting: wrap untrusted input in markers, or tag it, and instruct the model to treat anything inside as content only. Useful, but the markers are themselves text an attacker can imitate or close early.
• Input and output guardrail models: put a classifier or a second model in front of the input and behind the output to flag injection attempts and block sensitive data leaving. This is genuine defense in depth and catches known patterns, but a classifier can be fooled by the same framing that fools the first model, and it brings false positives.
• Canary tokens for leak detection: place a unique marker in the system prompt and alert if it ever appears in output, which tells you a leak happened. This detects a system-prompt leak rather than preventing it, and it is no substitute for keeping real secrets out of the prompt to begin with.

Treat every item here as a layer, never as the boundary. The boundary is set by the architectural controls in the next section, which hold whether or not the model is successfully injected.

How to prevent prompt injection: the controls that hold

Every control below is architectural, and none of them depends on the model behaving. That is the point. The model is the component you cannot trust to separate instructions from data, so the boundaries are drawn in the system around it. The Pwned Labs AI Attack Taxonomy groups AI attacks into seven families by where they act on the pipeline, from the instruction the model reads through to the actions it takes, and each control here closes one or more of those families. Layered together with the prompt-level defenses above, they are what turns a successful injection into a contained event instead of a breach.

Treat all non-developer text as untrusted

User input is the obvious case, but the dangerous sources are the ones that feel trustworthy: retrieved documents, tool and function results, API responses, uploaded files, and anything resolved from them. If text did not come from your own code at build time, it is data and not instructions, however authoritative it looks by the time it reaches the model. Keep a structural separation between the instructions your application issues and the content it feeds the model to work on, and never let retrieved content be promoted into the instruction role. This closes the Instruction and Context families.

Keep secrets out of the model's context

No credentials, tokens, signed URLs, or internal identifiers in the system prompt or anywhere the model can read. When an action needs a secret, have the application hold it and perform the action on the model's behalf, so the value never enters the text the model sees. A secret the model cannot read is a secret an injection cannot leak. This closes the Disclosure family.

Scope every tool to the authenticated session

This is the control teams miss most often, and the one that turns an injection into a data breach. When the model calls a tool, that tool runs with the application's authority, not the user's. If a lookup tool takes an account identifier as an argument and runs whatever it is handed, the model will pass it the identifier an attacker named, and the server will return another customer's record. This is the classic confused-deputy problem, and it needs no clever injection at all, only a tool that trusts the model's arguments. Bind every lookup and action to the authenticated session on the server side, ignore any identifier that arrives through the conversation, and re-check entitlement on every call rather than trusting the argument. This closes the Agency and Platform families.

Never pass model output to a shell or evaluator

If anything downstream executes the model's output, whether a shell, an eval, a template engine that runs code, or a command built from the model's text, then an attacker who can influence the model's input has code execution in your pipeline. Model output is untrusted by definition, because its input is untrusted. Render it, store it, and display it, but do not run it. This is Improper Output Handling, LLM05, and the mitigation is to validate and encode model output at every sink the same way you would any other untrusted input.

Allowlist where data is allowed to go

Give any tool that reaches outside the environment a fixed, allowlisted destination. A notification tool that posts to whatever URL the model provides is an exfiltration channel waiting to be used, and the same holds for tools that send email, write to storage, or call webhooks. Decide the allowed destinations in code and refuse anything else, so that even a model fully under an attacker's control has nowhere to send what it was tricked into reading. A default-deny egress policy collapses most exfiltration without ever having to detect the injection.

Give agent identities least privilege

An autonomous agent runs with an identity, and that identity's permissions are the real blast radius of any injection against it. If the agent can read every secret in the vault, a successful injection can too. Scope the agent's identity to exactly the resources its job requires and nothing more. In most real agent compromises the damage comes not from the injection itself but from an over-privileged identity sitting behind it, which is why the OWASP list treats this as Excessive Agency, LLM06.

Pin and isolate third-party and MCP tool servers

Modern agents load their tools from external servers, often over the Model Context Protocol, and update them like any other dependency, which makes the tool supply chain part of the attack surface. A poisoned or silently updated tool server can inject instructions into the agent with no user input at all, and the agent reads that tool output as trusted context. Pin third-party tool servers to known versions, isolate them, and treat their output as untrusted input. Apply the same care to packages an agent installs during its own setup, because a dependency's install step runs on the machine that installs it. This is Supply Chain, LLM03.

Keep a human in the loop for high-impact actions

For anything consequential and hard to undo, such as moving money, deleting data, changing access, or sending external communications, require explicit human confirmation rather than letting the model trigger it directly. This does not prevent injection, but it caps what an injection can accomplish on its own, and it is cheap insurance for the case where every other layer fails at once.

Prompt injection controls mapped to what they stop

Each control below closes a specific path an injection would otherwise take, mapped to its family in the Pwned Labs AI Attack Taxonomy and its entry in the OWASP Top 10 for LLM Applications. Read together they turn prompt injection from an open door into a contained nuisance.

How to detect prompt injection when prevention slips

Prevention is the priority, but you should assume some attempts get through and instrument for them. The most reliable signal is behavioral rather than signature-based: an identity or a tool doing something outside its normal pattern. Watch for an outbound tool contacting a destination it does not normally reach, an agent identity accessing a class of resource it never usually touches, or a sudden internal enumeration step just before an external call. Enable data-access audit logging on your sensitive services so those reads leave a record, and alert on the out-of-character event rather than trying to pattern-match the payload. When you investigate, the body of the outbound call usually shows exactly what left, and the sequence of tool calls in a single session usually shows the attacker's two moves, finding the target and then taking it. This is the same behavioral approach Pwned Labs uses when assessing an agent, covered in our AI penetration testing guide.

Prompt injection prevention checklist

• No secrets in the system prompt or anywhere in the model's context.
• All retrieved content, tool output, and uploads treated as untrusted data, never as instructions.
• Prompt-level layers in place (structured prompts, delimiters, input and output guardrails) and understood to be layers, not boundaries.
• Every tool scoped to the authenticated session on the server, ignoring identifiers from the conversation.
• Model output never executed by a shell or evaluator, and validated at every sink.
• Outbound tools restricted to allowlisted destinations, with default-deny egress.
• Agent identities scoped to least privilege.
• Third-party and MCP tool servers, and packages the agent installs, pinned and isolated.
• Human confirmation required for high-impact actions.
• Behavioral detection and data-access audit logging on sensitive services.

Practice AI security hands-on

The fastest way to understand these controls is to attack a system you are allowed to break and then defend it, because the gap between a defense that feels safe and one that holds is obvious the moment you watch the first one fail. The Pwned Labs AI Systems Attack and Defense bootcamp works every family of the AI Attack Taxonomy hands-on, from direct and indirect prompt injection through tool abuse, improper output handling, and full agentic compromise across CI/CD pipelines and MCP tool servers, using the same tooling a tester uses, including garak, PyRIT, and promptfoo, and you validate the skill in the AISRTP certification. You can drill individual techniques in the PromptStorm cyber range and hands-on labs such as breaching the perimeter via prompt injection and indirect prompt injection for Azure access, or start with the introduction to AI hacking and our AI security training overview.

Frequently asked questions

Can prompt injection be fully prevented?

Not at the model layer, because the model cannot reliably separate instructions from data in the text it reads. What you can do is prevent an injection from causing harm, by constraining what the model can see, what its tools are allowed to do, and where data can go. A well-designed application assumes injection will succeed at the prompt level and makes sure that nothing damaging follows.

How do you prevent prompt injection?

With architectural controls rather than prompt wording: treat all user and retrieved text as untrusted data, keep secrets out of the model's context, scope every tool to the authenticated session, never execute model output, allowlist where tools can send data, give agent identities least privilege, pin and isolate third-party and MCP tool servers, and require human confirmation for high-impact actions. Pwned Labs organizes these controls against its AI Attack Taxonomy, so each defense maps to the family of attack it closes, with prompt-level techniques such as delimiters and guardrail models sitting on top as extra layers.

What is the Pwned Labs AI Attack Taxonomy?

The Pwned Labs AI Attack Taxonomy groups AI and LLM attacks into seven families by where they act on the pipeline: Instruction, Context, Model, Disclosure, Output, Agency, and Platform. It maps to the OWASP Top 10 for LLM Applications and to MITRE ATLAS, and Pwned Labs uses it both to test an AI application and to organize the defenses that contain each family, so a prevention control can be traced to the exact attacks it closes.

Does a system prompt stop prompt injection?

No. Instructions in the system prompt express intent but are not enforced, so a convincingly framed request, or one delivered through content the model later retrieves, can override them. Treat the system prompt as a strong hint to a cooperative model, never as a control an adversary cannot cross, and never place a secret in it.

Do input and output filters prevent prompt injection?

They help but do not prevent it. Input filters and output guardrail models catch known attack patterns and block obvious data leaks, which is worth having as a layer, but they match patterns and can be fooled by a request framed to look harmless. Treat filters as defense in depth on top of the architectural controls, never as the boundary that stops an injection on their own.

What is the difference between direct and indirect prompt injection?

In direct injection the attacker types the malicious instruction into the input. In indirect injection the attacker plants it in content the model later reads as trusted context, such as a document, email, ticket, or web page, so the model is compromised during an ordinary user's request with no direct access to it. Indirect injection is the more dangerous form and the harder one to prevent.

How do you prevent indirect prompt injection?

Treat every piece of retrieved or tool-supplied content as untrusted data that can never act as an instruction, keep a structural boundary between instructions and data, and constrain what the model is allowed to do when it acts on either. Because the injected text arrives through a trusted-looking source, the defense is to remove the model's ability to do anything harmful with it rather than to try to detect the malicious text.

How do I learn to prevent prompt injection?

Learn it by attacking and then defending a system you are authorized to break, so you see which defenses hold under a real attack and which only feel safe. Pwned Labs runs hands-on AI security labs, the AI Systems Attack and Defense bootcamp, and the AISRTP certification for exactly this, using the same tooling professional testers use.