AI red teaming is the authorized, adversarial testing of an AI system to find how it can be made to behave against its operator's intent, then to prove the real-world impact of that behavior. When the target is an AI agent, one that reads untrusted content and calls tools, the red team's job is to turn a single steered response into an action the user never asked for. Pwned Labs teaches this agent-focused work hands-on in its AI Systems Attack and Defense bootcamp, which certifies the skill with the AISRTP exam against live LLM-backed and agentic applications.
What is AI red teaming?
AI red teaming is structured adversarial testing of an AI system, usually run in a controlled environment and often alongside the team that built it, to find flaws before an attacker does. The NIST glossary frames it that way, and the OWASP GenAI Red Teaming Guide breaks it into model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. In practice the interesting work sits in the last three: how the model is wired into a product, what it is trusted to do, and what happens when it is steered.
The term overlaps with AI penetration testing and is often used for the same work. A useful way to split them is scope: a penetration test is a bounded assessment of one application against a defined methodology, while red teaming tends to be objective-driven, chasing a goal such as "exfiltrate another tenant's data" across whatever surface reaches it. Neither is a model benchmark or an eval, which measures a model's quality or safety in isolation and never touches the application around it. Pwned Labs sets out the full engagement method, from scoping to the report, in its guide to AI penetration testing.
Why an AI agent widens the attack surface
Even a chat assistant that only answers questions has more reach than it looks. Its system prompt sits one successful injection away from the user, and that prompt often holds more than instructions: a connection string, an API key, a signed storage URL, or an authorization rule written as a sentence instead of enforced in code. The system prompt is not a secret and should not be used as a security control, so when it leaks, the real finding is rarely that the model repeated its instructions, it is whatever those instructions carried. A request that looks like the application's own job, such as writing a training example, can make the assistant quote a credential it refuses to print when asked directly. OWASP tracks this as sensitive information disclosure and system prompt leakage, LLM02 and LLM07 in the OWASP Top 10 for LLM Applications.
The same steering that makes an assistant quote a secret from its prompt becomes more dangerous once the assistant is an agent, because an agent can act. It decides which tool to call and with what arguments, and the server runs that call under an identity with real permissions, so a steered decision becomes a real action: reading a record, sending a message, writing a file, or calling a cloud API. OWASP tracks this as excessive agency, LLM06, and it names three ways an agent ends up with too much reach: excessive functionality, excessive permissions, and excessive autonomy.
Three agent properties widen the target further. Memory means an instruction planted once can persist and fire on a later, unrelated request. Retrieval means the agent pulls in documents, tickets and web pages that an attacker may be able to write to. Tool chaining means a request that starts in scope can drift out of it as one tool's output becomes the next tool's input. The red team's focus is not whether the model can be tricked, which it usually can, but how far a tricked agent can actually reach. Pwned Labs explains the agent attack surface in its glossary entry on AI agent security.
How a red team maps an AI agent before attacking it
The map, not the model, drives the test, so reconnaissance comes first. Against an agent the red team works out four things. What untrusted text can reach the model: the user message, yes, but also any document, ticket, email, web page or tool result the agent reads during a normal request. Which tools the agent exposes and what each one does, since a read-only lookup and a state-changing action carry very different risk. Which data sources it retrieves from, and who can write to them. And the identity each tool runs as, because the impact of any abuse is the tool's capability multiplied by the permissions that identity holds.
Fingerprinting the stack is part of this: confirming the application is model-backed, probing for a readable system prompt, and listing the tools the agent will admit to. A system prompt that leaks is both a finding, mapped to system prompt leakage, LLM07, and a map of the agent's capabilities, which is why the red team treats it as a design artifact to read rather than a secret to protect. Pwned Labs opens its AI Systems Attack and Defense bootcamp with this recon and fingerprinting step, so the attack that follows is aimed at a surface that has actually been enumerated.
Walking a red team attack through an AI agent
The clearest way to see AI red teaming is to follow one chain through a tool-using agent. Take a customer support agent that answers from a knowledge base built out of incoming tickets and can call a tool to look up order and account records. The engagement has an objective, set before any payload is written: read a record belonging to a customer the tester has no right to. Every step below works back from that goal, and none of them needs direct access to the model.
- Find the untrusted input. The agent reads ticket content into its context, so a ticket body is attacker-controlled text that the model will later treat with the same weight as the user's request. That is indirect prompt injection, the delivery route behind most real agent incidents.
- Plant the instruction where the agent will read it. The red team files a ticket whose body is aimed at the model, not at the staff member, so it sits in the knowledge base waiting to be retrieved. Nothing has happened yet.
- Trigger it through normal use. When someone later asks the agent to summarize open tickets, retrieval returns the poisoned ticket and the planted instruction is read as trusted context.
- Turn the steered response into a tool call. The instruction directs the agent to call the lookup tool with an account identifier the current user has no right to, which is a correctly formed call aimed at the wrong object. The server executes it under the tool's identity.
- Follow it to an observable effect. The tool returns another customer's record, and the agent places that data into its reply or into an outbound link that fetches a URL when the response renders. That rendered link or image is the exfiltration channel.
This is not hypothetical. ForcedLeak (CVSS 9.4, disclosed in 2025) chained an indirect injection in Salesforce Agentforce, delivered through a Web-to-Lead field, into CRM data exfiltration, and was reported and patched by Salesforce. A red team proves a chain like this end to end rather than stopping at "the agent can be injected," because the injection only matters once it reaches the record it should not have read. Pwned Labs builds its indirect prompt injection lab on this same delivery route, instructions planted in content an assistant later retrieves, and follows it into Azure.
How AI red teaming maps to OWASP and MITRE ATLAS
Red team findings are more useful when they line up with the frameworks a client already tracks. The steps in the chain above each map to an OWASP Top 10 for LLM Applications risk and to a technique in MITRE ATLAS, the public knowledge base of adversary tactics and techniques against AI systems, which has expanded to cover agentic AI. Mapping each attempt to both lets a report show coverage against a shared standard rather than an ad hoc list.
| Red team step against the agent | What it abuses | OWASP LLM Top 10 |
|---|---|---|
| Plant an instruction in a retrieved ticket or document | Indirect prompt injection into the context the agent reads | LLM01 Prompt Injection, LLM08 Vector and Embedding Weaknesses |
| Read the agent's system prompt and tool list | Disclosure of the hidden prompt and capabilities | LLM07 System Prompt Leakage, LLM02 Sensitive Information Disclosure |
| Coerce a tool call with an attacker-chosen argument | Excessive agency: a call aimed at the wrong object | LLM06 Excessive Agency |
| Exfiltrate through a rendered link or image in the reply | Improper handling of attacker-influenced output | LLM05 Improper Output Handling |
| Reach cloud identity, storage or another tenant behind the tool | The control plane the agent runs on | LLM10 Unbounded Consumption and the API layer |
Pwned Labs groups these into a seven-family taxonomy and uses it as a coverage record on assessments, so a client sees a result for every technique that was run, including the ones where nothing worked.
How AI red teaming differs from traditional red teaming
The mindset carries over from any red team: enumerate the surface, find where untrusted input meets trusted logic, and abuse the gap. What changes is the medium. Instead of crafting a payload for a parser you are persuading a model in natural language, and instead of a single deterministic exploit you sweep a catalog of phrasings and record a success rate, because the same input can succeed one time and fail the next. One refusal proves nothing, so a control is only called effective after many variations and encodings have been tried against it. The yield is often low and counterintuitive. In one Pwned Labs scenario, plainly asking for a secret held in the system prompt leaked it about one time in ten, the authority and config-override tricks did worse, and only a request framed as the application’s own work raised that to roughly one in five, so a test that fires a handful of payloads can miss a real flaw entirely.
The base skills transfer directly. Comfort with HTTP, reading and editing JSON, an intercepting proxy such as Burp Suite or Caido, and a general security foundation cover most of the day-to-day work, and a machine learning background is not required to start. What is specific to AI red teaming is the habit of treating every retrieved document and every tool response as a place an instruction might be hiding, and of following a steered response all the way to an effect. Pwned Labs sets the same bar for its AISRTP track, where lab guides give every command in full so no prior AI security experience is assumed.
That shared mindset is also why AI red teaming and AI penetration testing run many of the same techniques against an agent. What separates the red team is its starting point: it works from an adversary objective, like the record-read goal in the walkthrough above, and follows whatever surface reaches that goal rather than checking one application against a fixed list. What it reports is proof that the objective was reachable end to end on the live system, not a catalog of isolated weaknesses.
How do I start AI red teaming on agents?
AI red teaming is learned by doing it, in order, against targets you own or are explicitly authorized to test. A path that works starts with the beginner concepts in the Pwned Labs introduction to AI hacking, then moves through direct prompt injection, indirect injection through documents and tool output, the exfiltration and action sinks that give an injection its effect, and finally the agent layer where a tool call chains into cloud access. Agents reached over the Model Context Protocol add their own surface, covered in the guide to MCP and AI agent security.
Practice has to be hands-on, because steering a live agent and following its output to an effect cannot be learned from slides. Pwned Labs runs each scenario in a live but isolated environment, so you can work prompt injection, indirect injection and tool abuse against a real application without touching production. Only ever test systems you own or have written permission to assess.
Practice this at Pwned Labs
The fastest way to understand AI red teaming against agents is to run it in a live environment. These Pwned Labs scenarios put you against a vulnerable AI application rather than a slide.
- Breach the Perimeter via Prompt Injection (Premium, Beginner): extract secrets from an AI assistant with prompt injection, then follow a leaked token into Azure storage.
- Exploit Indirect Prompt Injection for Azure Access (Premium, Beginner): plant instructions in content an AI assistant retrieves, then follow what it exposes into Azure and on to deployment credentials.
- The AI Systems Attack and Defense Bootcamp (AISRTP): the end-to-end program, from direct and indirect injection through retrieval poisoning, tool and agent abuse, and the defenses for each, assessed hands-on toward the AI Systems Red Team Professional certification.
Frequently asked questions
What is AI red teaming?
AI red teaming is authorized, adversarial testing of an AI system to find how it can be made to behave against its operator's intent and to prove the real impact of that behavior. It targets the whole application, including the model's prompts, the documents it retrieves, the tools an agent can call, and the systems that consume its output, not the model's mathematics in isolation.
Is AI red teaming the same as AI penetration testing?
The terms overlap and are often used for the same work. A practical split is scope: a penetration test is a bounded assessment of one application against a defined methodology, while red teaming is usually objective-driven, chasing a goal across whatever surface reaches it. Both exploit the application around the model to demonstrate impact rather than running a model benchmark.
How do you red team an AI agent?
Map the surface first: what untrusted text reaches the model, which tools it can call, which data sources it retrieves from, and the identity each tool runs as. Then plant an instruction where the agent will read it, trigger it through normal use, coerce a tool call with an attacker-chosen argument, and follow the result to an observable effect such as an unauthorized action or data returned from outside the user's reach.
What is excessive agency in an AI agent?
Excessive agency is when an agent has more functionality, permissions or autonomy than its task needs, so a single abused tool call reaches further than it should. It is LLM06 in the OWASP Top 10 for LLM Applications. The defense is least privilege on each tool, server-side enforcement of high-impact approvals, and a limit on the agent loop.
Do I need a machine learning background to do AI red teaming?
No. AI red teaming targets the application around the model, so web and API testing skills, comfort with HTTP and JSON, an intercepting proxy, and a general security foundation are enough to start. Pwned Labs assumes no prior AI security experience on its AI Systems Attack and Defense bootcamp, and its lab guides give every command in full.
How do the OWASP LLM Top 10 and MITRE ATLAS relate to AI red teaming?
The OWASP Top 10 for LLM Applications names the risk categories, such as prompt injection and excessive agency, and MITRE ATLAS catalogs the techniques seen against real AI systems, including agentic ones. Mapping each red team attempt to both lets a report show coverage against a shared standard rather than an ad hoc list.
Where can I practice AI red teaming hands-on?
Pwned Labs runs standalone AI security labs such as prompt injection against a live assistant and indirect injection for Azure access, and the AI Systems Attack and Defense Bootcamp, which covers all the agent attack families and is assessed hands-on toward the AI Systems Red Team Professional certification.
About Pwned Labs
Pwned Labs is a hands-on cloud and AI offensive security training platform. It runs real-world attack labs and bootcamps for AWS, Azure, GCP, Kubernetes and AI systems, including the ACRTP, MCRTP, GCRTP, KRTP and AISRTP certifications. Free labs are at pwnedlabs.io/explore.