What Is AI Red Teaming?

How AI Red Teaming Differs from Traditional Red Teaming

Traditional red teaming targets infrastructure (networks, endpoints, identity systems, cloud environments) using a well-understood kill chain. AI red teaming adds an entirely new attack surface: the model itself and the application layer wrapping it. An LLM-powered chatbot deployed in production isn't just a web application; it's a system that interprets natural language as instructions, often with access to internal APIs, databases, and cloud services via tool use or function calling.

This creates a class of vulnerabilities that doesn't exist in traditional infrastructure. Prompt injection, where adversarial input causes the model to deviate from its intended behavior, has no direct analog in conventional security. You can't firewall against it. You can't patch it with a CVE. It exploits the fundamental mechanism by which the system operates: interpreting language.

Key Attack Vectors

The OWASP Top 10 for Large Language Model Applications (updated in 2025) catalogues the most critical risks. The vectors AI red teamers focus on include:

Prompt Injection (Direct and Indirect): Direct injection involves crafting inputs that override system instructions, convincing a model to ignore its safety guidelines or execute unintended actions. Indirect injection embeds malicious instructions in data the model processes: a poisoned document in a RAG pipeline, a malicious web page summarized by an AI agent, or adversarial text hidden in an email the AI triages. Indirect injection is particularly dangerous because the attack payload is separated from the user interaction. Data Poisoning: Manipulating training data or fine-tuning datasets to embed backdoors or biases. An attacker who compromises a training data pipeline can cause a model to behave normally on most inputs but produce attacker-controlled outputs on specific trigger phrases. Model Extraction and Inference: Systematically querying a model to reconstruct its weights, training data, or decision boundaries. This enables attackers to build surrogate models for offline attack development or extract proprietary data memorized during training. Tool and Agent Exploitation: Modern AI systems often have access to external tools: code execution, web browsing, database queries, API calls. Manipulating the model into misusing these tools can result in server-side request forgery, unauthorized data access, or code execution in the cloud environment hosting the AI system.

Real-World AI Attacks

These documented incidents show the attack classes in practice.

Bing Chat system prompt leak, 2023

Shortly after launch, researchers used prompt injection to make Microsoft's Bing Chat reveal its hidden system prompt and internal codename, demonstrating how easily crafted input overrides intended instructions.

Dealership chatbot manipulation, 2023

Users manipulated a car dealership's LLM-powered support chatbot into making absurd commitments and answering off-topic requests, a public illustration of what happens when a model with a business role has no guardrails around instruction following.

Indirect prompt injection exfiltration

Security researchers repeatedly demonstrated data exfiltration from AI assistants by planting instructions in content the model later processed, for example hiding a payload in a web page or document that caused the assistant to leak conversation data through a rendered image request. This is the technique that turns a chatbot into a cloud data exfiltration vector.

The Cloud Security Connection

AI red teaming and cloud security are deeply intertwined because LLMs run in cloud environments. An LLM agent with an IAM role that permits S3 access, a function-calling interface to internal APIs, and a RAG pipeline that indexes sensitive documents creates an attack surface that spans both AI and cloud security domains. Prompt injection against such a system is not only an AI safety issue, it is a cloud data exfiltration vector.

The 2024-2025 wave of AI agent frameworks (LangChain, AutoGPT, CrewAI, Amazon Bedrock Agents) dramatically expanded this intersection. These agents execute multi-step plans in cloud environments, often with permissions that would trigger alerts if a human user exercised them. Red teaming these systems requires understanding both the AI manipulation techniques and the cloud infrastructure they operate on.

Emerging Standards

NIST's AI Risk Management Framework (AI RMF) and the EU AI Act both reference adversarial testing as a core governance requirement. The MITRE ATLAS framework (Adversarial Threat Landscape for Artificial-Intelligence Systems) extends the MITRE ATT&CK model to cover AI-specific tactics and techniques, providing a structured taxonomy for AI red team findings. Microsoft, Google, and Anthropic have all published AI red teaming methodologies, and the discipline is rapidly moving from ad-hoc exercises to structured, repeatable assessment frameworks.

Related Labs

Pwned Labs covers AI and agentic security hands-on, both in dedicated labs and across the bootcamp curriculum, where you attack AI systems running in real cloud environments and chain prompt injection into cloud resource access. Browse the labs at pwnedlabs.io/explore.

AI Security at Pwned Labs

Pwned Labs addresses the AI-cloud security intersection through its PromptStorm cyber range, where practitioners attack AI systems deployed in real cloud environments, exploiting prompt injection to access cloud resources, pivoting through agent frameworks, and testing RAG pipeline security. As AI integrations become standard components of cloud architecture, Pwned Labs continues expanding its AI security content across all bootcamp tracks. Explore current AI security labs at pwnedlabs.io.

Frequently Asked Questions

What is AI red teaming?

The practice of adversarially testing AI systems, especially large language models and the applications built on them, to find failures such as prompt injection, jailbreaks, data leakage, and unsafe tool use before attackers do.

How is it different from traditional red teaming?

Traditional red teaming targets infrastructure with a known kill chain. AI red teaming targets model behavior, where the attack surface is natural-language input and the tools and permissions the model can reach, which have no direct analog in conventional security.

What is prompt injection?

Crafting input that overrides a model's intended instructions. Direct injection manipulates the prompt itself; indirect injection hides instructions in external data the model later processes, such as a document, email, or web page.

Why is AI red teaming a cloud security concern?

Because AI systems increasingly run in cloud accounts with tool access and IAM permissions. A successful injection against an agent can turn into cloud data access or resource abuse, using the agent's own permissions.

What standards guide AI red teaming?

The OWASP Top 10 for LLM Applications, MITRE ATLAS, and the NIST AI Risk Management Framework are common references for categorizing and testing AI-specific risks.