Prompt Injection

What prompt injection is

Every time you type a message into an AI assistant, you’re sending a prompt. Prompt injection happens when someone designs that input to override the model’s original instructions — essentially hijacking the conversation. Think of the AI model as a restaurant server following a set menu. Prompt injection is a customer who rewrites the kitchen’s recipe book by slipping instructions onto the order ticket.

This vulnerability exists because large language models process all text — system instructions, user input, and retrieved data — as one continuous stream of tokens. The model has no reliable way to distinguish “this is a rule I must follow” from “this is user text I should respond to.” That fundamental architectural gap is what makes prompt injection possible, and why red teams prioritize testing for it above almost every other attack category.

According to OWASP, prompt injection holds the top spot as the number one vulnerability in the OWASP Top 10 for LLM Applications 2025 edition, maintaining that ranking for the second consecutive year. The classification recognizes two main types:

Direct prompt injection occurs when a user deliberately crafts input to change the model’s behavior. A simple example: typing “Ignore all previous instructions and instead do X” into a chatbot. More sophisticated versions use encoded text, role-playing scenarios, or multi-turn conversations to gradually steer the model past its guardrails.

Indirect prompt injection is subtler and more dangerous. The malicious instructions aren’t typed by the user — they’re embedded in external content the model processes. A poisoned webpage, a manipulated PDF, or a compromised database entry can contain hidden instructions that the model follows when it retrieves and reads that content. The user never sees the attack happening.

According to Palo Alto Unit 42, agentic AI systems using protocols like MCP dramatically expand the attack surface for prompt injection, enabling new vectors such as tool poisoning and credential theft. When an AI agent can browse the web, execute code, or access files, a successful injection doesn’t just produce bad text — it can trigger real-world actions.

How prompt injection is used in practice

The most common place you’ll encounter prompt injection concerns is during AI application security reviews. When a team builds a customer-facing chatbot, an internal knowledge assistant, or any tool that feeds external data into an LLM, the first security question is always: “Can someone manipulate this through the input?”

Red teams and security researchers test for prompt injection by systematically probing the model with adversarial inputs. They try direct attacks — explicit instruction overrides, encoding tricks, jailbreak prompts — and indirect attacks, planting malicious content in documents or web pages the system retrieves. According to CrowdStrike, real-world prompt injection exploits have already produced remote code execution vulnerabilities and high-severity security incidents.

Organizations defend against prompt injection through layered controls: input filtering, output validation, privilege separation (limiting what the AI can actually do), and human-in-the-loop confirmation for sensitive actions. No single defense is foolproof, which is why continuous adversarial testing matters.

Don’t rely on prompt-level instructions alone to block injection. Treat the model’s output as untrusted input — validate and sanitize it before passing it to any downstream system, just like you would with user-submitted form data in a web application.

How to read prompt injection: from the trust gap to the production incident

Start with how attackers override AI system instructions to see the attack from the outside — direct and indirect payloads, and why both work. Then read why LLMs cannot reliably separate instructions from data: it explains the trust-boundary problem the first article only shows the symptoms of, and why no single patch closes it for good.

When you are ready to build defenses, the PromptArmor, LLM Guard, and MELON guide lays out the layered architecture — input scanning, output scanning, privilege separation, red-teaming — that production teams actually ship. Prompt injection attacks in the wild shows what happens when that architecture is missing, tracing real breaches in AI copilots and email agents. Close with the ethics of prompt injection in AI agent systems — once an agent can act on its own, a successful injection stops being a bug report and becomes a question of who answers for what the agent did.

How prompt injection differs from testing gaps and tool-use risk

Two neighbouring practices get confused with injection defense, and each confusion sends a team’s effort in the wrong direction.

A passing eval suite is not a security clearance. Prompt testing and evaluation checks a prompt against a fixed set of known inputs; it proves nothing about a payload nobody wrote into that set. Catching an input nobody anticipated is what dedicated red-teaming — not the evaluation pipeline — is built for.

Prompt injection is also not tool use, though the two compound each other. Injection hijacks what a model is told to do; tool use is the surface that turns a successful hijack into a real action — sending an email, calling an API, moving a record. Treat them as separable failure points: one line of defense keeps the model listening to the right instructions, a second, independent one keeps whatever it decides to do inside a bounded set of privileges.

When prompt injection fits — and when it doesn't

Common questions about prompt injection

Q: Can I stop prompt injection just by telling the model, in the system prompt, to ignore any instructions found in user input? A: No — that instruction is just more text sharing the same channel the attacker’s payload uses. LLMs process system prompts and user data through one flat stream with no structural boundary between them, so a firmly worded system prompt raises the bar slightly but does not close the gap.

Q: Will my prompt evaluation suite catch a prompt injection attack before it ships? A: Not reliably. Prompt testing and evaluation checks a prompt against a fixed test set of known inputs, while a real attacker writes a payload nobody added to that set. The PromptArmor and LLM Guard defense guide calls for dedicated red-teaming with tools like Garak and PyRIT precisely because standard evaluation was never built to generate adversarial input.

Q: Why do prompt injection defenses that work in benchmarks still fail once an agent ships to production? A: Benchmark suites like AgentDojo test known attack patterns under controlled conditions; production traffic serves untrusted content assembled by real users and real websites, recombined in ways no benchmark enumerates. Real-world breaches in AI copilots and email agents confirm research-grade defenses already work at benchmark scale — the open gap is deployment, not detection.

Q: If I add a vendor’s built-in injection filter, does that remove my liability when it fails? A: No. The ethics of prompt injection in agent systems argues responsibility sits with the organization that deploys the AI, the same way an employer answers for an employee’s conduct — a vendor’s scanner reduces risk, it does not transfer accountability for what your system does when it fails.

Part of the prompt ops and security theme · closest neighbour: tool use in prompts.

Sources

• OWASP: LLM01:2025 Prompt Injection - Definitive classification of prompt injection as the top LLM vulnerability with attack taxonomy and mitigation strategies
• CrowdStrike: Prompt Injection: Definition and Attack Taxonomy - Security-focused analysis of prompt injection types and real-world incidents