What is prompt injection?
Prompt injection is an attack where someone feeds text to a large language model (LLM) that makes it ignore its original instructions. The model then follows the attacker's instructions instead. An LLM is the kind of AI that powers chatbots and writing assistants. The attack works because an LLM cannot reliably tell the developer's instructions apart from the data it is reading. Everything arrives as text in the same context window, the block of text the model sees at once.
OWASP is a nonprofit that publishes free security guides. Its Top 10 for LLM Applications ranked prompt injection as the top risk in both its 2023 and 2025 editions. The harm depends on what the model can do. A chatbot that can only reply with text might be tricked into saying something embarrassing. An AI agent is an AI that can take actions, for example with access to email, code or a database. It can be tricked into leaking data or taking actions for the attacker.
Direct vs indirect prompt injection
Direct prompt injection is when the person typing into the app is the attacker. They write instructions meant to override the system prompt, the hidden instructions the developer gave the model. They may try to pull out those instructions or get around limits. Jailbreaking is related, but it targets the model's safety training rather than the app's instructions.
Indirect prompt injection is when harmful instructions arrive through content the model reads for someone else. That could be a web page, an email, a document, a code repository or a tool result. The victim is the real user, who may never see the attacker's text. This is usually the more dangerous form, because it scales and it targets systems people trust.
| Direct | Indirect | |
|---|---|---|
| Who is the attacker | The user of the application | A third party who controls content the model reads |
| Where the payload sits | In the chat input | In web pages, emails, files, tickets, repositories or retrieved documents |
| Who gets hurt | Usually the application owner | The user and the application owner |
| Main risk | Leaked system prompts, policy bypass, misuse of tools available to that user | Data theft and unwanted actions performed with the victim's permissions |
Prompt injection examples
These examples are described in words, not as working attacks. They follow patterns researchers have shown in public.
1. The customer support bot (direct). A retailer's support chatbot can look up orders and issue refunds up to a limit. A user claims to be a store manager running a test. They say the earlier rules no longer apply and ask for a full refund on an order that is not theirs. If the refund tool only checks what the model asks for, and not who the user is, the bot may comply. The real flaw is not the wording of the system prompt. It is that the refund tool trusted the model to check permissions.
2. The email assistant (indirect). An AI assistant can read a user's inbox, sum up messages and send replies. An attacker sends the user an email with hidden text, for example white text on a white background. The text tells the assistant to find password reset emails and forward them to an outside address. Later the user asks for a summary of today's mail. The assistant reads the attacker's email and may follow it.
3. The coding agent and the malicious README (indirect). A developer asks a coding agent to set up an open-source project. The project's README or a config file holds instructions aimed at AI agents. They tell the agent to run a setup command that secretly sends environment variables and API keys to an attacker's server. API keys are passwords that let programs talk to each other. An agent that can run shell commands without approval may do it. We cover this risk in more depth in AI coding agent vulnerabilities.
4. RAG over a poisoned document (indirect). A company's internal assistant answers questions using retrieval-augmented generation (RAG). That means it looks up documents on a shared drive or wiki, then answers from them. Someone with write access to one page adds hidden instructions. So might a vendor whose document was uploaded. The instructions tell the model to suggest a different bank account for supplier payments. Or they tell it to add a phishing link (a link to a fake login page) to every answer about benefits. Anyone whose question pulls up that document gets the poisoned answer, from a tool they trust.
Why prompt injection is hard to fully prevent
Classic injection attacks such as SQL injection have a clean fix. You keep code and data apart with parameterized queries. LLMs have no reliable equivalent today. To the model, the system prompt, the user's message and every retrieved document are all just tokens (chunks of text). Its skill at following plain-language instructions is the very feature you are paying for.
- Filters can be bypassed. Attackers rephrase, translate or encode their text. They split instructions across sources or hide them in images and markup. A blocklist of phrases will always miss something.
- Models are probabilistic. Their answers vary. An attack that fails once may work on a retry with slightly different wording.
- Detection models can be fooled too. Using a second model to spot injection helps, but it shares the same weakness.
- Agents multiply the impact. The more tools, data and freedom you give a model, the more an injected instruction can do.
Layered defenses against prompt injection attacks
No single control is enough. Use several layers that limit the damage when one fails.
| Layer | What it means in practice |
|---|---|
| Least privilege for tools | Give the model only the tools and data it needs for the task. Prefer read-only access, scope API tokens narrowly, and enforce authorization inside each tool based on the real user, never on what the model says. |
| Human confirmation for risky actions | Require the user to approve sending email, making payments, deleting data, running shell commands or changing permissions. Show exactly what will happen, not a summary written by the model. |
| Treat retrieved content as untrusted | Clearly separate external content in the prompt, restrict who can write to RAG sources, and avoid mixing untrusted content, sensitive data and outbound actions in the same session. |
| Output handling | Treat model output like user input. Encode it before rendering, block or sanitize links and images that could carry data to outside servers, and never pass it straight into a shell, SQL query or code evaluation. |
| Input and output screening | Use classifiers or rules to flag known injection patterns and sensitive data in responses. Useful as a tripwire, not a complete defense. |
| Monitoring and logging | Log prompts, retrieved sources, tool calls and outputs. Alert on unusual tool use, such as bulk data access or new outbound destinations, and keep a way to switch an agent off quickly. |
A helpful rule of thumb comes from security researcher Simon Willison. He calls it the "lethal trifecta". Be most careful when one agent has all three: access to private data, exposure to untrusted content and a way to send data out. Remove any one of the three and most data theft scenarios fall apart.
How to test for prompt injection
Test the whole app, not just the model. A useful test plan covers:
- Map the attack surface. List every source of text the model reads, such as user input, files, web pages, emails, tickets, RAG sources and tool outputs. Then list every tool or action it can take.
- Test direct injection. Try to extract the system prompt, override instructions, change the model's role and reach tools or data the user should not have.
- Test indirect injection. Plant instructions in each content source the model reads. Check whether they change its behavior when a normal user triggers it.
- Test data exfiltration. Check whether injected instructions can make the model leak data through links, rendered images, tool calls or outbound requests.
- Test authorization at the tool layer. Confirm that tools enforce the real user's permissions, even when the model asks for more.
- Check confirmation flows. See whether risky actions can run without approval, or whether the approval prompt itself can be tricked.
- Repeat with variations. Results vary from run to run. So run each test several times with different wording, languages and encodings.
Open-source tools such as garak, PyRIT and promptfoo run libraries of known attacks. They are useful for regression testing, which means repeat testing after changes. Manual testing is still needed to understand your own tools, data and business logic. For the broader picture, including harmful content and misuse, see what is AI red teaming.
How NextGarrison helps
NextGarrison's AI and LLM penetration testing covers direct and indirect prompt injection and data leak paths. It also covers tool and agent permissions, RAG data sources and output handling. We show which attacks succeed and what they could reach. Then we give your engineers specific fixes, from tool permissions to confirmation flows.
Need a hand with this?
Talk to a senior consultant. We reply within one business day with clear next steps and a scoped quote.
Written by
NextGarrison Security TeamPenetration testers, SOC analysts and compliance consultants
Our consultants test applications and networks, run 24/7 security monitoring and prepare companies for SOC 2, ISO 27001 and PCI DSS assessments. We write about what we see in real engagements, in plain English.