AI red teaming in plain words
Red teaming means attacking a system the way a real adversary would. AI red teaming does the same for AI systems. Testers try to make a model or an AI-powered app leak data, ignore its rules, produce harmful output or take actions it should not.
There are two kinds. Model red teaming, done by AI labs, tests the model itself for dangerous skills. Application red teaming is what most businesses need. It tests your product: the prompts, the data it can reach, the tools it can call and the people who can talk to it.
Why normal pentests miss AI risks
A large language model (LLM, the kind of AI behind chatbots) takes instructions in plain text. It cannot reliably tell your instructions apart from an attacker's. That creates a new way in for attackers. A web pentest checks your login and your APIs (the connections other software uses to talk to yours). It does not check whether a hidden sentence in a customer's uploaded PDF can make your support bot email out another customer's data.
What we test: the OWASP Top 10 for LLM Applications
OWASP, a nonprofit security group, publishes the Top 10 for LLM Applications. It is the most widely used checklist for these risks. A test covers these main areas:
- Prompt injection: tricking the AI with crafted text. It can be direct ("ignore your instructions") or indirect, where instructions hide in web pages, emails or files the AI reads.
- Sensitive information disclosure: leaking customer data, internal documents or secrets.
- System prompt leakage: exposing the system prompt (the hidden instructions you give the AI) and any keys or logic inside it.
- Excessive agency: AI agents (AI that can take actions) that send emails, issue refunds or change records with too little control.
- Improper output handling: AI output that ends up running as code in your app. Examples are XSS (harmful script in a web page) or SQL (database commands).
- Vector and embedding weaknesses: RAG systems (AI that looks up your own documents to answer) that return documents a user should not see.
- Unbounded consumption: abuse that runs up AI costs or takes the feature offline.
We also map findings to MITRE ATLAS, a knowledge base of real attacks on AI systems.
How an AI red teaming engagement works
- Map the system: which model, which prompts, what data it can read, what tools it can use and who can reach it.
- Threat model: decide what would hurt most. Examples are data leaks between customers, fraud through agent actions or harm to your brand.
- Attack: hands-on testing with custom prompts and payloads (attack inputs). This is backed by automated prompt fuzzing, which sends thousands of varied prompts.
- Report: each finding with a working example, impact and a fix, mapped to OWASP and MITRE ATLAS.
- Retest after you apply the fixes.
Common fixes
- Give the AI the least access it needs, and enforce permissions in your own code, not in the prompt
- Require a human to confirm risky actions such as payments, deletions or outbound emails
- Treat everything the model reads (files, web pages, emails) as untrusted input
- Check and clean model output before it reaches a browser, database or shell
- Set usage limits and spending caps on AI services
- Keep secrets out of system prompts
When to test
Test before launch. Test again whenever you change the model, the system prompt, the tools an agent can use or the data it can reach. Those changes can open new holes even when your code looks the same. Large customers now often ask about AI security in their reviews. So a recent test report also helps sales.
Need a hand with this?
Talk to a senior consultant. We reply within one business day with clear next steps and a scoped quote.
Written by
NextGarrison Security TeamPenetration testers, SOC analysts and compliance consultants
Our consultants test applications and networks, run 24/7 security monitoring and prepare companies for SOC 2, ISO 27001 and PCI DSS assessments. We write about what we see in real engagements, in plain English.