AI red teaming simulates attacks specifically against artificial intelligence systems, such as assistants, chatbots, agents, and applications that query knowledge bases, to discover whether someone can make them leak data, ignore their own rules, or perform unauthorized actions. It is penetration testing of the model's behavior, not just its surrounding infrastructure.
A traditional penetration test checks the application, API, and server. AI red teaming tests what the system says and does when it receives text designed to deceive it.
Why language models need their own testing
A traditional system executes what its code specifies. A language model responds to what incoming text appears to request and has difficulty distinguishing text from a legitimate user from text in a document, email, or web page. This creates an attack surface based on natural language.
Real-world cases already exist. In November 2025, Anthropic reported an espionage campaign in which attackers manipulated an AI model to execute 80% to 90% of tactical operations, and the Global Cybersecurity Outlook 2026, from the World Economic Forum, reports that 87% of leaders have seen an increase in AI-related vulnerabilities.
The most common risk categories are cataloged in OWASP Top 10 for LLM Applications.
What is attacked
In direct injection, the user tries to persuade the model to ignore its rules through role-playing, fragmented requests, or formatting that confuses the filter. In indirect injection, the malicious instruction is hidden in content the model reads, without the legitimate user even noticing.
The test also looks for leakage through context, when the system reveals excerpts from the knowledge base, internal instructions, or other users' data. For agents that send emails or query databases, it checks for tool abuse. It also assesses context poisoning, where manipulated content is inserted into the knowledge base being queried.
Fictional example: the virtual assistant with too much access
An insurer launches a virtual assistant that queries an internal document repository. The repository was assembled from the customer service shared folder, which includes product manuals and spreadsheets containing terms negotiated with brokers.
In the test, direct questions about the terms are refused. An indirect question asking for a sample proposal for a large brokerage prompts the assistant to use the actual percentages from the spreadsheet. In the same round, a user-uploaded document containing a hidden instruction makes the system ignore its topic restriction.
When to perform the test
When AI red teaming is needed:
- Before production: every AI system that accesses internal data or performs actions.
- With every significant change: a new model, rewritten instructions, an expanded knowledge base, or a newly connected tool.
- Periodically: because attack techniques against models evolve quickly.
The model provider tests the generic model, but not the application with each company's data, instructions, and tools. Input filters help but can be bypassed; the strongest protection comes from architecture that limits what the system can access and do.
How Vultus conducts AI red teaming
AI red teaming is one of the four pillars of the methodology for generative AI security from Vultus. The work starts with the system's context and threat modeling based on real scenarios, and attacks are conducted under controlled conditions in production or an equivalent staging environment. Deliverables include an executive report, a prioritized technical backlog, and a non-destructive demonstration of each flaw. Talk to Vultus about your company's AI systems.
