Red teaming

Deliberately attacking an AI system, as an adversary would, to find harmful outputs, security holes and failure modes before real users or attackers do.

Definition

AI red teams probe systems with jailbreaks, prompt injection, misleading inputs, sensitive topics and misuse scenarios. The practice comes from security testing and has been adopted by AI developers and regulators; the EU AI Act, for example, expects adversarial testing of general-purpose models with systemic risk.

For companies deploying AI, red teaming focuses on their own use case: what could a customer, employee or outside attacker make this assistant or agent do, and what would it cost the business?

Sources

  1. Microsoft, Planning red teaming for large language models and their applications

Related terms