Guardrails

Policy and technical limits on what an AI system can say or do, such as input and output filters, permission checks and rules for when to stop.

Definition

Guardrails sit around a model rather than inside it. Input guardrails screen requests for abuse, sensitive data or prompt injection; output guardrails check answers for policy breaches, personal data or unsupported claims; action guardrails limit which tools an agent may call and with what values.

They range from simple rules and allow-lists to classifier models and dedicated frameworks such as NVIDIA's NeMo Guardrails. No single guardrail is complete, so they are layered, tested with red teaming and monitored in production.

Sources

  1. NVIDIA NeMo Guardrails documentation