Safety & Red Teaming

AI safety and red teaming is the discipline of finding out how an AI system fails — through adversarial probing, toxicity and fairness evaluation, and data-integrity checks — before users, auditors, or attackers find out first. The theme spans two fronts: the model’s behavior under pressure, and the training data whose silent defects — leakage, bias, imbalance, drift, poisoning — produce failures no prompt-level test will catch. This page maps both fronts: what to read first, what each practice actually catches, and where the concepts get mistaken for one another.

Why safety and red teaming matter for engineers moving into AI

Traditional software fails loudly: an exception, a stack trace, a red build. Machine-learning systems fail statistically and in silence — a model ships with impressive accuracy that a skewed dataset made meaningless, an evaluation score inflated by leaked test data, an answer fabricated with complete confidence and no error signal anywhere. For a developer moving into AI, this theme replaces “does it pass the tests” with a harder question: what would make this system lie to me, and how would I know? Every topic below is one concrete answer to that question, with a measurable practice attached.

Start here: hallucination and data leakage, the foundations of AI safety testing

Two failure modes come before any attacker, and together they teach the mindset the rest of the theme depends on.

The first needs no adversary at all. Hallucination — a model generating plausible, confident, wrong output — is not a bug to patch but a structural property of how language models work, and how statistical next-token prediction creates confident falsehoods is the single best first read in this theme, because it explains why safety testing exists at all: the system’s default failure mode is one it cannot detect in itself. Once the mechanism is clear, the intrinsic vs. extrinsic, closed vs. open domain taxonomy gives you the vocabulary the detection literature assumes, and why zero-hallucination LLMs remain impossible sets the honest ceiling before you promise one to a stakeholder. When you are ready to act on it, the DeepEval, RAGAS, and RAG-grounding detection guide turns detection into a pipeline step.

The second foundation is subtler: the failure of your own measurement. Data leakage is information from outside the training set quietly contaminating the model, and how leakage inflates model accuracy shows why a spectacular offline score should be the first thing you distrust. Target, temporal, and preprocessing leakage breaks down why each pattern evades review, and at LLM scale the same failure goes public as benchmark contamination — how LiveCodeBench and LiveBench expose leaked evaluations.

These two establish the theme’s founding suspicion: trust neither the model’s confidence nor your own metrics. Everything in the next tier is a method for earning that trust back.

The core practice: red teaming the model and auditing the data behind it

This is the layer the theme is named for — deliberate attack from the outside, systematic audit from the inside. Both directions matter, because a model that survives adversarial prompts can still be quietly broken by its own training data.

Red teaming for AI is the attack direction: humans or automated probes hunting for jailbreaks, harmful outputs, and edge cases before deployment does it for you. How adversarial testing exposes model failures before deployment is the orientation read; OWASP LLM Top 10 and MITRE ATLAS supply the frameworks that keep probing systematic instead of anecdotal; and the Promptfoo, PyRIT, and Garak guide gets you a working red-team harness. Before you trust that harness alone, read why automated red teaming misses what humans catch — coverage gaps are the practice’s known blind spot.

The audit direction runs through three data threats every production team eventually meets. Class imbalance is the oldest: when one class vastly outnumbers another, accuracy stops meaning anything — why a 99% accurate model can be useless is the canonical demonstration, and the scikit-learn guide to class weighting, threshold moving, and SMOTE covers the standard remedies and their traps. Dataset bias is the harder cousin: systematic skews the model learns and amplifies — start with how skewed training data shapes model predictions, then selection, representation, and measurement bias to tell the three sources apart, because each needs a different fix. And data drift is the threat that arrives after launch: production data diverging from training data until the model quietly stops working — how production data silently degrades model accuracy explains the mechanism, and the Evidently, NannyML, and Alibi Detect monitoring guide makes watching for it operational.

With these four, you can attack the model and audit the data. The advanced tier makes both continuous, quantified, and resistant to a real adversary.

Advanced AI safety: guard models, fairness audits, provenance, and poisoning defenses

Everything above finds failures; this tier is what production systems run to keep finding them — and what they need when the failure was planted deliberately.

Toxicity and safety evaluation turns red-team findings into continuous measurement: guard models scoring every output against a harm taxonomy. How guard models score harmful AI outputs is the entry point, HarmBench, ToxiGen, and the MLCommons taxonomy covers the datasets and standards the scores rest on, and the Llama Guard, Perspective API, and promptfoo pipeline guide assembles a working stack. Balance it with false positives, dialect bias, and adversarial bypasses — automated moderation has hard limits, and knowing them is part of running it.

Bias and fairness metrics do the same for the bias work one tier down: they turn “the data looks skewed” into numbers a review board can act on. How fairness metrics detect discrimination in ML predictions introduces the instrument; demographic parity vs. equalized odds vs. calibration compares the core metrics; and the impossibility theorem explains why you must choose between them rather than satisfy all — a genuine design decision, not a checkbox. The AI Fairness 360, Fairlearn, and What-If Tool audit guide operationalizes the audit, and from COMPAS to the EU AI Act traces how these metrics became regulatory requirements.

Data versioning is the forensic backbone the other topics assume: when a model misbehaves, you need to reproduce exactly which data trained it. How content hashing tracks dataset changes like git maps the concept onto version control you already know, and the DVC and lakeFS setup guide builds the reproducibility layer.

The tier closes where the adversary gets serious. Data poisoning is an attacker manipulating your training data itself — how corrupted training data compromises model behavior covers the threat model, backdoor triggers, clean-label, and label-flipping explains why poisoned samples pass human review, and RAG poisoning and agent-memory corruption extends the attack surface to systems that never retrain. The ART, data provenance, and ML-BOM defense guide covers the countermeasures, and from Nightshade to constant-sample attacks documents the real incidents that moved poisoning from theory to threat model.

How the data threats differ: leakage vs. imbalance vs. bias vs. drift vs. poisoning

Five data-borne threats sound alike in an incident review and get conflated constantly. The distinctions that matter for diagnosis:

Three more distinctions sit on the behavioral side and trip teams just as often:

• Red teaming vs. safety evaluation. Red teaming is exploration — probing for failures nobody has named yet. Safety evaluation is regression testing — scoring outputs against known harm categories, continuously. A team that runs only benchmarks never finds a novel jailbreak; a team that only red-teams has no baseline to tell whether the model got safer.
• Dataset bias vs. fairness metrics. One is the condition, the other is the instrument. Dataset bias lives in the data before training; fairness metrics measure its effect in the model’s predictions. Fixing the metric without touching the data treats the symptom.
• Hallucination vs. data poisoning. Both produce a model that confidently says wrong things — but hallucination is intrinsic, a statistical property with no attacker, while a poisoned model fails by someone’s design. The distinction decides your response: mitigation and grounding for one, incident response and data provenance for the other.

Common questions

Q: Where should I start with AI safety testing as a software developer? A: With the two foundations: hallucination and data leakage. They require no security background and recalibrate how you read every metric afterward. How next-token prediction creates confident falsehoods is the first read; the leakage explainer is the second. Then work through the core tier in order.

Q: Do I need red teaming if I already run automated safety evaluations? A: Yes — they answer different questions. Benchmarks and guard models score outputs against known harm categories; red teaming hunts for failures nobody has categorized yet. Frontier labs run both, which is how red teaming became industry standard by 2026. Evaluation without probing leaves novel jailbreaks for your users to find.

Q: My model passed offline evaluation but degrades in production — which threat am I looking at? A: Check leakage first: if offline scores were inflated by contaminated splits, production is just the truth arriving. If the model genuinely worked and then decayed, it is drift — covariate shift, concept drift, and label drift gives you the statistics to tell which kind, which decides the retraining response.

Q: My model has high accuracy but fails one user group — is that class imbalance or dataset bias? A: Diagnose by slicing: imbalance shows up as poor recall on a rare class regardless of who it belongs to; bias shows up as performance gaps between groups at comparable base rates. Confusion matrices and PR-AUC are the tools for the first check — and both threats can be present at once.

Q: Which safety checks belong in CI, and which in production monitoring? A: Leakage detection and safety evaluations are pre-deployment gates — scikit-learn pipelines and Deepchecks run in CI. Drift is a production-only signal, so monitoring lives there. Red teaming fits neither loop cleanly: run it before major releases and after significant model or prompt changes.

Q: How do I trace a suspected poisoning incident back to its source? A: Through lineage — you can only investigate data you can reproduce. Storage backends, data lineage, and hashing covers the versioning components that make the question answerable; without them, a poisoning suspicion ends at “we can’t know which data trained this model.”