On November 13, 2026, Anthropic’s engineering team published guidance on building safety evaluations, which it defines as measuring a model’s propensity to cause harm rather than its capabilities. The post argues that a safety claim is only as sound as the eval behind it, and sets out recommendations: define the construct before collecting data, use multi-turn simulations because risk can build across a conversation, validate model graders against human labels with chance-corrected agreement such as Cohen’s kappa, preregister metrics and thresholds, use at least 100 items for complex multi-turn evals and 1,000 for simpler single-turn ones, and report uncertainty.
It backs the advice with its own numbers. Harmless response rates on single-turn API evals run 94% to 98% for every recent model, which Anthropic calls saturated; such suites span 16 policy areas in seven languages and now serve mainly as regression checks. Configuration matters: for Claude Fable 5.1 the gap between API and claude.ai appropriate response rates on multi-turn safety evals was as high as 34 percentage points, and a suicide and self-harm eval moved from 60% on the API to 94% on claude.ai after a system prompt instruction was added. On a romance scam multi-turn eval, Claude Opus 5 scored 98% and Opus 5.5 scored 94%, a difference that was not statistically significant, and multi-turn confidence intervals on the API can exceed plus or minus 10 points.
Why it matters: Published safety scores are increasingly used in deployment decisions and regulation. A frontier lab stating openly that its standard single-turn tests no longer discriminate, and that system prompts can swing results by tens of points, is a useful warning about comparing headline numbers.
What it does not show: This is guidance with illustrative figures, not a full results release, and the evals themselves are not published. Outsiders cannot check how representative the examples are.