Safety Evals Should Project Test-Time Compute

Abstract

Safety evaluations often measure model behavior under a fixed, limited inference budget, while real adversaries can spend more effort through repeated attempts, search, tools, long-context prompting, and agentic scaffolding. This article argues that safety claims should account for test-time compute and describe how harmful success changes with adversarial effort. It frames evaluation as a risk surface over budgets, deployment configurations, scaffolds, and attacker strategies, and highlights the economic gap between what evaluators can afford to test and what attackers may find worthwhile. A minimal evaluation protocol measures multiple effort tiers, tests different attacker types, reports uncertainty and cost, and separates observed results from higher-budget projections. Static safety checks remain useful, but release decisions should rely on scoped, budget-labeled claims rather than generalizing from a single low-budget score.