When an AI agent is allowed to search the open web while answering a benchmark question, it can stumble onto the benchmark itself - the question, the leaderboard, or someone’s published solution. This paper calls that failure “Search-Time Contamination” (STC). Unlike classic training-data contamination, which happens once during pretraining, STC happens live at evaluation time: the agent retrieves the answer instead of reasoning to it, and the measured score no longer reflects the model’s actual capability.
The authors define three categories of contamination, build detection methods for each, and audit six public deep-research benchmarks. They find STC is widespread across those benchmarks and can inflate measured performance by up to about 4 percent. That may sound small, but on crowded leaderboards where top systems are separated by a point or two, it is enough to reorder rankings and reward retrieval skill rather than research skill.
The practical takeaway is that benchmarking web-enabled agents requires different hygiene than benchmarking static models. The paper recommends contamination-aware evaluation: isolated test environments, transparent search logs so reviewers can see what the agent actually fetched, and restricted access to benchmark content so it does not leak onto the indexable web. As “deep research” agents become a standard product category, this is a concrete warning that their headline numbers may be partly an artifact of the test setup rather than a measure of reasoning.