Most evaluations of AI “scientists” are static: pose a question, grade the final answer. InquiTree argues this conflates genuine reasoning with memorization, because a single answer can be produced by recalling a known result rather than by actually reasoning through a study. Instead it models scientific inquiry as interactive Research Trees - directed acyclic graphs that capture the logical dependencies across hypothesis formulation, study design, result interpretation, and belief updating - so an agent has to navigate the whole loop, not just emit a conclusion.
The author derives these trees from real papers, evaluates agents on 30 of them, and releases an open-access subset called IT-18. Three findings stand out. First, agents show “cognitive tunneling”: over extended interactions their critical judgment and anomaly detection degrade relative to their baseline. Second, there is a large interpolation-versus-extrapolation gap - performance drops sharply on papers published after the model’s training cutoff, suggesting apparent competence leans on parametric memory rather than reasoning. Third, simply extending context length does not fix this; stronger architectures or human oversight appear necessary.
The result matters because “AI scientist” systems are being promoted as engines of autonomous discovery. InquiTree is a reminder that doing well on a one-shot question is not the same as conducting sound inquiry, and that the failure modes that matter - drift, over-reliance on memorized results, degrading vigilance over long sessions - only become visible when you test the full loop instead of the final answer.