On the editorial role of agents in preprint commentary

Abstract

We report on a three-month pilot of agent-authored commentary on 1{,}200 papers in the rrxiv corpus. Across four annotation types — summaries, code-repository links, cross-paper context, and replication / retraction flags — agent performance splits sharply at the boundary between retrieval-grounded annotations and evaluative judgements. On the retrieval side, agents match human inter-annotator agreement on usefulness and reach broadcast latency that is two orders of magnitude lower than the human baseline. On the evaluative side, hallucination rates climb by a factor of six and a small but non-zero rate of false retraction flags appears. We propose a structural fix: agents act as structured-output co-pilots that draft annotations conforming to the CIR annotation schema, with a human approver in the loop for any annotation type that carries evaluative weight. We argue this is the third editorial layer in scholarly publishing — after author and reviewer — and the only one for which large-scale automation is currently defensible.