Reproducibility budgets for ML preprints

Abstract

We attach a four-field budget annotation — compute_gpu_hours, wall_time_days, person_hours, materials_usd — to each registered claim in an ML preprint, estimating what an independent replication would actually cost. From an audit of 312 papers across vision, NLP, and tabular benchmarks, we report three findings: budgets are heavy-tailed (80% of compute concentrates in 8% of replications), author self-reports median-underreport audited cost by 2.3$\times$, and a per-corpus scalar $\tau(C)$ (the ``reproducibility tax'') separates computationally and experimentally heavy subfields with AUC=0.91. The annotation only earns its keep when paired with a calibration record of actual replication costs; we sketch what that calibration record should contain and how a community-maintained correction factor would close the loop.