I Checked the Arithmetic in 1,660 Economics Papers

Casey Handmer wrote this month that automated review of the scientific literature should cost about ten cents a paper, and that someone should point it at the most cited work first. I replied, “Entire literatures consistently measuring the wrong thing.” Then I wanted to know what a machine could check in economics today, so I built the check and ran it on NBER working papers.

The full report has every number below, every flagged table row with its page, and the method.

What was read

The index covers 35,897 working papers from 1973 to 2026. The full text was read for 1,660 of them: the most cited one percent of the series, the thirty most cited papers of each year since 2000, and thirty drawn at random from each of those years. From their tables the program redid the arithmetic on 137,326 printed results.

Stars the standard error cannot support

A regression table prints a coefficient, a standard error under it, and stars that claim a level of statistical significance. The coefficient divided by the standard error has to be large enough for the stars. That is a division, and it can be checked without judgment.

In 241 papers the tables say what the parentheses hold and what the stars mean. In 25 of them at least one star is not supported by the numbers printed beside it. There are 40 such stars, and I confirmed each one on the rendered page of the paper. Among the randomly drawn papers the rate is 13 of 96.

Most of these look like slips. One well known table prints a coefficient of 0.0125 with a star over a standard error of 0.072, next to a neighbor of 0.126. The rate is about the same for heavily cited papers and for the most prolific authors. I found the published journal version of six flagged papers. In four the star was still there.

The result a paper leads with

Deliberate tuning would leave a different mark: results that land just past the 5% line more often than just short of it.

I compared the first row of the first regression table in each paper with every other row in the same papers. Control variables have no reason to be pushed over a threshold, so they serve as the baseline. Close to the 5% line, 140 leading results sit just above it and 102 just below, which is 57.9%. For the other rows the share is 48.2%. The gap is 9.6 points, with a 95% interval from 2.9 to 16.6. At nine other cutoffs that carry no star, leading results show nothing like it.

One literature, read paper by paper

To test my own reply I took the retirement and Social Security literature, 607 papers, and drew 200 at random. A language model recorded how each paper defines its outcome, with a quotation that had to be found in the paper. A second model read 40 of them blind and agreed on the definition in 12 of 14.

Seventy of the papers study retirement itself. No single way of deciding who is retired holds a majority. The most common, having no job or no earnings, covers 29% of the papers that pick one definition. Others count a person as retired on leaving a career job, on leaving the labor force, on claiming benefits, or on saying so. Ten papers use more than one definition and five never give one. A person who claims Social Security and keeps a part time job is retired in some of these papers and working in others, so seventy papers about retirement are not about one variable.

What did not show up

I looked for citation rings inside the series and did not find them. The share of citations that go to a paper’s own authors has fallen in every decade, from 31.5% in the 1980s to 15.9% in the 2020s. I also tested the last digits of printed standard errors, which come out even when software writes them and uneven when a person types them. No paper stood out.

The model reading cost about forty dollars at list prices. The arithmetic runs on a laptop.