The inference crisis: when one third of experts draw the wrong conclusions with “way too much confidence”.

So much for expert judgement

In a test of scientists abilities, the same data was sent to 27 teams of researchers in cognitive psychology. The idea was to test the theoretical inferences they drew. But those expert teams drew conclusions from identical data that varied, oh boy, all the way from “zero to 100 percent.” One of the research team described it as a “jaw dropping” result – where only one third of the experts made the correct inferences about what that data meant. Two thirds of the experts were either totally wrong or just operating “a bit better than pure guessing”.

What are we teaching at universities?

Beyond the ‘replication crisis,’ does research face an ‘inference crisis’?

Researchers test expert inferences against known data, find inconsistency

What they found was “enormous variability between researchers in what they inferred from the same sets of data,” Starns says. “For most data sets, the answers ranged from 0 to 100 percent across the 27 responders,” he adds, “that was the most shocking.”

Rotello reports that about one-third of responders “seemed to be doing OK,” one-third did a bit better than pure guessing, and one-third “made misleading […]