The replication crisis may also be a theory crisis

From Ars Technica

Theory protects from “personal intuitions and culturally biased folk theories.”

Cathleen O’Grady – 2/16/2019, 11:00 AM

A jumbled jigsaw puzzle, AKA the state of theory in the behavioral sciences.A jumbled jigsaw puzzle, AKA the state of theory in the behavioral sciences.

A jumbled jigsaw puzzle, AKA the state of theory in the behavioral sciences.

A replication crisis has called into question results from behavioral (and other) sciences. Complaints have focused on poor statistical methods, the burying of negative results, and other “questionable research practices” that undermine the quality of individual studies.

But methods are only part of the problem, as Michael Muthukrishna and Joseph Henrich argue in a paper in Nature Human Behaviour this week. It’s not just that individual puzzle pieces are low in quality; it’s also that there’s not enough effort to fit those pieces into a coherent picture. “Without an overarching theoretical framework,” write Muthukrishna and Henrich, “empirical programs spawn and grow from personal intuitions and culturally biased folk theories.”

Doing research in a way that emphasizes joining the dots constrains the questions you can ask in your research, says Muthukrishna. Without a theoretical framework, “the number of questions that you can ask is infinite.” This makes for a scattered, disconnected body of research. It also feeds into the statistical problems that are widely considered the source of the replication crisis. Having too many questions leads to a large number of small experiments—and the researchers doing them don’t always lay out a strong hypothesis and its predictions before they start gathering data.

This isn’t the first time someone’s argued that better theory makes for better science. It’s a conversation that’s been going on for some time among the people agitating for more robust research. But this is a particularly loud klaxon, in one of the biggest journals in the field, meaning that it might make people sit up and take notice—and possibly start some concrete initiatives to improve theory at the same time as current attempts being made to improve statistical rigor.

Theories about theory

Paul Smaldino, a cognitive scientist who has also been vocal about the need for better theory, points to an infamous psychology paper as a perfect example of what happens when experimental work is divorced from theoretical scientific frameworks. The paper, published in 2011, reported finding evidence of precognition. But that, says Smaldino, “is not a psychology finding. That’s a physics finding. That is everything we know about the laws of physics and causality and how time works, all being wrong.”

The problem with the paper wasn’t just that the methods were bad, he argues, but also that “theory [in psychology] is so weak that something that completely contradicts hundreds of years of science was evaluated without that context.”

Science, he explains, is about accumulating sets of observations that occur reliably—the Sun appears at different places in the sky depending on the season and time of day; finches have different shaped beaks depending on what they eat. “That’s the raw ingredients,” he says. “To make sense of it requires a framework to say, this is how all these different facts fit together, and this is why.” We explain these observations by developing theoretical models—of how the Earth rotates around the Sun on a tilted axis, of natural selection.

Having a good theoretical framework makes it possible to make sense of sets of disconnected facts, and to explain why things happen sometimes and not at other times. Perhaps most importantly, it allows for predictions of what will be found in the data: if our model of human evolution is true, we could predict that we should find huge similarities in the genomes of humans and other great apes—and that’s exactly what we do find. If we made a prediction like this and found it to be false—say, our genomes turned out to be more similar to birds than to other great apes—it would undermine the theoretical framework.

Doing scientific research within a theoretical framework can help to highlight results that are surprising—like finding precognition—and that might therefore need a closer look and plenty of replications to test whether the finding holds up. By drawing connections between behavioral science findings and findings across other fields, “overarching theoretical frameworks pave the way toward a more general theory of human behavior,” write Muthukrishna and Henrich.

More math for the behavioral sciences

Part of what Muthukrishna and Henrich are advocating is a greater use of formal models—a way of setting down ideas about something in cold, hard math. For instance, you might think that, when children are learning whether it’s “toh-may-toe” or “toh-mah-toe” (or many of the other myriad arbitrary cultural variants that humans swim in), they’ll copy the majority of speakers they hear around them.

Read the full story here.

Original Paper

Nature Human Behaviour, 2018. DOI: 10.1038/s41562-018-0522-1  (About DOIs).


via Watts Up With That?

February 17, 2019 at 05:07PM

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: