Scientific Reproducibility Or Opinion Polls?

Posted: 3/20/2016

Caveat

As usual, I don’t have the time to more thoroughly investigate this subject. Thus, I am relying here more on hunches and common sense ...

Trigger

A recent study showing that a large proportion (about 50% or so) of the results of psychological studies cannot be reproduced caused quite a stir, I believe. See e.g. articles Scientists Replicated 100 Psychology Studies, and Fewer Than Half Got the Same Results  The massive project shows that reproducibility problems plague even top scientific journals and Over half of psychology studies fail reproducibility test Largest replication study to date casts doubt on many published positive results..

Then, we have a recent study to show that a similar result can be obtained from replicating experimental economics studies. See e.g. About 40% of economics experiments fail replication survey. Experimental studies in economics have been suspect for quite some years due to the overuse of young college/university students as subjects.

Scientific Publication Bias

It was argued that reputable scientific journals rather choose sensational studies over less sensational studies. Sure this sells better!

It was reported that scientific journals are already working on improving the selection of articles based on the above studies.

Counter Study

Study that undercut psych research got it wrong  Widely reported analysis that said much research couldn’t be reproduced is riddled with its own replication errors, researchers say

Among other things, the study criticized the sampling used by the above replication study: “... an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on.” The 98 studies selected were taken from three, probably reputable, psychology journals. Thus, this argument should not matter so much.

They also criticized “the methods introduced statistical error into the data, which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true.” If we apply the “by chance alone” argument, then how many of the 98 studies have achieved their results by chance alone?

I believe, these two Harvard University​ professors got it wrong too. They confused reproduction of scientific results and replication of scientific experiments. Scientific results ought to be kind of robust to some, especially minor, changes (as the Harvard termed it infidelities) in the replication of experiments.

Reproducibility In The Natural Sciences

The above Nature article points out that there are also reproducibility projects under way in e.g. cancer biology. First results seem to indicate that reproducibility is also a challenge. At least in the natural sciences challenges or a refutation lead possible to better studies or approaches.

Quality Of Opinion Polls

I would posit that if the results of scientific studies like the ones above (especially in the social sciences) cannot be reproduced, then they are more like mere opinion polls. The results are very specific to or heavily depend on time, place, and people.