Across the social sciences, half of research doesn’t replicate | Science | AAAS

Nothing new! This too has been studied and reported frequently over the past several decades!

"A sweeping project involving hundreds of researchers in several dozen countries showed that across the social sciences, the findings of roughly half of all papers cannot be replicated independently, and there’s no reliable way to tell in advance which ones will falter. Called Systematizing Confidence in Open Research and Evidence (SCORE), the effort investigated more than 100 papers published in dozens of leading journals in business, economics, education, political science, psychology, and sociology. The replication success rate—49% for the 164 papers evaluated, reported today in Nature—is consistent with findings from previous studies in individual fields such as psychology, suggesting the problem is pervasive in the social sciences. ..."

From the abstract:

"Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol.

Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper.

We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results.

We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability."

Across the social sciences, half of research doesn’t replicate | Science | AAAS (no public access)

Investigating the replicability of the social and behavioural sciences (no public access)

Scientific Reproducibility Or Opinion Polls?

Posted: 3/20/2016

Caveat

As usual, I don’t have the time to more thoroughly investigate this subject. Thus, I am relying here more on hunches and common sense ...

Trigger

A recent study showing that a large proportion (about 50% or so) of the results of psychological studies cannot be reproduced caused quite a stir, I believe. See e.g. articles Scientists Replicated 100 Psychology Studies, and Fewer Than Half Got the Same Results  The massive project shows that reproducibility problems plague even top scientific journals and Over half of psychology studies fail reproducibility test Largest replication study to date casts doubt on many published positive results..

Then, we have a recent study to show that a similar result can be obtained from replicating experimental economics studies. See e.g. About 40% of economics experiments fail replication survey. Experimental studies in economics have been suspect for quite some years due to the overuse of young college/university students as subjects.

Scientific Publication Bias

It was argued that reputable scientific journals rather choose sensational studies over less sensational studies. Sure this sells better!

It was reported that scientific journals are already working on improving the selection of articles based on the above studies.

Counter Study

Study that undercut psych research got it wrong  Widely reported analysis that said much research couldn’t be reproduced is riddled with its own replication errors, researchers say

Among other things, the study criticized the sampling used by the above replication study: “... an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on.” The 98 studies selected were taken from three, probably reputable, psychology journals. Thus, this argument should not matter so much.

They also criticized “the methods introduced statistical error into the data, which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true.” If we apply the “by chance alone” argument, then how many of the 98 studies have achieved their results by chance alone?

I believe, these two Harvard University​ professors got it wrong too. They confused reproduction of scientific results and replication of scientific experiments. Scientific results ought to be kind of robust to some, especially minor, changes (as the Harvard termed it infidelities) in the replication of experiments.

Reproducibility In The Natural Sciences

The above Nature article points out that there are also reproducibility projects under way in e.g. cancer biology. First results seem to indicate that reproducibility is also a challenge. At least in the natural sciences challenges or a refutation lead possible to better studies or approaches.

Quality Of Opinion Polls

I would posit that if the results of scientific studies like the ones above (especially in the social sciences) cannot be reproduced, then they are more like mere opinion polls. The results are very specific to or heavily depend on time, place, and people.