Study can't confirm/reproduce lab results for about 50% of cancer experiments

Very concerning!

Can we expect similar devastating results for other areas of medicine? How about SARS-CoV-2/Covid-19?

The wild west of science! Scientists are all too human!

"Eight years ago, a team of researchers launched a project to carefully repeat early but influential lab experiments in cancer research.

They recreated 50 experiments, the type of preliminary research with mice and test tubes that sets the stage for new cancer drugs. The results reported Tuesday: About half the scientific claims didn’t hold up. ...

For the project, the researchers tried to repeat experiments from cancer biology papers published from 2010 to 2012 in major journals such as Cell, Science and Nature.

Overall, 54% of the original findings failed to measure up to statistical criteria set ahead of time by the Reproducibility Project, according to the team’s study published online Tuesday by eLife. The nonprofit eLife receives funding from the Howard Hughes Medical Institute ...

Often, they couldn’t get help from the scientists who did the original work when they had questions about which strain of mice to use or where to find specially engineered tumor cells.

... it is concerning that about a third of scientists were not helpful, and, in some cases, were beyond not helpful,” ..."

Study can't confirm lab results for many cancer experiments | AP News

Reproducibility Project: Cancer Biology

Investigating the replicability of preclinical cancer biology (open access, it is not clear whether this study was peer reviewed)

Scientific Reproducibility Or Opinion Polls?

Posted: 3/20/2016

Caveat

As usual, I don’t have the time to more thoroughly investigate this subject. Thus, I am relying here more on hunches and common sense ...

Trigger

A recent study showing that a large proportion (about 50% or so) of the results of psychological studies cannot be reproduced caused quite a stir, I believe. See e.g. articles Scientists Replicated 100 Psychology Studies, and Fewer Than Half Got the Same Results  The massive project shows that reproducibility problems plague even top scientific journals and Over half of psychology studies fail reproducibility test Largest replication study to date casts doubt on many published positive results..

Then, we have a recent study to show that a similar result can be obtained from replicating experimental economics studies. See e.g. About 40% of economics experiments fail replication survey. Experimental studies in economics have been suspect for quite some years due to the overuse of young college/university students as subjects.

Scientific Publication Bias

It was argued that reputable scientific journals rather choose sensational studies over less sensational studies. Sure this sells better!

It was reported that scientific journals are already working on improving the selection of articles based on the above studies.

Counter Study

Study that undercut psych research got it wrong  Widely reported analysis that said much research couldn’t be reproduced is riddled with its own replication errors, researchers say

Among other things, the study criticized the sampling used by the above replication study: “... an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on.” The 98 studies selected were taken from three, probably reputable, psychology journals. Thus, this argument should not matter so much.

They also criticized “the methods introduced statistical error into the data, which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true.” If we apply the “by chance alone” argument, then how many of the 98 studies have achieved their results by chance alone?

I believe, these two Harvard University​ professors got it wrong too. They confused reproduction of scientific results and replication of scientific experiments. Scientific results ought to be kind of robust to some, especially minor, changes (as the Harvard termed it infidelities) in the replication of experiments.

Reproducibility In The Natural Sciences

The above Nature article points out that there are also reproducibility projects under way in e.g. cancer biology. First results seem to indicate that reproducibility is also a challenge. At least in the natural sciences challenges or a refutation lead possible to better studies or approaches.

Quality Of Opinion Polls

I would posit that if the results of scientific studies like the ones above (especially in the social sciences) cannot be reproduced, then they are more like mere opinion polls. The results are very specific to or heavily depend on time, place, and people.