The replicability of scientific studies is under the microscope like never before, as scientists are increasingly examining just how many studies can be repeated with the same results a second or third time around.
If a study doesn't pass the so-called replication test, that casts some doubt over the findings, and new investigations suggest we could have a significant replication problem in cancer research.
The research found that no experiments could be set up again using only the information published in the papers. 50 experiments from 23 papers were reproduced after getting help from the original study authors.
The results of the reproduced tests showed that the effect sizes were often smaller than what the original studies yielded, which is concerning because some of the original authors never responded to requests for help.
The Center for Open Science in Virginia says that the evidence for the replication experiments was much weaker than the original findings.
There is room for improvement in cancer research.
Less than half of the effects measured in the follow-up experiments were able to pass the five replicability criteria. The effect size and the overall positive or negative conclusions were covered by the criteria.
The report tells us a lot about the culture and realities of the way cancer biology works, and it's not a flattering picture at all, according to a commentary by a bioethicist.
We might be wasting time testing drugs on patients that aren't going to have any effect on the disease if the uncertainty around the conclusions of the cancer studies is any indication.
Scientists are doing more than ever to address and tackle the problem of reproducibility, which is good news. It should lead to improvements in the future because experts are now better aware of some of the issues around clarity and thoroughness.
Being unable to replicate a study doesn't mean the original study is wrong or inaccurate, it just means that no experiment can ever be reproduced perfectly a second time.
A failure to replicate does not mean that a finding is not true, but it does suggest that more investigation is needed to establish its reliability.
What level of replicability would be acceptable for cancer research?
The team behind the research is hopeful that the report will lead to less research being conducted that is not transparent and open.
Brian Nosek is the Executive Director of the Center for Open Science. The evidence suggests that we could be doing better.
The research was published in two studies, one on the difficulty of replicating experiments and the other analyzing the experiments that were replicated.
