The scientific process itself is one of the more productive ways that the methods of science can be used. A meta-science study can help tell us when research approaches aren't producing reliable data and can show what we need to change to get those approaches to work.
We shouldn't expect to see any improvements in the area of research where we shouldn't expect to see it. A group of Austrian researchers looked into why a small percentage of clinical trials produce positive results. The researchers found that the trials that show that homeopathy isn't effective are less likely to be published.
Problems have been identified in almost all of the ways that have been used to test potential treatments. The double-blind, randomized clinical trial is the most reliable method of getting rid of biases that make other approaches less reliable. Problems can happen even in double-blind trials. There is always a bias towards publishing positive results where the treatments have an effect.
We can't always be sure if a treatment works or not because we can't always find out. This has been an issue with some of the fad cures.
Preregistering clinical trials has been used to deal with that issue. The design of the trial, the outcomes being measured, and other details are put in a public database before the trial starts. Preregistering would be a requirement for later publication, meaning that anyone who hoped to publish results in the future would have a compelling reason to do so. Lower-profile journals are usually where unregistered trials can be published.
When positive results are being published, this can help us identify. That was one of the analyses done by the Austrian researchers.
AdvertisementThe team of researchers scanned a set of clinical-trial registration databases for trials. The researchers searched the literature on the topic and were able to match a publication to the trial that produced it. In some cases, publications were the results of trials that had not been registered, in other cases, a pre registered trial produced no publications.
There were a few trends that were clear. The number of papers that are the product of preregistered trial designs has grown to 75 percent in the two decades since pre registration started. Roughly half of the pre registered trials don't result in publication. Some of these trials don't go to completion for a variety of mundane, uninteresting reasons; the rate is not very different from what you see in studies of actual medicine. There is a lot of opportunity for a bias against publishing negative results.
Is there any indication of this bias? The new paper has its strongest results at that location. If you do a meta-analysis of all the publications from trials that weren't registered, you will find that the treatments were better than placebo. There was no statistical difference between placebo and homeopathy when the trials were pre registered.
When researchers have to commit to a study design, their results don't show how effective homeopathy is. When researchers can write up their results, it looks good.
There was a strange aspect to the data. One explanation for the apparent success of homeopathy trials is a strong placebo effect, which is generated by the extensive personal interaction between people seeking treatments and the practitioners. The researchers were able to find a single trial with a protocol that included these interactions.
The results are exactly what you would expect, given that there is no reason for homeopathy to do anything. The first paper in the new study is about proposed mechanisms for homeopathy.
The new study goes beyond debunking obvious nonsense. The paper shows that one of the tools we have developed to help us analyze bias can work as it was intended.
Evidence-Based Medicine is published by the British Medical Journal. There is a DOI called 10.1136/bmjebm-2021-111846.
All Rights Reserved © 2024
