The big picture of the evidence for a particular medical intervention can be seen through systematic reviews. They are not easy to do well and fall prey to their own biases. It is better to look at a systematic review in a different way. The overall pattern is what tells us if we are dealing with a real phenomenon or just a placebo effect.
A systematic review of systematic reviews of acupuncture for a variety of indications was published in November 2022. This is the same evidence we have been looking at for a long time, and so I don't think there will be any surprises. Before diving into the study, I would like to review what acupuncture is and why I am skeptical of its effectiveness.
Specific clinical outcomes can be achieved through the use of specific acupuncture points on the body that correspond to specific functions. I recently reviewed a systematic review of back pain in pregnant women and went over some of the problems with the research. A detailed analysis of acupuncture was published by me. Here are my main criticisms of the system of medicine.
There is no legitimate reason to think that placebo medicine is real, because it is all fake. Thousands of clinical studies looking at acupuncture for various conditions have been found to be very low quality and flawed. Many of the studies will be positive even if they have a completely null effect. This is seen in the literature. If my analysis is correct and fair, we would expect the same pattern of outcome in the acupuncture literature that we see in the homeopath literature.
It is culture- dependent, which is a red flag. The effect is much more dramatic. A 1998 systematic review shows that the trials based in China are positive. This is an "open secret" that the data is fabricated in China to support the practice of Chinese medicine. We wouldn't see a 100% positive rate in the literature even if the treatment had a large effect size. It's pretty clear that there is some combination of publication bias and researcher bias. Unless they are systematically excluded, the biased Chinese acupuncture studies ruin all systematic reviews of acupuncture.
The systematic reviews which are negative are presented as if they are positive. The authors conclusions seem to be taken at face value by the current reviewers.
What is the current review of reviews showing? All the biases in the literature make the evidence weak and negative. The author took a big picture look at the evidence.
Despite a vast number of randomized trials, systematic reviews of acupuncture for adult health conditions have rated only a minority of conclusions as high- or moderate-certainty evidence, and most of these were about comparisons with sham treatment or had conclusions of no benefit of acupuncture. Conclusions with moderate or high-certainty evidence that acupuncture is superior to other active therapies were rare.
The majority of the evidence is not good. We have to seriously ask, after decades and thousands of studies of acupuncture, why do a low quality study at all? There are methods and technology for high quality Chinese medicine. It is possible that there are so many low quality studies being performed because they are frustratingly negative.
Only four health conditions had high-quality evidence. There wasn't an increase in the chance of success with implanting for one of the four. One of the methods used to recover from a stroke was the use of electrical nerve stimulation. Conventional interventions were included. There was a negative review of stroke that did not include acupuncture.
The other two that had positive results were for shoulder pain and Fibromyalgia. There were no indications with a high number of high quality studies showing a positive effect. Two of the studies contained in the review had mixed results. Even though some of the studies compared the two therapies, there was no agreement on which was the correct one.
The review looked at a variety of conditions. The low number of studies and heterogeneity in terms of controls were the same problems as the Fibromyalgia review. The risk of bias is high. There's no control for the publication bias in Chinese studies.
I would think that by now we would see a clear signal in the data, but we are not. The review shows that most of the evidence is low quality. The two subsets that had high quality evidence showing an effect were overlooked by the reviewers. If you do enough studies, you will get some positive evidence just by chance.
The overall pattern of the literature is not positive. I would argue that it is consistent. The pattern is the same with other implausible claims. It probably isn't a coincidence.
Steven Novella is the founder and currently Executive Editor of Science-Based Medicine. The Great Courses and The Skeptics Guide to the Universe were both written by Dr. Novella.
All posts are available to view.