For the first time in a few years, Dr. Gorski was able to take a week off between Christmas and New Years to work. He decided to take a week after Christmas off from writing on his website, although he might not be able to resist writing about it on his other website. He will probably not take a break, as his batteries are definitely exhausted after a long and painful 2021. We are fortunate to have a guest post this week who will give us an update on stem cell therapies. Dr. Gorski will be able to see you on January 3 if he is able to restrain himself. Happy holidays and new years!
flawed or misleading evidence is worse than no evidence at all. The state ofIgnorance resulting from a lack of evidence is recognized as a state ofIgnorance, whereas the state ofIgnorance resulting from misleading evidence is not.
Berger and Alperson wrote a general framework for the evaluation of clinical trial quality.
Stem cell therapies have been promoted and sold too early on this site. I wrote a series about US Stem Cell, a clinic in Florida that gained unwanted attention when reports emerged of patients being blinded after receiving stem cell therapy. Part 1, Part 2, and Part 3. The FDA acted through the federal courts to stop U.S. Stem Cell.
There is a stem cell operation in Florida. They are using stem cell therapies for diseases of the eye. This group is treating patients in a clinical trial way.
Study subjects are required to pay thousands of dollars for the privilege of receiving experimental treatments in clinical trials like SCOTS. David Gorski and Jann Bellamy wrote about this practice.
Turner has been vocal. His indictment, "ClincalTrials.gov, stem cells and 'pay-to-participate' clinical studies", is behind a paywall. He criticizes the investigators who run the studies, but he doesn't spare the agencies responsible for the health and welfare of Americans.
SCOTS is aStem Cell Treatment Study.
SCOTS was launched in August of 2012 and is expected to end in 2020. It is a treatment study using autologous bone marrow derived stem cells to treat a variety of eye conditions. 300 total patients is the target sample size. Clinical trials.gov has a recruitment status that is unknown.
Steven Levy, MD is the Study Director and Jeffrey Weiss, MD is the Principal Investigator of MD Stem Cells. Both are affiliates of MD Stem Cells.
The documentation shows that all the procedures for the study, including the harvesting of bone marrow and the administration of stem cells, were done in Florida. The patients paid for the privilege of participating in the study.
Gold-standard evidence for the evaluation of clinical interventions is high quality clinical trials. The strengths and weaknesses of the study will be evaluated in this article. I've read about the trial on the website, the publications from the trial, and read some stories in the lay press that were relevant to the study.
Is SCOTS a good trial?
It is not a high quality trial.
The design of the SCOTS study abandons the qualities that make a clinical trial gold-standard. A coherent hypothesis, a well-defined study population, randomization, matched study groups, an appropriate control group, blinding, standardized measurement of critical variables, and adherence to pre-specified analysis methods are missing. The study is unsuitable to answer the questions it purports to address.
The SCOTS investigators have chosen to publish their results in the form of case reports and small case series, representing subsets of patients, vacating the pretense of a clinical trial altogether.
Entry criteria.
A single disease or category of conditions is usually the limit of a clinical trial. SCOTS entry criteria cover a wide range of diseases. Any condition that can cause mild to severe loss of visual function is eligible. There is a broad spectrum of genetic, traumatic, and inflammatory conditions. According to the MD Stem Cell website, they have treated over 47 eye diseases.
Concerns are raised by a broad set of eligible diseases. It is thought that a specific treatment method will be effective in a wide range of diseases. It's nearly impossible to sort out which conditions may be helped, harmed, or unaffected by a novel treatment, because of the 47 different diseases in the trial.
There is a structure.
It would be hard to do better than the SCOTS authors if the study was designed to make the results uninterpretable.
The SCOTS trial is a parallel group design. This is a typical clinical trial. Patients are split into 2 or more groups. The members of each group are assigned to one of the experimental treatments. The groups are compared at the end of the study. The groups should be as similar as possible, except for the assigned treatment. Randomly assigning patients to groups is the best way to achieve this. SCOTS authors have deviated from best practice. The subjects are not assigned randomly to treatment groups. There is no control group because the groups are defined to be different from the start.
COTS treatments.
Stem cells are taken from the patient's bone marrow. The cells are delivered in 5 different ways.
Stem cells are injected into the eye.
Stem cells are injected next to the eye.
Stem cells can be given by injection.
Stem cells are injected into the eye.
Stem cells are injected into patients after they have a vitrectomy, which is a surgery to remove the vitreous humor.
into the eye.
under the eye.
The nerve that connects the eye to the brain.
The IO group consists of 3 treatment options.
There are three treatment arms for the experimental stem cell treatments. The method of assigning patients to treatment groups is described in the publication, but not on clinical trials.gov. The assignment is based on a variety of criteria. The criteria are not clear. It will be nearly impossible to sort out the treatment effect from the baseline differences because these groups are not exactly the same.
Stem cells are administered in 3 different ways. There is no control group.
Group 1:RB,ST,IV.
Group 2 includesRB+ST+ IV.
Group 3 includes RB+ST+IV+IO.
300 patients with 47 different diseases are subject to 7 experimental treatments among 3 treatment groups without a control group. What could possibly go wrong?
This isn't good science. One can not identify treatment effects when sorting experimental treatments. The effect of IV stem cells would never be detected because it is common in every group. The design would prevent the discovery of a harmful effect. Adding in 47 diseases adds to the confusion. The harvest of cherry-picked data can be achieved by the various permutations of interventions and diseases.
The SCOTS trial has visual acuity.
SCOTS uses visual acuity as the primary measure of success. This is an appropriate way to measure vision, but it is difficult to measure. In trials using visual acuity as a critical endpoint, investigators usually maximize efforts to standardize measurement. Standard charts are used at a standard distance. Patients are re-spared to make sure they are measuring their vision at each visit. Even with all these protections, visual acuity is a variable. It depends on testing conditions and patient motivation. The subject and tester can have an effect on visual acuity. If possible, vision testers should be blinded to the treatment of the subject they are testing. A standard script is followed by visual acuity tester in many trials.
Change in vision is a common practice in clinical trials. Valid vision measurement is required at 2 points in time. When vision will be tested is important. Is the change from baseline to 6 months? Is the baseline to 12 months? What if the patient misses the visit? Will the data from that subject be excluded? Will the data be carried forward? All of these decisions should be applied consistently in the data analysis.
It was not possible to standardize the measurement of vision in the SCOTS study. Patients had their vision measured by the SCOTS investigator. Allow me to quote from one of their publications.
The patients from distant geographical locations had their eyes examined by their local eye physician at 1, 3, 6, and 12 months.
The investigators were not in control of the measurement of post treatment visual acuity. We will see later that they sometimes used a different method of vision than the investigators control. The primary outcome variable in the study is change in vision. This is a recipe for a lot of missing data and has the potential for bias. Patients who think they are doing well may be more motivated to send data back to the study investigators. Patients who investigators expect to be doing well might be more motivated to provide data.
SCOTS results.
The main publication of a clinical trial is usually a comprehensive report of the primary analysis of the between-group comparisons, focusing on the primary outcome variable. In the SCOTS, that would be a comparison of the change from pre-treatment to post-treatment vision compared among the 3 treatment arms. All the patients should be accounted for in a report like this. How many people were Enrolled? How many people did the study? How many left? How many people missed the post treatment visit? How many died before the study was done? A detailed amount of the analysis would be given. How many people contributed to the results? How were the missing data addressed? No such report has emerged from SCOTS.
The SCOT investigators decided to publish results as ad hoc disease-specific case-reports and case-series. This mode of reporting undermines the purpose of a clinical trial.
The SCOTS study is said to have started in August of 2012 300 patients were expected to be completed in July 2020. The clinicaltrials.gov listing was updated on Oct 23, 2019. The study is closed to enroll and they are now in SCOTS2.
A case report of a single patient was the first publication for SCOTS. The next two papers were single-patient case reports. They were all in journals that were open to the public. There were many violations of the Health Insurance Portability and Accountability Act, including a patient's date of birth, among the 3 case reports. This is a major violation of patient confidentiality and a violation of the standards demanded by the IRB, and the investigators claimed that they followed the standards.
There are ten publications from SCOTS reporting on a total of 79 patients. Each report has a small group of patients with a specific disease. We don't know how many subjects were actually Enrolled but the target was 300 so we are likely to have a small subset. Current reports account for selected patients from a pool of up to 800 patients, because they have been combining patients from SCOTS. The missing patients have not been accounted for.
The purpose of a clinical trial is not compatible with the publishing of single case success stories and small case series. There are case reports and case series on the MD Stem Cell website. This is questionable from an ethical point of view. Potential study participants could be misled and coerced into taking part in the trial if they see selected successes from the trial.
The conduct and analysis of the study are not detailed on Clinicaltrials.gov. The publications give us a lot of information. There are a lot of discrepancies andIrregularities. There are many unanswered questions.
There are more visual acuity questions.
Vision is discussed as the best corrected visual acuity. The process of finding the optimal set of glasses for the patient to use when reading the eye chart is a tedious one. This is important because we want to know the level of the patients vision, not because they forgot their glasses, or have glasses that are not optimal. This is a standard procedure for trials. In the publications, baseline vision is listed as "PH", which is an abbreviation for "pinhole". Pinhole vision is a dirty and quick way of approximating best corrected vision, but is not a good substitute for a protocol. In several publications, the contradiction of specifying best-corrected vision in the entry criteria and reporting PH vision in the results of the same paper is repeated. Poor measurement of baseline vision could allow ineligible patients to be included, and make biases towards improved vision if it is measured more carefully.
The explanation does not make the situation better. I assumed baseline vision was what the investigator measured before the stem cell intervention. There would be no reason to think otherwise, except that in some of the papers they explicitly state that they used "historical" vision measured at an outside provider before the patient was Enrolled. Measurement of the primary outcome variable was outside of the investigators control at both critical timepoints.
They confessed in one paper that they used different sources of baseline vision to make the data look better. This was part of a case series. The authors wrote in their own words.
When compared to immediate preop vision in the Principal Investigator's examination lane, the visions were minimally decreased in the remaining patient. The post-operative visions which were obtained in his primary eye doctor's examination lane were the same as the pre-operative visions. The visions were judged to be the same.
Historical visual acuity data, PI-measured visual acuities, and a mix of both are used in at least one paper. This is p-hacking at the most basic level and the quote above is a startling confession.
There are many different ways in which visually acuities were reported and analyzed. The investigators ability to perform valid, unbiased analyses of their own data is in serious question because of theseIrregularities.
Incoherent design, haphazard conduct, and uninterpretable results were the conclusion.
SCOTS is not a reliable source of information about stem cell therapies for ocular disease due to the incoherent design, the flawed analysis, and theselective reporting of results.
What do the investigators do now that they have finished a study with unreliable results?
SCOTS 2.
When you think it's safe to seek medical care in Florida, SCOTS 2 comes along. The same investigators are featured in SCOTS 2, which has an additional site in Dubai. The main difference between SCOTS 2 and SCOTS is that Group 3 is a bit more detailed. This seems to be a documentation of the SCOTS practices. SCOTS had a sample size of 300 patients and SCOTS2 had 500 subjects. The estimated starting and ending dates of SCOTS2 are 2016 and 2024.
I would like to ask the SCOTS investigators a lot of questions. Here are a few.
SCOTS was designed to establish the safety and efficacy of any of the 7 stem cell treatments.
The data regarding the other SCOTS patients is not available.
SCOTS established the safety and efficacy of stem cell treatment.
How do you justify doing another clinical trial using a failed study design?
How do you justify doing another clinical trial for a question that has already been answered?
