Paper Title: Prevalence of Multiplicity and Appropriate Adjustments Among Cardiovascular Randomized Clinical Trials Published in Major Medical Journals
Link at JAMA Network Open
The paper is open access. It was led by Dr. Muhammad Shahzeb Khan from John H. Stroger, Jr. Hospital of Cook County, Chicago. I was one of many co-researchers.
Summary:
This paper looks at the presence of multiplicity in clinical trials in cardiology. Multiplicity refers to the possible chance of getting a falsely positive result because a study looks at many different outcomes.
If you study one unbiased outcome, say being alive or dead at the end of a study, it’s pretty simple to understand if a treatment works.
But that is not how clinical trials in cardiology work. In cardiology, we often study composite outcomes. We call this MACE or major adverse cardiac events.
A typical clinical trial might look at whether a drug vs a placebo reduces the combination of 4 outcomes: death, heart attack, stroke or hospitalization for any cardiac cause.
Cardiology trials can get even more complex because researchers can look at secondary outcomes and they can also study how a treatment might look in different subgroups, say men or women, or younger or older patients.
All these looks at the data make it more likely that you can find a positive finding. That’s multiplicity. There are complicated ways to make statistical adjustments for it.
But… After looking at 6 big journals over three years, here was our conclusion:
Findings from this study suggest that cardiovascular RCTs published in medical journals with high impact factors demonstrate infrequent adjustments to correct for multiple comparisons in the primary end point. These parameters may be improved by more standardized reporting.
The meaning I take from this study is that you have to be careful with positive findings. Here is what it says in academic jargon:
These findings adversely reflect on the robustness of data published in journals that carry global reach and generate evidence that can transform clinical guidelines and practice.Â