Meta-analyses function under the assumption that the summation of data from multiple sources is a more accurate estimate of the true effect size than any one individual trial. And yet sometimes such statistical endeavors serve only to add dirt to the already muddy water. Such is the case with the recent trials examining endovascular therapy for acute ischemic stroke. Prior to the publication of MR CLEAN and its’ band of statistically significant misfits, the data regarding endovascular therapy has been consistently negative. Over the past year five RCTs examining endovascular therapy for acute ischemic stroke have been published. In direct contrast to the three trials published in 2013, all of the recent trials were impressively positive (1,2,3,4,5). So much so that the AHA, in their recently updated guidelines, recommended the use of endovascular therapy in a population far broader than any of the studies examined. Even more concerning was their whole-hearted support of the development of a regionalized system capable of instituting the use of endovascular therapy at a national level (6). Stating:
Regional systems of stroke care should be developed. These should consist of:
(a) Healthcare facilities that provide initial emergency care including administration of intravenous r-tPA, including primary stroke centers, comprehensive stroke centers, and other facilities.
(b) Centers capable of performing endovascular stroke treatment with comprehensive periprocedural care, including comprehensive stroke centers and other healthcare facilities, to which rapid transport can be arranged when appropriate (Class I; Level of Evidence A). (Revised from the 2013 guideline)
They go on to say:
It may be useful for primary stroke centers and other healthcare facilities that provide initial emergency care including administration of intravenous r-tPA to develop the capability of performing emergency noninvasive intracranial vascular imaging to most appropriately select patients for transfer for endovascular intervention and reduce time to endovascular treatment (Class IIb; Level of Evidence C). (Revised from the 2013 guideline)
Even the biggest cynics must concede there is a signal of benefit demonstrated throughout the recent trials examining endovascular therapy for acute ischemic stroke. How much of this is due to the true effect of the treatment in question, and how much is in fact due to statistical noise is far more difficult to discern. Due to a lack of both blinding and equipoise of the trialists and the premature stoppage of all four trials following MR CLEAN, the data is likely to be a distortion of reality (7). And yet without a clear understanding of this effect size, it is difficult to assess whether this benefit justifies the resources required to support its implementation on a national level.
In an attempt to clarify this benefit, Badhiwala conducted a systematic review and meta-analysis of RCTs examining the efficacy of endovascular treatment for acute ischemic stroke (8). Published in JAMA, the authors included the five most recent positive trials, as well as the three negative trials published in the NEJM in February of 2013. The primary outcome was the frequently used and widely misunderstood ordinal analysis of functional status (mRS) at 90-days. The authors also examined a dichotomous outcome of the percent of patients alive and independent (mRS of 0-2) at 90 days, rate of revascularization at 24-hrs, symptomatic intracranial hemorrhage, and all cause 90-day mortality (8).
In their pooled analysis, the authors found a shift towards improved functional outcomes at 90-days in the patients randomized to receive endovascular therapy when compared to standard care. This difference demonstrated an odds ratio of 1.56 (1.14–2.13 p = 0.005), which translates into a 12% absolute increase in the amount of a patients alive and independent at 90-days (3.8%-20.3%; p = 0.005). Not surprisingly, patients randomized to the endovascular arm had a significantly higher rate of revascularization at 24-hours when compared to the patients in the standard care group (75.8% vs. 34.1%). No difference was observed in the 90-day mortality (15.8% vs. 17.8%) or the rate of symptomatic intracranial hemorrhage (5.7% vs. 5.1%) (8).
An interesting side note, one of the subgroups the authors examined in their secondary analysis was whether time to randomization had any effect on the efficacy of endovascular therapy. Specifically they looked at time from symptom onset to randomization. They examined the effect size of endovascular therapy as compared to standard care depending on whether patients were randomized before or after three hours from symptom onset. Temporality did not seem to affect outcomes (8). Once again calling into question the time is brain mantra so frequently proclaimed.
Although these results are positive, they are a far cry from the astonishing outcomes reported in the four trials released in the wake of the publication of MR CLEAN. Each of these trials was stopped prematurely because of an unplanned interim analysis provoked by the positive results found in MR CLEAN. And though it is unlikely that four trials would all demonstrate positive results due to random chance, it is well known that trials stopped early for benefit will demonstrate significantly more optimistic results than the intervention’s true effect size (6). In fact, the Badhiwala meta-analysis demonstrates a clear inverse association with the magnitude of benefit and the size of the sample at termination (8). And so the question becomes is it appropriate to pool this data at all?
To answer that question one has to ask, what is the goal of a meta-analysis such as this in the first place? The assumed benefit to performing a meta-analysis is that the summation of these data sets provides a more accurate description of the true effect size than each individual data set can provide. This supposition rests on the notion that all the studies included in the meta-analysis are examining the same study population, and that the variance of results is due to random errors in sampling. This is what is known as a “fixed effect” model. Unfortunately most data is not so homogeneous, and it is common for the variation observed between trials to be due to more than just random error, but to considerable differences in the populations being compared. In such cases, the results of a direct-pooled analysis will likely deviate from reality. Statistical models that attempt to account for these random deviations should be utilized. These are known as “random-effect” models (9).
In the Badhiwala et al meta-analysis the heterogeneity of the results between the trials examined was high and the authors correctly utilized a random-effect model. The authors used the I2 index to assess the degree of variation between studies. I2 describes the extent of variation across trials that cannot be explained by random chance. An I2 score of 0.0 implies all of the variation observed between trials can be accounted for by random errors in sampling. Conversely if the I2 is 75, only 25% of the variation can be accounted for by sampling error with the remaining variation (75%) due to heterogeneity between trials (10). In the Badhiwala et al meta-analysis the I2 = 75.4, confirming that the authors are likely attempting to pool trials with very different populations. Whether this variation is due to methodological differences or the bias that accompanies premature stoppage is unclear. Applying a statistical value to this uncertainty does not legitimize it.
Despite its methodologic rigor, Badhiwala et al's meta-analysis brings us no closer to certitude. It serves to place an objective number on the current ambiguous state of the data concerning endovascular therapy for acute ischemic stroke. But the inherent value of its statistical manipulations in a pooled data set is unclear. This analysis provides little utility over our unstructured judgment of each respective trial’s importance, while validating our suspicion that these trials are examining very different populations. By combining these trials, Badhiwala et al have attempted to augment statistical power in a dataset that already boasts effect sizes well below statistical significance. When truly, what is required, is a clinical homogeneity that no amount of statistical manipulations can supplant. I am sure in the coming months and years this study will be cited in vain. Used to champion the fortification of endovascular therapy’s place in the ivory tower of medicine. What will be remembered are odds ratios and p-values, while the true meaning will be forgotten. That what we have is a heterogeneous data set, ultimately providing more questions than answers.
Sources Cited:
- Berkhemer OA, Fransen PS, Beumer D, et al. A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med. 2015;372:(1)11-20.
- Campbell BC, Mitchell PJ, Kleinig TJ, et al. Endovascular Therapy for Ischemic Stroke with Perfusion-Imaging Selection. N Engl J Med. 2015.
- Goyal M, Demchuk AM, Menon BK, et al. Randomized Assessment of Rapid Endovascular Treatment of Ischemic Stroke. N Engl J Med. 2015.
- Saver JL, Goyal M, Bonafe A, et al. Stent-Retriever Thrombectomy after Intravenous t-PA vs. t-PA Alone in Stroke. N Engl J Med. 2015.
- Jovin TG, Chamorro A, Cobo E, et al. Thrombectomy within 8 Hours after Symptom Onset in Ischemic Stroke. N Engl J Med. 2015.
- Powers WJ, Derdeyn CP, Biller J, et al. 2015 American Heart Association/American Stroke Association Focused Update of the 2013 Guidelines for the Early Management of Patients With Acute Ischemic Stroke Regarding Endovascular Treatment: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association. Stroke. 2015;46(10):3020-35.
- Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010;303(12):1180-7.
- Badhiwala, JH et al. Endovascular Thrombectomy for Acute Ischemic Stroke A Meta-analysisJAMA. 2015;314(17):1832-1843.
- Cornell JE, Mulrow CD, Localio R, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med. 2014;160(4):267-70.
- Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-60.
- EM Nerd-The Case of the Partial Cohort - May 24, 2020
- EM Nerd: The Case of the Sour Remedy Continues - January 20, 2020
- EM Nerd-The Case of the Adjacent Contradictions - December 23, 2019
Well written Rory. Good job in helping readers understand the subject and limitations of SR-MA.
Thanks Ken!!