- Meta-analysis
Often the results for the same intervention differ across clinical trials and it may not be
clear whether one therapy provides more benefit than another. As CER increases and more
studies are conducted, clinicians and policymakers are more likely to encounter this scenario.
In a systematic review, a researcher identifies similar studies and displays their results
in a table, enabling qualitative comparisons across the studies. With a meta-analysis,
the data from included studies are statistically combined into a single “result.” Merging
the data from a number of studies increases the effective sample size of the investigation,
providing a statistically stronger conclusion about the body of research. By so doing,
investigators may detect low frequency events and demonstrate more subtle distinctions
between therapeutic alternatives.
When studies have been properly identified and combined, the meta-analysis produces a
summary estimate of the findings and a confidence interval that can serve as a benchmark
in medical opinion and practice. However, when done incorrectly, the quantitative and
statistical analysis can create impressive “numbers” but biased results. The following are
important criteria for properly conducted meta-analyses:
- Carefully defining unbiased inclusion or exclusion criteria for study selection.
- Including only those studies that have similar design elements, such as patient
population, drug regimen, outcomes being assessed, and timeframe.
- Applying correct statistical methods to combine and analyze the data.
Reporting this information is essential for the reader to determine whether the data were
suitable to combine, and if the meta-analysis draws unbiased conclusions. Meta-analyses of
randomized clinical trials are considered to be the highest level of medical evidence as
they are based upon a synthesis of rigorously controlled trials that systematically reduce
bias and confounding. This technique is useful in summarizing available evidence and will
likely become more common in the era of publicly funded comparative effectiveness research.
The following case study will examine several key principles that will be useful as the
reader encounters these publications.
- Clinical Application
Heart disease is the leading cause of mortality in the United States, resulting in
approximately 20% of all deaths. Diabetics are particularly susceptible to heart disease,
with more than 65% of deaths attributable to it. The nonfatal complications of diabetes
are wide-ranging and include kidney failure, nerve damage, amputation, stroke and blindness,
among other outcomes. In 2007, the total estimated cost of diabetes in the United States
was $174B; $116B was derived from direct medical expenditures and the rest from the
indirect cost of lost productivity due to the disease. With such serious health effects
and heavy direct and indirect costs tied to diabetes, proper disease management is critical.
Historically, diabetes treatment has focused on strict blood sugar control, assuming
that this goal not only targets diabetes but also reduces other serious comorbidities
of the disease.
Anti-diabetic agents have long been associated with key questions as to their benefits/risks
in the treatment of diabetes. The sulfonylurea tolbutamide, a first generation anti-diabetic
drug, was found in a landmark study in the 1970s to significantly increase the CV mortality
rate compared to patients not on this agent. Further analysis by external parties concluded
that the methods employed in this trial were significantly flawed (e.g., use of an “arbitrary”
definition of diabetes status, heterogeneous baseline characteristics of the populations studied,
and incorrect statistical methods). Since these early studies, CV concerns continue to be an
issue with selected oral hypoglycemic agents that have subsequently entered the marketplace.
A class of drugs, thiazolidinedione (TZD), was approved in the late 1990s, as a solution to the
problems associated with the older generation of sulfonylureas. Rosiglitazone, a member of the
TZD class, was approved by the FDA in 1999 and was widely prescribed for the treatment of
type-2 diabetes. A number of RCTs supported the benefit of rosiglitazone as an important new
oral antidiabetic agent. However, safety concerns developed as the FDA received reports of
adverse cardiac events potentially associated with rosiglitazone. It was in this setting that a
meta-analysis by Nissen and Wolski was published in the New England Journal of Medicine in June 2007.
- What was done?
Nissen and Wolski conducted a meta-analysis examining the impact of rosiglitazone on
cardiac events and mortality compared to alternative therapeutic approaches. The study
began with a broad search to locate potential studies for review. The authors screened
published phase II, III, and IV trials; the FDA website; and the drug manufacturer’s
clinical-trial registry for applicable data relating to rosiglitazone use. When the
initial search was complete, the studies were further categorized by pre-stated inclusion
criteria. Meta-analysis inclusion criteria were simple: studies had to include rosiglitazone
and a randomized comparator group treated with either another drug or placebo, study arms
had to show similar length of treatment, and all groups had to have received more than 24
weeks of exposure to the study drugs. The studies had to contain outcome data of interest
including the rate of myocardial infarction (MI) or death from all CV causes. Out of 116
studies surveyed by the authors, 42 met their inclusion criteria and were included in the
meta-analysis. Of the studies they included, 23 had durations of 26 weeks or less, and only
five studies followed patients for more than a year. Until this point, the study’s authors
were following a path similar to that of any reviewer interested in CV outcomes, examining
the results of these 42 studies and comparing them qualitatively. Quantitatively combining
the data, however, required the authors to make choices about the studies they could merge
and the statistical methods they should apply for analysis. Those decisions greatly influenced
the results that were reported.
- What was found?
When the studies were combined, the meta-analysis contained data from 15,565 patients in
the rosiglitazone group and 12,282 patients as comparators. Analyzing their data, the
authors chose one particular statistical method (the Peto odds ratio method, a fixed-effect
statistical approach), which calculates the odds of events occurring where the outcomes of
interest are rare and small in number. In comparing rosiglitazone with a “control” group
that included other drugs or placebo, the authors reported odds ratios of 1.43 (95% CI,
1.03-1.98; P=0.03) and 1.64 (95% CI, 0.98-2.74; P=0.06) for MI and death from CV causes,
respectively. In other words, the odds of an MI or death from a CV cause are higher for
rosiglitazone patients than for patients on other therapies or placebo. The authors
reported that rosiglitazone was significantly associated with an increase in the risk of
MI and had borderline significance in increasing the risk of death from all CV causes.
These findings appeared online on the same day that the FDA issued a safety alert
regarding rosiglitazone. Discussion of the meta-analysis was immediately featured
prominently in the news media. By December 2007, prescription claims for the drug at
retail pharmacies had fallen by more than 50%.
As diabetic patients and their clinicians reacted to the news, a methodologic debate also
ensued. This discussion included statistical issues pertaining to the conduct of the analysis,
its implications for clinical care, and finally the FDA and drug manufacturer’s roles in
overseeing and regulating rosiglitazone. The concern among patients with diabetes regarding
treatment, continues in the medical community today.
- Was this the right answer?
Should the studies have been combined? Commentators faulted the authors for including several
studies that were not originally intended to investigate diabetes, and for combining both
placebo and drug therapy data into one comparator arm. Some critics noted that despite the
stated inclusion criteria, some data were derived from studies where the rosiglitazone
arm was allowed a longer follow-up than the comparator arm. By failing to account for
this longer follow-up period, commentators felt that the authors may have overestimated
the effect of rosiglitazone on CV outcomes. Many reviewers were concerned that this
meta-analysis excluded trials in which no patients suffered an MI or died from CV causes
- the outcomes of greatest interest. Some reviewers also noted that the exclusion of
zero-event trials from the pooled dataset not only gave an incomplete picture of the
impact of rosiglitazone but could have increased the odds ratio estimate. In general,
the pooled dataset was criticized by many for being a faulty microcosm of the information
available regarding rosiglitazone.
It is essential that a meta-analysis be based on similarity in the data sources. If studies
differ in important areas such as the patient populations, interventions, or outcomes,
combining their data may not be suitable. The researchers accepted studies and populations
that were clinically heterogeneous, yet pooled them as if they were not. The study reported
that the results were combined from a number of trials that were not initially intended to
investigate CV outcomes. Furthermore, the available data did not allow for time-to-event
analysis, an essential tool in comparing the impact of alternative treatment options.
Reviewers considered the data to be insufficiently homogeneous, and the line of cause and
effect to be murkier than the authors described.
- Were the statistical methods optimal?
The statistical methods for this meta-analysis also came under significant criticism.
The critiques focused on the authors’ use of the Peto method as being an incorrect choice
because data were pooled from both small and very large studies, resulting in a potential
overestimation of treatment effect. Others reviewers pointed that the Peto method should
not have been used, as a number of the underlying studies did not have patients assigned
equally to rosiglitazone and comparator groups. Finally, critics suggested that the
heterogeneity of the included studies required an altogether different set of analytic
techniques.
Demonstrating the sensitivity of the authors’ initial analysis to the inclusion criteria and
statistical tests used, a number of researchers reworked the data from this study. One
researcher used the same studies but analyzed the data with a more commonly used statistical
method (Mantel-Haenszel), and found no significant increase in the relative risk or common
odds ratio with MI or CV death. When the pool of studies was expanded to include those
originally eliminated because they had zero CV events, the odds ratios for MI and death from
CV causes dropped from 1.43 to 1.26 (95% CI, 0.93-1.72) and from 1.64 to 1.14
(95% CI, 0.74-1.74), respectively. Neither of the recalculated odd ratios were significant
for MI or CV death. Finally, several newer long-term studies have been published since the
Nissen meta-analysis. Incorporating their results with the meta-analysis data showed
that rosiglitazone is associated with an increased risk of MI but not of CV death.
Thus, the findings from these meta-analyses varied with the methods employed, the
studies included, and the addition of later trials.
- Emerging Data
The controversy surrounding the rosiglitazone meta-analysis authored by Nissen and Wolski
forced an unplanned interim analysis of a long-term, randomized trial investigating the
CV effects of rosiglitazone among patients with type 2 diabetes. The authors of the
RECORD trial noted that even though the follow-up at 3.75 years was shorter than expected,
rosiglitazone, when added to standard glucose-lowering therapy, was found to be associated
with an increase in the risk of heart failure but was not associated with any increase in
death from CV or other causes. Data at the time were found to be insufficient to determine
the effect of rosiglitazone on an increase in the risk of MI. the final report of that trial,
published in June 2009, confirmed the elevated risk of heart failure in people with type
2 diabetes treated with rosiglitazone in addition to glucose-lowering drugs, but continued
to show inconclusive results about the effect of the drug therapy on the risk of MI.
Further, the RECORD trial clarified that rosiglitazone does not result in an increased risk of
CV morbidity or mortality compared to standard glucose-lowering drugs. Other trials
conducted since the publishing of the meta-analysis have corroborated these results,
casting further doubt on the findings of the meta-analysis published by Nissen and Wolski.
- Now what?
Some sources suggest that the original Nissen meta-analysis delivered more harm than benefit,
and that a well-recognized medical journal may have erred in its process of peer review.
Despite this criticism, it is important to note that subsequent publications support the
risk of adverse CV events associated with rosiglitazone, although rosiglitazone use does
not appear to increase deaths. These results and emerging data point to the need for further
rigorous research to clarify the benefits and risks of rosiglitazone on a variety of outcomes,
and the importance of directing the drug to the population that will maximally benefit from
its use.
- Lessons Learned From this case Study
Results from initial randomized trials that seem definitive at one time may not be conclusive,
as further trials may emerge to clarify, redirect, or negate previously accepted results.
A meta-analysis of those trials can lead to varying results based upon the timing of the
analysis and the choices made in its performance.
- Meta-Analysis: Tips for CER Practitioners
- The results of a meta-analysis are highly dependent on the studies included
(and excluded). Are these criteria properly defined and relevant to the purposes of
the meta-analysis? Were the combined studies sufficiently similar? Can results from
this cohort be generalized to other populations of interest?
- The statistical methodology can impact study results. Have there been reviews
critiquing the methods used in the meta-analysis?
- A variety of statistical tests should be considered, and perhaps reported, in the
analysis of results. Do the authors mention their rationale in choosing a statistical
method? Do they show the stability of their results across a spectrum of analytical
methods?
- Nothing is permanent. Emerging data may change the playing field, and meta-analysis
results are only as good as the data and statistics from which they are derived.
SOCR Resource Visitor
number 
