- An observational study
An observational study is a very common type of research design in which the effects of a
treatment or condition are studied without formally randomizing patients in an experimental
design. Such studies can be done prospectively, wherein data are collected about a group of
patients going forward in time; or retrospectively, in which the researcher looks into the past,
mining existing databases for data that have already been collected. Latter studies are
frequently performed by using an electronic database that contains, for example, administrative,
“billing,” or claims data. Less commonly, observational research uses electronic health
records, which have greater clinical information that more closely resembles the data
collected in an RCT. Observational studies often take place in “real- world” environments,
which allow researchers to collect data for a wide array of outcomes. Patients are not
randomized in these studies, but the findings can be used to generate hypotheses for
investigation in a more constrained experimental setting. Perhaps the best known observational
study is the “Framingham study,” which collected demographic and health data for a group of
individuals over many years (and continues to do so) and has provided an understanding of
the key risk factors for heart disease and stroke.
Observational studies present many advantages to the comparative effectiveness researcher.
The study design can provide a unique glimpse of the use of a health care intervention in
the “real world,” an essential step in gauging the gap between efficacy (can a treatment
work in a controlled setting?) and effectiveness (does the treatment work in a real-life
situation?). Furthermore, observational studies can be conducted at low cost, particularly
if they involve the secondary analysis of existing data sources. CER often uses
administrative databases, which are based upon the billing data submitted by providers
during routine care. These databases typically have limited clinical information, may have
errors in them, and generally do not undergo auditing.
The uncontrolled nature of observational studies allows them to be subject to bias and
confounding. For example, doctors may prescribe a new medication only for the sickest patients.
Comparing these outcomes (without careful statistical adjustment) with those from less
ill patients receiving alternative treatment may lead to misleading results. Observational
studies can identify important associations but cannot prove cause and effect. These
studies can generate hypotheses that may require RCTs for fuller demonstration of those
relationships. Secondary analysis can also be problematic if researchers overwork datasets
by doing multiple exploratory analyses (e.g., data-dredging): the more we look, the more we
find, even if those findings are merely statistical aberrations. Unfortunately, the growing
need for CER and the wide availability of administrative databases may lead to selection of
research of poor quality with inaccurate findings.
In comparative effectiveness research, observational studies are typically considered to be
less conclusive than RCTs and meta-analyses. Nonetheless, they can be useful, especially
because they examine typical care. Due to lower cost and improvements in health information,
observational studies will become increasingly common. Critical assessment of whether the
described results are helpful or biased (based upon how the study was performed) are necessary.
This case will illustrate several characteristics of the types of studies that will assist
in evaluating newly published work.
- Clinical Applications
Cardiovascular diseases (CVD) are the leading cause of death in women older than the age of 50.
Epidemiologic evidence suggests that estrogen is a key mediator in the development of CVD.
Estrogen is an ovarian hormone whose production decreases as women approach menopause.
The steep increase in CVD in women at menopause and older and in women who have had
hysterectomies further supports a relationship between estrogen and CVD. Building on this
evidence of biologic plausibility, epidemiological and observational studies suggested that
estrogen replacement therapy (a form of hormone replacement therapy, or HRT) had positive
effects on the risk of CVD in postmenopausal women, (albeit with some negative effects in
its potential to increase the risk for breast cancer and stroke).65 Based on these findings,
in the 1980s and 1990s HRT was routinely employed to treat menopausal symptoms and serve as
prophylaxis against CVD.
- What was done?
The Nurses’ Health Study (NHS) began collecting data in 1976. In the study, researchers intended to examine a broad range of health effects in women over a long period of time, and a key goal was to clarify the role of HRT in heart disease. The cohort (i.e., the group being followed) included married registered nurses aged 30-55 in 1976 who lived in the 11 most populous states. To collect data, the researchers mailed the study participants a survey every 2 years that asked questions about topics such as smoking, hormone use, menopausal status, and less frequently, diet. Data were collected for key end points that included MI, coronary-artery bypass grafting or angioplasty, stroke, total CVD mortality, and deaths from all causes.
- What was found?
At a 10-year follow-up point, the NHS had a study pool of 48,470 women. The researchers found that estrogen use (alone, without progestin) in postmenopausal women was associated with a reduction in the incidence of CVD as well as in CVD mortality compared to non-users. Later, estrogen-progestin combination therapy was shown to be even more cardioprotective than estrogen monotherapy, and lower doses of estrogen replacement therapy were found to deliver equal cardioprotection and lower the risk for adverse events. NHS researchers were alert to the potential for bias in observational studies. Adjustment for risk factors such as age (a typical practice to eliminate confounding) did not change the reported findings.
- Was this the right answer?
The NHS was not unique in reporting the benefits associated with HRT; other observational studies corroborated the NHS findings. A secondary retrospective data analysis of the UK primary care electronic medical record database, for example, also showed the protective effect associated with HRT use. Researchers were aware of the fundamental limitations of observational studies, particularly with regard to selection bias. They and practicing clinicians were also aware of the potential negative health effects of HRT, which had to be constantly weighed against the potential cardioprotective benefits in deciding a patient’s course of treatment. As a large section of the population could experience the health effects of HRT, researchers began planning RCTs to verify the promising observational study results. It was highly anticipated that those RCTs would corroborate the belief that estrogen replacement can reduce CVD risk.
- Randomized Controlled Trial: The Women’s Health Initiative
The Women’s health Initiative (WHI) was a major study established by the National Institutes of health in 1992 to assess a broad range of health effects in postmenopausal women. The trial was intended to follow these women for 8 years, at a cost of millions of dollars in federal funding. Among its many facets, it included an RCT to confirm the results from the observational studies discussed above. To fully investigate earlier findings, the WHI had two subgroups. One subgroup consisted of women with prior hysterectomies; they received estrogen monotherapy. The second group consisted of women who had not undergone hysterectomy; they received estrogen in combination with progestin. The WHI enrolled 27,347 women in their HRT investigation: 10,739 in the estrogen-alone arm and 16,608 in the estrogen plus progestin arm. Within each arm, women were randomly assigned to receive either HRT or placebo. All women in the trial were postmenopausal and aged 50-79 years; the mean age was 63.6 years (a fact that would be important in later analysis). Some participants had experienced previous CV events. The primary outcome of both subgroups was coronary heart disease (CHD), as described by nonfatal MI or death due to CHD.
The estrogen-progestin arm of the WHI was halted after a mean follow-up of 5.2 years, 3 years earlier than expected, as the HRT users in this arm were found to be at increased risk for CHD compared to those who received placebo. The study also noted elevated rates of breast cancer and stroke, among other poor outcomes. The estrogen-alone arm continued for an average follow-up of 6.8 years before being similarly discontinued ahead of schedule. Although this part of the study did not find an increased risk of CHD, it also did not find any cardioprotective effect. Beyond failing to locate any clear CV benefits, the WHI also found real evidence of harm, including increased risk of blood clots, breast cancer and stroke. Initial WHI publications therefore recommended against HRT being prescribed for the secondary prevention of CVD.
- What Next?
Scientists and the clinicians who relied on their data for guidance in treating patients, were faced with conflicting data: epidemiological and observational studies suggested that HRT was cardioprotective while the higher-quality evidence from RCTs strongly suggested the opposite. Clinicians primarily followed the WHI results, so prescriptions for HRT in postmenopausal women quickly declined. Meanwhile, researchers began to analyze the studies for potential discrepancies, and found that the women being followed in the NHS and the WHI differed in several important characteristics.
First, the WHI population was older than the NHS cohort, and many had entered menopause at least 10 years before they enrolled in the RCT. Thus, the WHI enrollees experienced a long duration from the onset of menopause to the commencement of HRT. At the same time, many in the NHS population were closer to the onset of menopause and were still displaying hormonal symptoms when they began HRT. Second, although the NHS researchers adjusted the data for various confounding effects, their results could still have been subject to bias. In general, the NHS cohort was more highly educated and of a higher socioeconomic status than the WHI participants, and therefore more likely to see a physician regularly. The NHS women were also leaner and generally healthier than their RCT counterparts, and had been selected for their evident lack of pre-existing CV conditions. This selection bias in the NHS enrollment may have led to a “healthy woman” effect that in turn led to an overestimation of the benefits of therapy in the observational study. Third, researchers noted that dosing differences between the two study types may have contributed to the divergent results. The NHS reported beneficial results following low-dose estrogen therapy. The WHL, meanwhile, used a higher estrogen dose, exposing women to a larger dosage of hormones and increasing their risk for adverse events. The increased risk profile of the WHI women (e.g., older, more comorbidities, higher estrogen dose) could have contributed to the evidence of harm seen in the WHI results.
- Emerging Data
In addition to identifying the inherent differences between the two study populations, researchers began a secondary analysis of the NHS and WHI trials. NHS researchers reported that women who began HRT close to the onset of menopause had a significantly reduced risk of CHD. In the subgroups of women that were older and had a similar duration after menopause compared with the WHI women, they found no significant relationship between HRT and CHD. Also, the WHI study further stratified these results by age, and found that women who began HRT close to their onset of menopause experienced some cardioprotection, while women who were further from the onset of menopause had a slightly elevated risk for CHD.
Secondary analysis of both studies was therefore necessary to show that age and a short duration from the onset of menopause are crucial to HRT success as a cardioprotective agent. Neither study type provided “truth” or rather, both studies provided “truth” if viewed carefully (e.g., both produced valid and important results). The differences seen in the studies were rooted in the timing of HRT and the populations being studied.
- Lessons Learned From this case Study
Although RCTs are given a higher evidence grade, observational studies provide important clinical insights. In this example, the study populations differed. For policymakers and clinicians, it is crucial to examine whether the CER was based upon patients similar to those being considered. Any study with a dissimilar population may provide non-relevant results. Thus, readers of CER need to carefully examine the generalizability of the findings being reported.
- Observational Studies: Tips for the CER Practitioners
- Different study types can offer different understandings; neither should be discounted without closer examination.
- RCTs provide an accurate understanding of the effect of a particular intervention in a well-defined patient group under “controlled” circumstances.
- Observational studies provide an understanding of real-world care and its impact, but can be biased due to uncontrolled factors.
- Observational studies differ in the types of databases used. These databases may lack clinical detail and contain incomplete or inaccurate data.
- Before accepting the findings from an observational study, consider whether confounding factors may have influenced the results.
- In this scenario, subgroup analysis was vital in clarifying both study designs; what is true for the many (e.g., overall, estrogen appeared to be detrimental) may not be true for the few (e.g., that for the younger post-menopausal woman, the benefits were greater and the harms less frequent).
- Carefully examine the generalizability of the study. Do the study’s patients and intervention match those under consideration?
- Observational studies can identify associations but cannot prove cause-and-effect relationships.
SOCR Resource Visitor
number 
