3.2 - Controlled Clinical Trials Compared to Observational Studies

Medical research, as a scientific investigation, is based on careful observation and theory. Theory directs the observation and provides a basis for interpreting the results. The strength of the evidence from a clinical study is proportional to amount of the control of bias and variability when the study was conducted as well as the magnitude of the observed effect. Clinical studies can be characterized as uncontrolled observations, observational comparative and controlled clinical trials.

Case reports and case-series are uncontrolled observational studies.

Case Report

A case report only demonstrates that a clinical event of interest is possible. In a case report, there is no control of treatment assignment, endpoint ascertainment, or confounders. There is no control group for the sake of comparison. The report is descriptive in nature, not a formal statistical analysis.

Case reports are useful in generating hypotheses for future testing. For example, a physician may report that a patient in his practice, who was taking a specific anorexic drug, developed primary pulmonary hypertension (PPH), a rare condition that occurs in 1-2 out of every million Americans. Is this convincing evidence that the anorexic drug causes PPH?

Case Series

A case series carries more weight than a single case report, but cannot prove the efficacy of a treatment. Case series and case reports are susceptible to large selection biases. Consider the example of laetrile, an apricot pit extract that was reputed to cure cancer. Seven case series were reported; the strength of evidence from these studies has been summarized by the US National Cancer Institute (NCI). While a proportion of patients may have experienced spontaneous remission of cancer, rigorous testing in controlled environments was never performed. After an estimated 70,000 patients had been treated, the NCI undertook a retrospective analysis of laetrile only to decide no definite conclusions supporting anti-cancer activity could be made (Special Report on Laetrile: The NCI Laetrile Review). The Cochrane review on laetrile (2015), states, “ there is no reliable evidence for the alleged effects of laetrile or amygdalin for curative effects in cancer patients.” Based on a series of reported cases, many believed laetrile would cure their cancer, perhaps refusing other effective treatments, and subjecting themselves to adverse effects of cyanide, for many years, this continued for many years with anti-tumor efficacy of laetrile unsupported while associated adverse effects were coming to light.

Database Analysis

A database analysis is similar to a case series but may have a control group, depending on the data source. The source and quality of the data used for this secondary analysis are key. If the analysis attempts to evaluate treatment differences from data in which treatment assignment was based on physician and patient discretion, nonrandomized and open-label, bias is likely.

Databases are best used to study patterns with exploratory statistical analyses. For example, the NIH sponsored a database analysis of interstitial cystitis (IC) during the 1990s. This consisted of data from over 400 individuals with IC who underwent various and numerous therapies for their condition. The objective of the database analysis was to determine if there were patterns of treatments that may be effective in treating the disease. (Rovner et al. 2000).

As another example, in the case of genomic research, specific data mining tools have been developed to search for patterns in large databases of genetic data, leading to the discovery of particular candidate genes.

Epidemiologic Study

An epidemiologic study is often a case-control or a cohort design, both comparative observational studies. An observational study lacks the key component of an experiment, namely, control over treatment assignment. Commonly these designs are used in assessing the influence of risk factors for a disease. Subjects meeting entrance criteria may have been identified through a database search. The choice of the control group is a crucial design component in observational studies.

Case-Control Study

In a case-control study, the investigator identifies cases (subjects with the disease) and controls (subjects without the disease) and retrospectively assesses some type of treatment or exposure. Because the investigator has selected the cases and controls, relative risk cannot be calculated directly from a case-control study.

In addition, levels of treatment or exposure may be recorded based on a subject’s recall of events that occurred many years previously, thus recall bias,(systematic differences in accuracy or completeness of recall) can affect the study results.

Prospective Cohort Study

In a prospective cohort study, individuals are followed forward in time with subsequent evaluations to determine which individuals develop into cases. The relationship of specific risk factors that were measured at baseline with the subsequent outcome is assessed. The cohort study may consist of one or more samples with particular risk factors, called cohorts. It is possible to control some sources of bias in a prospective cohort study by following standard procedures in collecting data and ascertaining endpoints. Since the subjects are not assigned risk factors in a randomized manner, however, there may remain covariates that are confounded with a risk factor. Sometimes, a particular treatment group (or groups) from a randomized trial is followed as a cohort, providing a cohort in which the treatment was assigned at random.

Prospective studies tend to have fewer design problems and less bias than retrospective studies, but they are more expensive with respect to time and cost.

An example of a case-control study: A cardiologist identifies 36 patients currently in his practice with a specific form of cardiac valve disease. He identifies another group of relatively healthy patients and matches two of them to each of the patients with cardiac valve disease according to age (± 5years) and BMI (± 2.5). He plans to interview all 36 + 72 = 108 patients to assess their use of diet drugs during the past ten years.

A classic example of a cohort study: U.S. National Heart Lung and Blood Institute Framingham Heart Study

Piantodosi (2005) lists the following conditions for convincing non-experimental comparative studies:

1. The treatment of interest occurs naturally.
2. The study subjects provide valid observations for the biological question.
3. The natural history of the disease with standard therapy, or in the absence of therapy, is known.
4. The effect of the treatment is large enough to overshadow random error and bias.
5. Evidence of efficacy is consistent with biological knowledge.

Controlled Clinical Trial

A controlled clinical trial contains all of the key components of a true experimental design. Treatments are assigned by design; administration of treatment and endpoint ascertainment follows a protocol. When properly designed and conducted, especially with the use of randomization and masking, the controlled clinical trial instills confidence that bias has been minimized. Replication of a controlled clinical trial, if congruent with the results of the first clinical trial, provides verification.