10  Late Phase Studies: Interim Analyses

Overview

Monitoring treatment effects and safety during the conduct of a late-phase clinical trial is essential to protect participants and ensure the scientific integrity of the trial. This lesson covers frequentist methods for interim reporting and the statistical frameworks used by Data Safety Monitoring Boards (DSMBs).

Objectives

Upon completion of this lesson, you should be able to:

1. Compare α spending functions for 3 group sequential methods for interim analysis.
2. Comment on the use of a group sequential method in a published statistical analysis.
3. Recognize a futility assessment and define conditional power.
4. List topics that should be covered in an interim report to an IRB.
5. List the advantages and disadvantages of a DSMB and describe who might compose the DSMB.
6. List the issues of concern to a DSMB in a typical clinical study.

10.1 Frequentist Methods: O’Brien-Fleming, Pocock, Haybittle-Peto

From a frequentist point of view, repeated hypothesis testing of accumulating data increases the type I error rate of a clinical trial. Therefore, the frequentist approach to interim monitoring of clinical trials focuses on controlling the type I error rate.

In most clinical trials, it is not necessary to perform a statistical analysis after each patient is accrued. In fact, for most multi-center clinical trials, interim statistical analyses are conducted only once or twice per year. Usually this frequency of interim analyses detects treatment effects nearly as early as continuous monitoring. The group sequential analysis is defined as the situation in which only a few scheduled analyses are conducted. Again, let’s focus more on the concepts than the statistical details.

Suppose that the group sequential approach consists of R analyses, and we let \(Z_1, \dots , Z_R\) denote the test statistic at the R times of hypothesis testing. So, we are accumulating data over time. We are adding to the dataset and analyzing the current set that has been collected. Also, we let \(B_1, \dots , B_R\) denote the corresponding boundary points (critical values). At the \(r^{th}\) interim analysis, the clinical trial is terminated with rejection of the null hypothesis if:

\[ |Z_r| \ge B_r, r = 1, 2, ... , R \]

The boundary points are chosen such that the overall significance level does not exceed the desired \(\alpha\). There are primarily three schemes for selecting the boundary points which have been proposed. These are illustrated in the following table for an overall significance level of \(\alpha = 0.05\) and for R = 2,3,4,5. The table is constructed under the assumption that n patients are accrued at each of the R statistical analyses so that the total sample size is \(N = nR\).

R	Interim Analysis Number	O'Brien-Fleming		Haybittle-Peto*		Pocock
		B	\(\alpha\)	B	\(\alpha\)	B	\(\alpha\)
2	1	2.782	0.0054	3.0	0.002	2.178	0.0294
	2	1.967	0.0492	1.960	0.0500	2.178	0.0294
3	1	3.438	0.0006	3.291	0.0010	2.289	0.0221
	2	2.431	0.0151	3.291	0.0010	2.289	0.0221
	3	1.985	0.0471	1.960	0.0500	2.289	0.0221
4	1	4.084	0.00005	3.291	0.00100	2.361	0.0182
	2	2.888	0.0039	3.291	0.00100	2.361	0.0182
	3	2.358	0.0184	3.291	0.00100	2.361	0.0182
	4	2.042	0.0412	1.960	0.0500	2.361	0.0182
5	1	4.555	0.000005	3.291	0.00100	2.413	0.0158
	2	3.221	0.0013	3.291	0.00100	2.413	0.0158
	3	2.630	0.0085	3.291	0.00100	2.413	0.0158
	4	2.277	0.0228	3.291	0.00100	2.413	0.0158
	5	2.037	0.0417	1.960	0.0500	2.413	0.0158

For example, if we plan one interim analysis and a final analysis, we will select the row in this table with R=2. Using these first two rows of the table, we find the critical values for the interim analysis and for the final analysis. If using O’Brien-Fleming approach, the interim analysis is conducted with bound 2.782 and final analysis with bound 1.967. On the other hand, had the choice been a Haybittle-Peto approach, the first test would be conducted with bound 3.0 and the final analysis at 1.96.

In another situation with three interim analyses and a final analysis, R=4. View the corresponding four rows in the middle of the table to determine critical values for each interim and the final analysis. Notice different approaches ‘spend’ or distribute the overall significance differently across the interim and final analyses.

The Pocock approach uses the same significance level at each of the R interim analyses. Of the three procedures described in the table, it provides the best chance of early trial termination. Many investigators dislike the Pocock approach, however, because of its properties at the final stage of analysis. For example, suppose R = 3 analyses are planned and that statistical significance is not attained at any of the analyses. Suppose that the p-value at the final analysis is 0.0350 (this is > 0.0221 found in the table for the Pocock approach). If interim analyses had not been scheduled, however, this p-value would be considered to provide a statistically significant result \(\left(cp = 0.0350 < 0.0500 \right)\).

The Haybittle-Peto (based on intuitive reasoning) and O’Brien-Fleming (based on statistical reasoning) approaches were designed to avoid this problem. On the other hand, these two approaches render it very difficult to attain statistical significance at an early stage.

Example 10.1 An example of the Pocock approach is provided in Pocock’s book (Pocock. 1983. Clinical Trials: A Practical Approach, New York, John Wiley & Sons). A trial was conducted in patients with non-Hodgkin’s lymphoma, in which two drug combinations were compared, namely cytoxan-prednisone (CP) and cytoxan-vincristine-prednisone (CVP). The primary endpoint was presence/absence of tumor shrinkage, a surrogate variable.

Patient accrual lasted over two years and 126 patients participated. Statistical analyses were scheduled after approximately every 25 patients. Chi-square tests (without the continuity correction) were performed at each of the five scheduled analyses. The Pocock approach to group sequential testing requires a significance level of 0.0158 at each analysis. Here is a table with the results of these analyses.

	Tumor shrinkage treatment: CP	Tumor shrinkage treatment: CVP	p-value
Analysis #1	3/14	5/11	p > 0.10
Analysis #2	11/27	13/24	p > 0.10
Analysis #3	18/40	17/36	p > 0.10
Analysis #4	18/54	24/48	0.05 < p < 0.10
Analysis #5	23/67	31/59	0.0158 < p < 0.10

Thus, the researchers were concerned that the CVP combination appeared to be clinically better than the CP combination (53% success versus 34% success), yet it did not lead to a statistically significant result with Pocock’s approach. Further analyses with secondary endpoints convinced the researchers that the CVP combination is superior to the CP combination.

How would you decide which of these group sequential methods to use? Since a major concern is the significance level at the final analysis and O’Brien-Fleming preserves close to the desired alpha for final analysis as well as allowing a strong result to terminate a trial, this has been a popular approach. The (REMATCH.pdf) clinical trial is a good example. Regardless of your choice, it is important to make it clear to study investigators the operating characteristics of any approach selected for interim analyses.

10.2 Alpha Spending Function approach

A few drawbacks to the group sequential approach to interim statistical testing include the strict requirements that:

1. The number of scheduled analyses, R, must be determined prior to the onset of the trial
2. There is equal spacing between scheduled analyses with respect to patient accrual.

The alpha spending function approach was developed to overcome these drawbacks: (DeMets DL, Lan KK, 1994, Interim analysis: The alpha spending function approach, Statistics in Medicine 13: 1341-1352.)

Let τ denote the information fraction available during the course of a clinical trial. For example, in a clinical trial with a target sample size, N, in which treatment group means will be compared, the information fraction at an interim analysis is \(\tau = \dfrac{n}{N}\), where n is the sample size at the time of the interim analysis. If your target sample size is 500, and you have taken measurements on 400 patients then \(\tau = .8\)

If the clinical trial involves a time-to-event endpoint, then the information fraction is \(\tau = \dfrac{d}{D}\), where D is the target number of events for the entire trial and d is the events that have occurred at the time of the interim analysis.

The alpha spending function, \(\alpha(\tau)\), is an increasing function. At the beginning of trial: \(t = 0\) and \(\alpha(t) = 0\); at the end of trial: \(t = 1\) and \(\alpha(t) = \alpha\), the desired overall significance level. In other words, every time an analysis is performed, part of the overall alpha is “spent”. For the \(r^{th}\) interim analysis, where the information fraction is \(\tau_r, 0 ≤ \tau_r ≤ 1, \alpha(\tau_r)\) determines the probability of any of the first r analyses leading to rejection of the null hypothesis when the null hypothesis is true. Obtaining the critical values consecutively requires numerically integrating the distribution function. A program is available in this module, along with the Demets-Lan paper.

As a simple example, suppose investigators are planning a trial in which patients are examined every two weeks over a 12-week period. The investigators would like to incorporate an interim analysis when one-half of the subjects have completed at least one-half of the trial. This corresponds to \(\tau = 0.25\).

A simple spending function that is a compromise between the Pocock and O’Brien-Fleming functions, is \(\alpha(\tau) = \tau\alpha, 0 ≤ \alpha ≤ 1\). This leads to a significance level of 0.012 at the interim analysis and a significance level of 0.04 at the final analysis (calculations not shown). Many variations of spending functions have been devised.

Regardless of whether a sequential, group sequential or alpha spending function approach is invoked, the estimates of a treatment effect will be biased when a trial is terminated at an early stage. The earlier the decision, the larger the bias. Intuitively, if the target sample size is 200 and the trial terminates after 25 patients because of a significant difference between treatment groups, you recognize the potential for a lot of bias in this situation. Are 25 patients a representative sample from the population?

10.3 Futility Assessment with Conditional Power; Adaptive Designs

As an alternative to the above methods, we might want to terminate a trial when the results of the interim analysis are unlikely to change after accruing more patients (futility assessment/curtailed sampling). It just doesn’t look like there could ever be a significant difference!

Unconditional power, as we have used in earlier sample size calculations is the probability of achieving a significant result at a pre-specified alpha under a pre-specified alternative treatment effect as calculated at the beginning of a trial. Conditional power is an approach that quantifies the probability of rejecting the null hypothesis of no effect once some data are available. If this quantity is very small, a conclusion can be reached that it would be futile to continue the investigation.

As a simple example, consider the situation in which we want to determine if a coin is fair, so the hypothesis testing problem is:

\[ H_0: p = Pr[\text{Heads}] = 0.5 \text{ versus } H_1: p = Pr[\text{Heads}] > 0.5. \]

The fixed sample size plan is to toss the coin 500 times, count the number of heads, X. But do we actually need to flip the coin 500 times? Using this futility assessment procedure we could reject \(H_0\) at the 0.025 significance level if:

\[ Z=\dfrac{X-250}{\sqrt{(500)(0.5)(0.5)}} \ge 1.96 \]

This is equivalent to rejecting \(H_0\) if X ≥ 272. Suppose that after 400 tosses of the coin there are 272 heads. It is futile to proceed further because even if the remaining 100 tosses yielded tails, the null hypothesis still would be rejected at the 0.025 significance level. The calculation of the conditional power in this example is trivial (it equals 1) because no matter what is assumed about the true value of p, the null hypothesis would be rejected if the trial were taken to completion.

You can also look at this in the other direction. Suppose that after 400 tosses of the coin there are 200 heads. The null hypothesis will be rejected if there are at least 72 heads during the remaining 100 tosses.

Even if p = 0.6 (arbitrary assignment), the conditional power is:

\[ Pr[X \ge 72 | n=100, p=0.6] \]

\[ = Pr\left[\dfrac{X-60}{\sqrt{(100)(0.6)(0.4)}} \ge \dfrac{72-60}{\sqrt{(100)(0.6)(0.4)}} \right] \]

\[ = Pr[X \ge 2.45] = 0.007 \]

The probability based on a standard normal table is calculated to be .007, a very small probability. Thus, it is futile to continue because there is such a small chance of rejecting \(H_0\).

Similarly, two clinical trial scenarios can be envisioned:

1. A trend in favor of rejecting \(H_0\) is observed at \(t < T\), with intervention \(>\) control. Compute conditional probability of rejecting \(H_0\) at T given current data. If probability is sufficiently large, one might argue trend not going to disappear.
2. A negative trend consistent with \(H_0\) at t . Compute conditional probability of rejecting \(H_0\) at end of trial at T given some alternative \(H_1\) is true. How large does the true effect need to be before the negative trend is reversed? If the probability of trend reversal is highly unlikely, termination might be considered.

Adaptive Designs

As we have seen, emerging trends may cause investigators to consider making changes in a study, such as increasing a sample size or terminating the study. An adaptive design which pre-specifies how the study design may change based on observed results can be useful. Group sequential strategies that we have already discussed are examples of a classical approach to some adaptation, that is early termination. In confirmatory trials, any adaptive design must maintain the statistical validity of the conclusions; control of Type I error is critical. On the other hand, adaptive designs for studies aimed at finding safe and effective doses emphasize strategies for assigning more participants to treatments with favorable responses and do not consider control of the type I error rate as important as identifying the most effective doses to enter confirmatory trials.

 View the table from Bhatt DL, Mehta C. Adaptive designs for clinical trials. New England Journal of Medicine. 2016;375(1):65–74. doi: 10.1056/NEJMra1510061. pmid:27406349, to see the strengths and weaknesses of different adaptive designs. In the paper, the authors examine 4 case studies of different adaptive designs used in confirmatory trials. 

10.4 Monitoring and Interim Reporting for Trials

Single-Center Trials

Here are some practical issues as they relate to single-center trials. Typically, an investigator for a single-center trial needs to submit an annual report to his/her IRB. The report should address whether the study is safe and whether it is appropriate to continue.

The report should include the following topics:

1. Compliance with governmental and institutional oversight,
2. Review of eligibility (low frequency of ineligible patients entering the trial),
3. Treatment review (most patients are adhering to the treatment regimen),
4. Summary of response,
5. Summary of survival,
6. Adverse events,
7. Safety monitoring rules (possibly statistical criteria for evaluating safety endpoints), and
8. Audit and other quality assurance reviews.

Multi-Center Trials

A multi-center trial is one in which there are one or more clinical investigators at each of a number of locations (centers). Obviously, multi-center trials are of great importance when the disease is not common and a single investigator is capable of recruiting only a handful of patients.

Advantages of a multi-center trial (Pocock, 1983) include the following:

1. Larger sample size and quicker patient accrual,
2. Broader interpretations of results because of the multiple participants involved in the study across various geographic regions, (this adds to external validity), and
3. Increased scientific merit of the trial because of collaborations among experienced clinical scientists involved in the design and implementation of the study.

Of course there is a downside. Disadvantages of a multi-center trial include the following:

1. Planning is more complex,
2. The study is going to be more expensive,
3. More effort to needs to go into ensuring compliance to clinical protocol across all centers,
4. Quality control must be implemented for taking measurements and recording data,
5. You need a data coordinating center (DCC) for storing, monitoring data and organizing investigators,
6. A need develops to keep all investigators involved and motivated,
7. Avoidance of passive investigators,
8. Compromise between quantity and quality of centers, and
9. A need for strong leadership.

The NIH requires a Data and Safety Monitoring Board (DSMB) to monitor the progress of a multi-center clinical trial that it sponsors. Although the FDA does not require a pharmaceutical/biotech company to construct a DSMB for its multi-center clinical trials, many companies are starting to use DSMBs on a regular basis.

There are several advantages that a DSMB provides, such as yielding a mechanism for protecting the interests and safety of the trial participants, while maintaining scientific integrity. The manner in which it is constructed should ensure that the DSMB is financially and scientifically independent of the study investigators so that decisions about early stopping or study continuation are made objectively. Depending on the circumstances, a DSMB may be composed of anywhere from three to ten experts in medicine, statistics, epidemiology, data management, clinical chemistry, and ethics. None of the study investigators should be a part of the DSMB. In addition, the DSMB should not be masked to treatment assignment when it is evaluating a clinical trial. Although investigators and statisticians may submit information and materials to the DSMB for their study, most of the deliberations made by the DSMB are kept confidential. The DSMB reports directly to the sponsor of the multi-center trial (the NIH or the company) and does not report to the investigators.

A DSMB typically examines the following issues when assessing the worth of a multi-center clinical trial:

1. Are the treatment groups comparable at baseline?
2. Are the accrual rates meeting initial projections and is the trial on its scheduled timeline?
3. Are the data of sufficient quality?
4. Are the treatment groups different with respect to safety and toxicity data?
5. Are the treatment groups different with respect to efficacy data?
6. Should the trial continue?
7. Should the protocol be modified?
8. Are other descriptive statistics, graphs, or analyses needed for the DSMB to make its decisions?

The major disadvantage of a DSMB holding the decision-making authority in a multi-center clinical trial, instead of the investigators, is that expertise may be sacrificed in order to maintain impartiality. Investigators gain valuable knowledge during the course of the trial and it is not possible to provide the DSMB with the totality of this knowledge. Nevertheless, the advantages of a DSMB seem to outweigh this disadvantage during the conduct of a multi-center trial.

A comprehensive book on the aspects of DSMBs is available: Ellenberg, SS. Fleming, TR. DeMets, DL. 2002, Data Monitoring Committees in Clinical Trials, New York, NY: Wiley.

10.5 Summary

In this lesson, among other things, we learned:

• Differentiate between valid and invalid reasons for interim analyses and early termination of a trial.
• Identify the characteristics of a sound plan for interim analysis.
• Understand the theoretical framework for a likelihood-based interim analysis.
• Compare and contrast the Bayesian approach to analysis with the frequentist approach.
• Recognize the general effects of the choice the prior on the posterior probability distribution from a Bayesian analysis.
• Compare α spending functions for 3 group sequential methods for interim analysis.
• Comment on the use of a group sequential method in a published statistical analysis.
• Recognize a futility assessment and define conditional power.
• List topics that should be covered in an interim report to an IRB.
• List the advantages and disadvantages of a DSMB and describe who might compose the DSMB.
• List the issues of concern to a DSMB in a typical clinical study.