11.8 - Special Methods of Analysis

One of the most difficult statistical tasks is assessing the precision of an estimator, i.e., determining the variance of an estimator can be more difficult than determining the appropriate estimator. In complicated situations, the bootstrap method can be applied to estimate the variance of an estimator. The bootstrap is essentially a resampling plan.

For example, suppose an investigator collects a sample of N observations, denoted as \(Y_1, Y_2, \dots , Y_N\), and wants to estimate the median, \(η\), and get an expression for its variance. If the investigator does not want to make any assumptions about the distribution of the sample, then an explicit expression for the variance of the sample median does not exist.

The bootstrap can be used to construct a variance estimate of the sample median.

The bootstrap process consists of constructing B data sets, each with N observations, from the original data set. Each bootstrap sample is constructed by sampling with replacement from the original data set. This means that when constructing a bootstrap sample, N observations are generated one at a time where each \(Y_i\) has \(dfrac{1}{N}\) probability of being selected. Here is an example of the resampling of the original data:

Original sample: 17, 25, 16, 32, 27, 19, 25, 23, 22, 30

Thus, for \(b = 1, \dots,B\), the bootstrap sample \(Y_{b1}, Y_{b2}, \dots , Y_{bN}\)  is constructed and the sample median within the bth bootstrap sample is formed as:

\(\hat{\eta}_b= \text{ median }(Y_{b1},Y_{b2}, ... , Y_{bN})\)

From the B estimates of the median we construct the estimated variance as:

\(S^2=\dfrac{1}{B-1}\sum_{b=1}^{B}(\hat{\eta}_b-\bar{\eta})^2 \text{ where}\bar{\eta}=\frac{1}{B}\sum_{b=1}^{B}\hat{\eta}_b\)

to get a sense about how these medians are varying over the 100 samples. The variance estimate can then be used to construct a Z statistic for hypothesis testing, i.e.,

\(Z=(\hat{\eta}-\eta)/S\)

Some statisticians at first were leery of this approach, essentially using one sample to create many others samples from this original, i.e., "pulling oneself up by your bootstraps". The bootstrap process, however, over time has shown to have sound statistical properties. The disadvantage of this approach has to do with the random selection with replacement which could result in slight variations in results. The FDA, for instance, requires definitive results. This is simply a non-parametric approach for estimating the variance of a sample.

Clinical trial data provide the opportunity for exploratory analyses, which are analyses in addition to those specified by the primary objectives in the protocol. A trial design is usually not well-suited for all of the exploratory analyses that are performed, so the results may not have much validity.

The results from exploratory analyses should not be regarded as confirmatory, but rather as hypothesis-generating for future research. As a general rule, the same data should not be used both to generate a new hypothesis and to test that hypothesis. Unfortunately, many investigators do not follow this principle.

Data in sufficient quantity and detail can be made to yield some effect. A few statistical sayings attest to this, such as "the data will confess to anything if tortured enough." It has been well documented that increasing the number of hypothesis tests inflates the Type I error rate. Exploratory analyses typically fall into this category and the chances of finding statistically significant results, when none truly exists, can be very high.

Subset analyses are a form of exploratory analysis that are very popular with clinical trial data. For example, after performing the primary statistical analyses, the investigators might decide to compare treatment groups within certain subsets, such as male subjects, female subjects, minority subjects, subjects over the age of 50, subjects with serum cholesterol above 220, etc. Unless it is planned ahead of time, such analyses should remain exploratory.