Printer-friendly version

Pr(V > 0) is called the family-wise error rate or FWER. It is easy to show that if you declare tests significant for \(p < \alpha\) then FWER ≤  \(min(m_0\alpha,1)\).

The most commonly used method which controls FWER at level \(\alpha\) is called Bonferroni's method. It rejects the null hypothesis when \(p < \alpha / m\). (It would be better to use \(m_0\) but we don't know what it is - more on that later.)

The Bonferroni method is guaranteed to control FWER, but it has a big problem. It greatly reduces your power to detect real differences. For example, suppose the effect size is 2 and you are doing a t-test, rejecting for p < 0.05. With 10 observations per group, the power is 99%. Now suppose you have 1000 tests, and use the Bonferroni method. That means that to reject, we need p < 0.00005. The power is now only 29%. If you have 10 thousand tests (which is small for genomics studies) the power is only 10%.

Sometimes the "Bonferroni-adjusted p-values are reported". They are just:

\(p_b=min(mp,1)\).

Another simple more powerful but less popular method uses the sorted p-values:

\[p_{(1)}\leq p_{(2)} \leq \cdots  \leq p_{(m)}\]

Holmes showed that the FWER is controlled with the following algorithm:

Compare \(p_{(i)}\) with \(\alpha / (m-i+1)\). Starting from i = 1, reject until \(p_{(i)}\) is greater.

The most significant test must therefore pass the Bonferroni criterion. However, if it is significant, the next most significant is tested at a less stringent level. Heuristically, after rejecting the most significant test, we conclude the \(m_0 \leq m-1\) and use \(m-1\) for the next correction, and so on sequentially.

The Holmes method is more powerful than the Bonferroni method, but it is still not very powerful. We can also compute "Holmes-adjusted p-values" \(p_{h(i)} = min((m-i+1)p_{(i)},1)\).