When the null hypothesis of independence is rejected, the nature of dependence---its direction and magnitude---needs to be measured. A statistic that measures the direction and magnitude of the relationship between two variables is called an effect size or a measure of association.

One of the most intuitive measures of association is the difference in proportions, which compares the relative frequency of important characteristics between two groups. For example in the Vitamin C study, we want to know if the probability of a member of the placebo group contracting cold is the same as a probability of a member of the ascorbic group contracting cold.

Regarding \(Z\) as a response and \(Y\) as explanatory variable, the difference in proportions for a \(2 \times 2\) table is

\(\delta =P(Z=1|Y=1)-P(Z=1|Y=2)= \dfrac{\pi_{11}}{\pi_{1+}}-\dfrac{\pi_{21}}{\pi_{2+}} = \pi_{1|1}-\pi_{1|2}\)

where \(\pi_{1|1}\) is the probability of "success" (e.g., "cold"), given row 1, and \(\pi_{1|2}\) is the probability of "success", given row 2. Recall that these are the conditional probabilities we already described for the Vitamin C example. Thus, the probability of "failure" (e.g., "no cold"), given row 1 is \(\pi_{2|1}\) and \(\pi_{1|1}+ \pi_{2|1}=1\). Similarly, we can find conditional probabilities, given row 2.

In social sciences and epidemiology, these are sometimes referred to as "risk" values. That is, we may refer to the probability that a person gets a cold, given that he/she took a placebo pill, as the risk of such an event. Furthermore, for diagnostic tests, the conditional probability that the diagnostic test is positive, given that the subject has a disease, is called sensitivity. The conditional probability that the diagnostic test is negative, given that the subject does NOT have a disease, is called specificity.

Finally, because \(\delta\) is a function only of the parameters of \(P(Z | Y )\), likelihood-based inferences will be the same, regardless if calculations assume Poisson sampling or multinomial sampling.

The natural estimate of \(\delta\) is the difference in the conditional sample proportions:

\(\hat{\delta}=d=\dfrac{n_{11}}{n_{1+}}-\dfrac{n_{21}}{n_{2+}}=p_{1|1}-p_{1|2}\)

Properties

• It takes values between \(-1\) and \(+1\),
• If variables are independent, the difference in the proportions equals 0.

This is the maximum-likelihood estimate (MLE) because under product-multinomial sampling, the numerators are independent binomials:

For the Vitamin C example, the estimated or sample difference of proportions of getting a cold is \(d=17/139 - 31/140 = 0.12 - 0.22 = -0.10\).

 Question: Is this difference "big" or "small"?

If \(n_{1+}\) and \(n_{2+}\) are large, the estimate \(d\) is approximately normal with variance

\(\displaystyle V(d)=\left[\frac{ \frac{\pi_{11}}{\pi_{1+}} (1-\frac{\pi_{11}}{\pi_{1+}})} {n_{1+}} + \frac{\frac{\pi_{21}}{\pi_{2+}} (1-\frac{\pi_{21}}{\pi_{2+}})} {n_{2+}} \right] \)

This expression follows from the fact that if \(X_1\) and \(X_2\) are independent random variables, then \(Var(X_1 − X_2) = Var(X_1) + Var(X_2)\). Both \(X_1\) and \(X_2\) here are Bernoulli random variables.

Plugging in the estimates, we get the estimate of the variance

\(\hat{V}(d)=\dfrac{n_{11}n_{12}}{(n_{1+})^3}+\dfrac{n_{21}n_{22}}{(n_{2+})^3}\)

which is used for computing the standard errors and confidence intervals.

A large sample, \((1- \alpha) 100\%\) CI for the Vitamin C example is

\(SE(d)=\sqrt{\hat{V}(d)}=\sqrt{\dfrac{0.12\times 0.88}{139}+\dfrac{0.22\times 0.78}{140}}=0.045\)
\(\Rightarrow -0.10\pm 1.96\times 0.045=(-0.19,-0.01)\)

Interpret

We are 95% confident that the true difference in proportions of people getting cold given the placebo or a vitamin C is somewhere between 1% and 19%. Note that the value of 0 (corresponding to no difference) does not fall within that boundary.

Under the null hypothesis of no difference, \(H_0: \delta = 0\), the rows of the table can be pooled to get an estimate of the common proportion, \( P(Z = 1 | Y = 1) = P(Z = 1 | Y = 2)\).

The pooled estimate is

\(\hat{\pi}=\dfrac{n_{11}+n_{21}}{n_{1+}+n_{2+}}\)

Under \(H_0: \delta = 0\), a more efficient estimate of variance \(V (d)\) is

\(\dfrac{\hat{\pi}(1-\hat{\pi})}{n_{1+}}+\dfrac{\hat{\pi}(1-\hat{\pi})}{n_{2+}}\)

and the test statistic

\(z=\dfrac{d}{\sqrt{\frac{\hat{p}(1-\hat{p})}{n_{1+}}+\frac{\hat{p}(1-\hat{p})}{n_{2+}}}}\)

is approximately distributed as \(N(0,1)\). Many elementary textbooks in statistics use this test to compare two proportions. For the Vitamin C example,

\(z=\dfrac{31/140-17/139}{\sqrt{\frac{48}{279}\times \frac{231}{279}\times (\frac{1}{140}+\frac{1}{139})}}=2.19\)

This value is significant at the 0.05 level, so if data are taken at face value, we could conclude that the proportion of colds in vitamin C group is smaller than in the placebo group. The result is consistent with confidence interval inference.

It is useful to note that the null hypothesis \(H_0:\delta = 0\) is equivalent to independence. Using conditional probabilities, this corresponds to

\(\pi_{j|1} = \pi_{j|2}\)

Thus we can test \(\delta = 0\) by the usual \(X^2\) or \(G^2\) test for independence in a \(2\times 2\) table already discussed in the previous sections. In fact, we can show that \((\text{z-statistic})^2\) above is algebraically equal to \(X^2\). So a two-sided test based on comparing the z-statistic to a \(N(0, 1)\) distribution is identical to comparing \(X^2\) from the test of independence to a chi-squared distribution with \(\chi^2_1\). In the Vitamin C example, note that \(X^2 = 4.81 \approx (2.19)^2\).

Even though the difference of two proportions is very easy to interpret, one problem with using \(\delta\) is that when \(Z = 1\) is a rare event, the individual probabilities \(P(Z = 1 | Y = 1)\) and \(P(Z = 1 | Y = 2)\) are both small, i.e., close to zero. The absolute value of \(\delta\) will be close to zero even when the effect is strong. In the following sections, we study two other common measures of association which compare the relative value of the proportions, rather than the absolute values.