We can use a randomization distribution to determine how likely our sample statistic is given that the null hypothesis is true. This probability is known as the p-value. The p-value is the proportion of samples on the randomization distribution that are more extreme than our observed sample in the direction of the alternative hypothesis. The p-value is compared to the alpha level (typically 0.05).
Making a Decision
If \(p > \alpha\) then we "fail to reject the null hypothesis" and conclude that there is not enough evidence of a difference in the population. This does not mean that the null hypothesis is true, it only means that we do not have sufficient evidence to say that it is likely false. These results are not statistically significant.
If \(p \leq \alpha\) then we "reject the null hypothesis" and conclude that there is a difference in the population. These results are statistically significant.