In Lesson 10 we learned that the purpose of a significance test is to answer the question "Does the null hypothesis provide a reasonable explanation for the data?" The question is addressed by carrying out a probability calculation assuming the null hypothesis is true.

With a small p-value we answer "No" - the null hypothesis is a poor explanation of the data.  

With a big p-value we answer "Yes" - the null hypothesis is a reasonable explanation of the data.

But, as with any statistical answer, we must understand the ways in which our answer could be wrong or misleading.

• There is a small chance that the null hypothesis provides a poor explanation of the data even when it is true (a type 1 error).  
• It is possible that the null hypothesis provides a reasonable explanation of the data even when it is false (a type 2 error).
• A significant test can be misleading if the sample size is so large that significant result is of little practical importance.
• A significant test can be misleading if the sample size is so small that an important effect goes undetected.
• There can be a very high chance of at least one type 1 error occurring when many significance tests are carried out.

Finally, we remember the caveats from Lessons 2 and Lesson 3 - that misleading interpretations of any statistical results can occur if we do not gather data in a thoughtful and unbiased manner to make sure samples are representative of populations and to make sure groups are similar in nature for comparative studies.  Another requirement of the statistical design of a study is to ensure the ethical treatment of subjects. Animal subjects should not be subjected to needless pain & suffering.  The safety of human subjects must be paramount; they should be taking part voluntarily and informed of the nature of the experiment they are taking part in and the possible risks entailed. Further, these ethical concerns must trump other aspects of statistical design.