Power Analysis Section
After completing a statistical test, conclusions are drawn about the null hypothesis. In cases where the null hypothesis is not rejected, a researcher may still feel that the treatment did have an effect. Let's say that three weight loss treatments are conducted. At the end of the study, the researcher analyses the data and finds there are no differences among the treatments. The researcher believes that there really are differences. While you might think this is just wishful thinking on the part of the researcher, there MAY be a statistical reason for the lack of significant findings.
At this point, the researcher can run a power analysis. Recall from your introductory text or course, that power is the ability to reject the null when the null is really false. The factors that impact power are sample size (larger samples lead to more power), the effect size (treatments that result in larger differences between groups will have differences that are more readily found), the variability of the experiment, and the significance of the type 1 error.
As a note, the most common type of power analysis are those that calculate needed sample sizes for experimental designs. These analyses take advantage of pilot data or previous research. When power analysis is done ahead of time it is a PROSPECTIVE power analysis. In our example, using a power analysis after the experiment is done is a retrospective power analysis.
So back to our example. Typically we want power to be at 80%. Again, power represents our ability to reject the null when it is false, so a power of 80% means 80% of the time we do this correctly. The converse of this is that 20% of the time we risk not rejecting the null when we really should be rejecting the null.
Using our greenhouse example, we can run a retrospective power analysis (just a reminder we typically don't do this unless we have some reason to suspect the power of our test was very low). This is one analysis where Minitab is much easier and still just as accurate as SAS so we will use Minitab to illustrate this simple power analysis.
Minitab: Power Analysis
You can follow along with the screen shots below to access the Power menu in Minitab.
Since we have a one-way ANOVA we select this test (you can see there are power analysis for many different tests and SAS will allow even more complicated options)
Next the dialogue box will ask for the four elements described above.
When you look at our filled in dialogue box you notice we have not entered a value for power. This is because Minitab will calculate whichever box you leave blank (so if we needed sample size we would leave sample size blank and fill in a value for power. From our example, we know the number of levels is 4 because we have four treatments. We have six observations for each treatment so sample size is 6. The value for the maximum difference in the means is 8.2 (we simply subtracted the smallest mean from the largest mean, and the standard deviation is 1.747. Where did this come from? The MSE represents the error in the model. We also consider the MSE the variability in the model. To get the standard deviation we simply take the square root of the MSE (in this case the MSE was about 3).
After we click OK we get the following output:
If you follow this graph you see that power is on the y axis. Since we are interested in determining what the power of the green house example is, all we need to do is fin the little red dot on the graph. It is hard to find, but if you look carefully the red dot corresponds to a power of 1. That is very unusual, but can be easily explained that the greenhouse data was put together to show differences.
We can ask the question, what about differences among the treatment groups, not considering the control? All we need to do is modify some of the input in Minitab.
Note the differences here as in the previous screen shot. We now have 3 levels because we are only considering the three treatments. The maximum differences among the means and also the standard deviation are also different.
The output now is much easier to see:
Here we can see the power is lower than including the control. The main reason for this decrease is because the difference among the means is smaller.
You can experiment with the power function in Minitab to provide you with sample sizes, etc. for various powers. Below is some sample output when we ask for various power curves for various sample sizes, a kind of "what if" scenario.
Just as a reminder, power analyses are most often performed BEFORE an experiment is conducted, but occasionally, a power analysis can provide some evidence as to why significant differences were not found.