Statistical power

Many of the test statistics calculated on the other pages report a p-value. p-values are associated with type I errors. In particular, they are the probability (under the null hypothesis) that a given result would have been achieved by random chance. Therefore, a result is only considered statistically significant if its p-value is below a predetermined threshold.

While p-values are used to minimize the probability of a type I error, statistical power is related to type II errors. Power is the probability of correctly rejecting the null hypothesis. For example, statistical power answers the following question "Assuming exercise has a positive impact on mood, how likely is the experiment to come to the correct conclusion?".

The power of an experiment depends on a number of factors:

Sample size

The more subjects there are in a trial, the greater its statistical power.

Significance level

The significance level is the cut-off point for determining statistical significance. If a p-value is less than the significance level, the null hypothesis is rejected. Decreasing this value (e.g. from the conventional value of 0.05 to a more stringent value of 0.01) decreases power, since the alternative hypothesis is less likely to be accepted, even if it is true.

Effect size

Intuitively, it is easier to detect strong phenomena than weak ones. For example, if exercise has a dramatic impact on mood, an experiment to test this hypothesis will have high power since it will be easy to find the effect. On the other hand, if the impact of exercise is real but small, there will be low power, meaning there is a risk that the experiment will not find a stastically significant result (see this page for information on computing effect sizes).

Directionality

Power is greater for a one-tailed test than for a two-tailed test (see here for information on directionality).

t-test power calculator

Use this calculator to compute the power of an experiment designed to determine if two data sets are significantly different from each other.

F-test power calculator

Use this calculator to compute the power of an experiment designed to determine if more than two data sets are significantly different from each other.

Note: This calculator assumes sphericity (i.e. nonsphericity correction = 1).