The F-statistic

ANOVA breaks up the total variability of the sample values into two kinds of variability: 1.) the variability within the groups and 2.) the variability between the groups. If the variability between the groups is large compared to the variability within the groups, we can say that the groups have different means. In equation form, this idea is noted:

$$SS_{Total} = SSG + SSE$$

The terms are defined as:

• $SS_{Total}$ : the total sum of squared deviations. This takes the distance of each data point from the mean of all data, squares that distance (so they’re all positive), and adds them all up. Thus, it measures the TOTAL VARIABILITY of the data.

• $SSG$: the sum of squared deviations for groups. This takes the distance of each group’s mean from the mean of all data (ungrouped), squares that distance, and adds the values up. Thus, it measures the VARIABILITY BETWEEN GROUPS.

• $SSE$: the sum of squares for error. For each group, this takes the distance of each data point in the group from the mean of that group, squares that distance, and adds them all up. Thus, it measures the VARIABILITY WITHIN THE GROUPS.

Our goal with the ANOVA test is to compare the variability BETWEEN groups to the variability WITHIN groups, however we can’t properly compare $SSG$ to $SSE$ directly since these use different amounts of data. So we need to look at the “mean square” for each:

$$\begin{align} \text{Mean square for groups: } MSG &= \frac{SSG}{k-1} \\ \text{Mean square for error: } MSE &= \frac{SSE}{n-k} \end{align}$$

Here, $n$ is the total sample size (number of data values in all groups together) and $k$ is the number of groups. The denominators, $k-1$ and $n-k$ represent the “degrees of freedom” (“df” or “dof” for short) for each term. We can then find the F-statistic as:

$$F = \frac{MSG}{MSE}$$

which effectively compares the variability BETWEEN groups (numerator) to the variability WITHIN groups (denominator). Thus, if the null hypothesis is true and the group means are actually equal, we expect the F-statistic to be about 1. Larger values of the F-statistics indicate a larger $MSG$ relative to $MSE$, and thus a difference in means (or at least one mean).

The F-distribution and p-value

The p-value for our ANOVA test depends on the F-statistic above and the degrees of freedom. The degrees of freedom determines the shape of the probability density function for F (an example is pictured below). The p-value is then the area under the probability density function and GREATER THAN the F-statistic calculated from the sample data (shaded area in the figure below).

In this figure, the curve is the probability density function for F with degrees of freedom k − 1 = 4 and n − k = 95. The p-value is the shaded area under the curve and greater than the F-statistic value of 2.5, in this example.

Credit: © Penn State is licensed under CC BY-NC-SA 4.0

FAQ