One-Way ANOVA vs. Multiple Comparisons

One-way ANOVA vs. Multiple Comparisons

One-way ANOVA is useful when we have three or more groups whose means we want to compare simultaneously. For two groups, we can resort to our usual two-sample comparison of means hypothesis test (Lesson 5). We could, instead of ANOVA, perform multiple two-sample comparison of means hypothesis tests, one for every possible pair of groups. For example, with three groups, 1, 2, and 3, we would have the following set of hypotheses to test:

Multiple, pairwise comparisons for three groups.
Test 1	$H_O: \mu_1 = \mu_2$ $H_A: \mu_1 \neq \mu_2 \ \text{or} \ \mu_1 > \mu_2 \ \text{or} \ \mu_1 < \mu_2$
Test 2	$H_O: \mu_1 = \mu_3$ $H_A: \mu_1 \neq \mu_3 \ \text{or} \ \mu_1 > \mu_3 \ \text{or} \ \mu_1 < \mu_3$
Test 3	$H_O: \mu_2 = \mu_3$ $H_A: \mu_2 \neq \mu_3 \ \text{or} \ \mu_2 > \mu_3 \ \text{or} \ \mu_2 < \mu_3$

In fact, the number of paired tests we would need to perform for $k$ groups is equal to:

$\frac{k(k-1)}{2}$

So for 4 groups, we would need 6 tests, and for 5 groups, we would need 10 tests. The number of tests we need can be quite cumbersome if we have several or more groups. Thus, ANOVA is an efficient way to test for equality in means across many groups, as it just has the one test:

H_O:

All population means,

\mu_i

, are equal for all

k

Type I Error

Furthermore, the multiple pairwise testing approach would increase the chance of committing Type I error (falsely rejecting the null hypothesis) because there would be multiple “attempts” of generating a p-value that falls below the significance level by random chance. For example, with the three tests above, at the 0.05 significance level, the expected number of tests that commit Type I error is $3 \times 0.05 = 0.15$ ; or, in other words, there’s a ~15% chance of having at least one test falsely reject the null.

Which Group?

Note that the ANOVA test doesn’t indicate which group, or groups, have means that aren’t equal to the others. The test just indicates that there is some significant inequality in the means. Usually, one can identify the standout group(s) through visualization (for example, a bar chart or comparative boxplots). If it is still not clear which group(s) are different, multiple comparison of means tests can be performed.

One-Way ANOVA vs. Multiple Comparisons