Modified Tukey-Kramer multiple comparisons procedure
Scans from Y. Hochberg and A.C. Tamhane, "Multiple Comparison Procedures," Wiley, New York, 1987.
While conducting background research into the procedure that Matlab uses to perform a multi-factor ANOVA, I came across this reference material in a book borrowed from the library. Am posting a selection online for interested users.
I wanted to find out how Matlab calculates its error bars for the plots generated by the multcompare function. (See the Matlab forum discussion thread.)
I eventually gathered that these 'error bars' are not used to describe the variance associated with each data point (based on taking the mean across observations); rather, they represent an 'equal width interval' that is applied across an entire 'family' of comparisons and which is used solely for statistical analysis.
The interval size is common to all the observed data points and is used as a criterion when evaluating pairwise differences during the ANOVA- if confidence intervals for two observed data points fail to overlap during a particular pairwise comparison, then one concludes that a significant difference is present. This value, which remains fixed across comparisons, is what you see in the plots generated by multcompare.
The actual calculation of interval width, as performed in Matlab, is based on a modified ('improved') version of the Tukey-Kramer procedure (the TK-procedure is popularly implemented due to its simplicity and 'nearly accurate' control of the FWE [familywise error rate]).
I haven't burrowed deep enough to fully understand the math behind the equations, but the qualitative difference is that the error bars in published images typically describe the variance associated with individual data points, whereas the error bars plotted in multcompare are calculated for the sake of statistical analysis and are based on the pooling of data across treatments and groups.
The formula implemented by Matlab is given in equation 3.32.
The introductory chapter provides a useful overview of the definitions of familywise, per-family and per-comparison error rates, so I've included that as well.
Chapter 3. Section 3.2.4 (pp. 96-98)
Procedures Designed for Graphical Display