When teaching my Applied Statistics class, I’ll often use the following xkcd comic to reinforce the meaning of statistical significance.
The idea that’s being communicated is that, when performing an hypothesis test, the observed significance level is the probability that the null hypothesis is correct due to dumb luck as opposed to a real effect (the alternative hypothesis). So if the significance level is really about and the experiment is repeated about 20 times, it wouldn’t be surprising for one of those experiments to falsely reject the null hypothesis.
In practice, statisticians use the Bonferroni correction when performing multiple simultaneous tests to avoid the erroneous conclusion displayed in the comic.
I'm a Professor of Mathematics and a University Distinguished Teaching Professor at the University of North Texas. For eight years, I was co-director of Teach North Texas, UNT's program for preparing secondary teachers of mathematics and science.
View all posts by John Quintanilla
Mean Green Math 