The Permutation Test is an intuitive statistical test that relies on computation, rather than complex formulae and assumptions. Consequently, it is far easier to understand and implement than the “classical” statistical alternatives that are generally taught in quantitative methods classes. A detailed description of permutation testing (written in the context of teaching statistics) is given by Cobb (2007). A shorter descrition is given by Allen Downey in his well known blog articles there is only one test (2011) and there is still only one test (2016). A nice visual illustration of how it works is given is given by Jared Weiber (2019).

Most statistical analyses use a ‘significance level’ of 0.05. 0.05 = 1/20, meaning fewer than one in twenty of your data points ,ust be sufficietly different…

This tool is intended to provide a simple interface that allows students to use a permutation test to compare the mean of two groups of data (sample X and sample Y).

Hypotheses

There is no difference between the values in Sample X and Sample Y (i.e., they are overall neither larger nor smaller).

Data

Now paste your data for the two samples from Excel into the boxes below.

For a relatively straightforward discussion on the interpretation of p-values (including a list of common misinterpretations), see Dahiru (2008). For a more detailed exploration of the challenges associated with the misinterpretation of statistical testing, see Ioannidis (2005). For a discussion on the importance of testing for effect size (not just significance), see see Sullivan and Feinn (2012).