next up previous
Next: Practical Goals Up: Permutation Testing Made Practical Previous: Introduction

The Resampling Method

The overly conservative nature of the Bonferroni correction was noted by Blair and Karniski [1994]. As an alternative they proposed the permutation test. The permutation test is one of a family of methods known collectively as resampling procedures [Efron 1982]. Taking advantage of the speed of modern computing systems, these methods construct an explicit, nonparametric model of the actual distribution from which a set of observations has been drawn.

The reasoning behind the permutation test can be developed from an examination of more traditional tests. A common parametric method of fMRI data analysis involves correlating the observed time series at each voxel with an ideal time series. This ideal series can be viewed as an indicator variable that describes the experimental condition at the time of each observation. The simplest form of such a series is a square wave whose value is one during the experimental condition and zero during the control condition.

Suppose that the null hypothesis is true, that is, the experimental condition has no effect on the value of the fMRI time series. In that case (and assuming that the effect of autocorrelation is negligible), the pairing between time points in the observed series and time points in the ideal series is of no consequence; any ordering of the observations will produce a similarly low correlation value.

The permutation test uses such re-orderings (or `resamplings') of the observations to construct an empirical estimate of the distribution from which the test statistic has been drawn. On each of a large number (typically 10,000) of iterations, the sequence of the observations is randomised, and the test statistic is calculated with respect to the data in this randomised sequence. Each iteration produces one point in the empirical distribution. The probability that the test statistic will be less than or equal to a certain value k under the null hypothesis can then be computed as the rank of k within the empirical distribution, divided by the number of points in the distribution [Blair & Karniski 1994].


next up previous
Next: Practical Goals Up: Permutation Testing Made Practical Previous: Introduction