Since its inception in 1990, functional Magnetic Resonance Imaging (fMRI) has revolutionised the way we think about the brain. In association with other techniques, it provides invaluable clues to understand brain disorders, like psychosis or dementia. At its heart, fMRI is a medical imaging technique that allows scientists to look at where oxygenated blood is being directed around the brain – if a brain area is being used for a particular task, it will need more oxygen. So by extension, fMRI allows us to look at how information is processed in the brain, and is one of the few techniques we have to look directly into a person’s brain while they are thinking.
However, this past summer has seen a series of alarmist headlines about the technique cropping up in the media. There have been claims that ‘Tens of Thousands of fmri brain studies may be flawed’, and that a ‘bug in fMRI software calls 15 years of research into question’ (also here, and up to 20 years here), and even that fMRI as no scientific value (‘the great brain scan scandal’) or that much of what we know about the brain may be wrong. But is fMRI research really in such a sorry state?
The furor finds its origins in a study in the July issue of the Proceeding of the National Academy of Science by Anders Eklund, Thomas Nichols and Hans Knutsson. Despite being in use for some 25 years, some of the most common statistical analyses used on fMRI data haven’t been fully assessed for accuracy. To address this, Eklund and his team set out to measure the false positive rates of standard task fMRI methods. A false positive is when we wrongly conclude an effect is present, when in reality there is no effect. A pregnancy test that comes out positive when the woman is not pregnant is one example of this type of error. When it comes to fMRI data, a false positive could be that an area of the brain is declared activated by a task, when in reality it is not. Eklund’s work is essential because it allows us to ascertain the scientific validity of fMRI – in other words, how much we can be sure that we’re measured what we think we’re measuring.
Once signals from the MRI machine are recorded, many analysis steps take place and ‘maps’ of activation can be created, showing which areas of the brain are in use, and which are not. Given the complexity of the method, it is impossible to completely eliminate false positives from these maps, so researchers use statistical tests that try to control errors to a desired (usual small) rate. The key finding from the PNAS paper is that, one method typically used in fMRI analysis can give a much higher false positive rate than expected – in some cases, a chance of up to 70% that the software might produce a false positive when 5% was expected.
This might sound disastrous, but it’s not actually completely new. As Neuroskeptic has pointed out, this study is the culmination of work that has been going on for a number of years, with the novel contribution here being that Eklund’s team illustrated the problem empirically, based on openly available data, identified the origin of the problems, and provided an alternative solution to standard methods.
So what can we really conclude from the study? First, it is not true that tens of thousands of studies are affected. Not all fMRI studies use the methods called into question. As pointed out on one of the author’s blog, around 3500 studies (or less than 10% of existing fMRI studies) used the method called into question. It is equally important to note that this also does not mean the discoveries from all 3500 of these studies are false; the effects in some studies could be so strong to be detected even despite this problem, or the findings in others could be weak but have subsequently been replicated. Unfortunately, we have no way of knowing the status of each of these studies, as data from published studies was not shared or archived for reanalysis.
Another headline claim was that 15 years of research are called into question because of a bug. The bug reported in the paper concerns an error in one piece of software (AFNI, the least used of the three tested) leading to higher error rates than expected (this bug was fixed in 2015). As the authors note, this bug contributed to inflated false positive rates, but is only a small factor relative to the problems common to all three software packages.
In short, for those who see these problems as the death knell for fMRI, they are simply missing the point of this line of work - which is to make the science more robust. When controversial issues like these have arisen in the past, far from throwing the baby out with the bath water, the scientific community has instead responded strongly and raised standards. There is no doubt this is going to be the same this time around.
Find out more:
Eklund et al. (2016). Cluster-failure: why fMRI inferences for spatial extent have inflated false-positive rate. PNAS 113, 7009-7009
A technical comment on the Eklund et al. paper from the Organization of Human Brain Mapping blog.
About the authors:
Dr Cyril Pernet is an Academic Fellow in Neuroimaging Sciences at the University of Edinburgh.
Professor Tom Nichols is a Wellcome Trust Senior Research Fellow in Basic Biomedical Science at the University of Warwick.
