Big problems for common fMRI thresholding methods

December 8, 2015

A new preprint has been posted to the ArXiv that has very important implications and should be required reading for all fMRI researchers.  Anders Eklund, Tom Nichols, and Hans Knutsson applied task fMRI analyses to a large number of resting fMRI datasets, in order to identify the empirical corrected “familywise” Type I error rates observed under the null hypothesis for both voxel-wise and cluster-wise inference.  What they found is shocking: While voxel-wise error rates were valid, nearly all cluster-based parametric methods (except for FSL’s FLAME 1) have greatly inflated familywise Type I error rates.  This inflation was worst for analyses using lower cluster-forming thresholds (e.g. p=0.01) compared to higher thresholds, but even with higher thresholds there was serious inflation.  This should be a sobering wake-up call for fMRI researchers, as it suggests that the methods used in a large number of previous publications suffer from exceedingly high false positive rates (sometimes greater than 50%).  

Figure 1 from Eklund et al., showing substantial inflation of familywise error rates for most common cluster-based thresholding methods.

They also examined the commonly used heuristic correction (what they call “ad-hoc cluster inference”) of p=0.001 and a cluster extent threshold of 80 mm^3.  This method showed a shockingly high rate of false positives, up to 90% familywise error in some cases.  Hopefully this paper will serve as the deathknell for such heuristic corrections.

Figure 7 from Eklund et al., showing massive inflation of familywise Type I error rate using ad-hoc clustering inference.

Another set of serious concerns were raised about the simulation-based methods implemented in AFNI’s 3DclustSim tool:

Firstly, AFNI estimates the spatial group smoothness differently compared to SPM and FSL. AFNI averages smoothness estimates from the first level analysis, whereas SPM and FSL estimate the group smoothness using the group residuals from the general linear model [39]. The group smoothness used by 3dClustSim may for this reason be too low (compared to SPM and FSL, see Figure 10); the variation of smoothness over subjects is not considered. Secondly, a 15 year old bug was found in 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The effect of the bug was an underestimation of how likely it is to find a cluster of a certain size (in other words, the p-values reported by 3dClustSim were too low).

I have been disturbed in recent years to see an increasing number of papers that perform most of their analyses using FSL or SPM, but then use the AFNI tool for multiple comparison correction.  It is now clear why this tool was so attractive to so many researchers: It systematically undercorrects for multiple comparisons, leading to “better results” (i.e. inflated false positives).  As an aside, this also points to the critical need for reporting of software versions in empirical papers; without this, it is impossible to know whether results obtained using AFNI suffered from this bug or not.  

To the degree that there is a star of the Eklund et al. paper, it was nonparametric testing using permutation tests (for example, as implemented in the FSL randomise tool), though even those methods did not escape unscathed.  The standard objection to nonparametric testing has been the substantial computational load of these methods. However, Eklund et al. also raise concerns about their ability to control Type I error rate, particularly in the one-sample t-test case (as opposed to the two-sample group comparison case where it did well).  It appears that the assumption of symmetrically distributed errors required by the one-sample permutation approach may be violated, leading to inflated error, though these violations appear to be relatively minor compared to those seen with the parametric methods.

What are we to take away from this?  First, the results clearly show that cluster-based inference with traditional parametric tools should always use cluster-forming thresholds no less stringent than p<0.001, as error inflation was much worse for less stringent cluster-forming thresholds.  Second, the results strongly suggest that permutation testing is likely the best approach to prevent false positives without being overly stringent (e.g., using voxelwise inference rather that cluster inference). The platform being developed by the Center for Reproducible Neuroscience should make this much easier for researchers to apply through the use of high-performance computing.  If one must use a parametric method, then FSL’s FLAME 1 with a cluster-forming threshold appears to be the best bet, though it is sometimes quite conservative.  Third, the results should cause substantial suspicion about the use of AFNI’s 3DClustSim tool, as even the new version with the bug fix doesn’t bring false positive rates into line.  
More generally, the fact that it took our field more than twenty years to discover that some of our most common methods are badly flawed is bracing.  At the same time, it is only through the availability of massive open data repositories (including 1000 Functional Connectomes and OpenfMRI) that this kind of analysis could be done.  We need much more work of the kind presented in the Eklund et al. paper in order to better understand how we can ultimately ensure that the results of fMRI studies are reproducible. 


  1. formerstudentofknutsson 7 years

    Hey come on. At least get Knutssons name right. It’s the least you could do.

  2. Chris Gorgolewski 7 years

    Thanks for pointing this out – we have fixed the typo.

  3. 5 years

    Toledo tem um carro para cada 2,5 habitantes).

  4. Os críticos têm, portanto, um tanto de razão.

  5. Palavras-Chave: Dignidade da Pessoa Humana.

  6. Clientes, faço este porta tiaras da cor que desejar.

  7. Lançamento: Nova Coleção de Tiaras de Faixa.

  8. Se escreve no bósforo Aberta na Barga ai a seguir.

  9. Gostaria de saber Maria Inez faz para vender.

  10. Peça às crianças que cada uma faça um desenho.

  11. Catarina 5 years

    A palavra de Deus esta acima da igreja de Deus”.

  12. Sonho com baralho indica que você deve tentar
    a sorte.

  13. Creches: atividades para crianças de zero a seis

  14. amit 5 years

    good article thank you so much

  15. Blog muito legal! O seu tema foi personalizado por você?
    ou baixou de algum lugar? O seu tema com alguns pequenos
    ajustes faria meu blog bombar. Por favor,
    se puder dizer onde você conseguiu seu tema compartilha comigo, ok?
    Muito obrigado!

  16. Domenic 4 years

    Hurrah, that’s what I was seeking for, what a information!
    present here at this blog, thanbks admin off
    this web page.

  17. Bridgett 4 years

    At this time it souinds lіke Expression Engine is tthe beѕt blogging platform avаilable гight now.
    (from what I’νe read) Is that what you arе ᥙsing on your blog?

  18. Natalie 4 years

    Este Texto me convenceu de uma coisa, eu absolutamente percebo que este
    site precisa de muita atenção por parte das pessoas, porque é realmente muito
    interessante. Eu provavelmente vou retornar.
    Os textos são muito bem escritos. Obrigado pela informação e Parabéns.

  19. Estou gostando dos seus artigos, eu acompanho esse web site a algum tempo, Parabéns pela iniciativa
    de falar sobre este tema conteúdo de qualidade. Coloquei Teu site nos meus favoritos.
    Obrigado! 🙂

  20. cute kittens 4 years

    What’s up, its pleasant piece of writing regarding media print, we all be aware of media is a impressive source of facts.

  21. forex forum 4 years

    I am actually grateful to the holder of this website who has shared this great paragraph
    at at this place.

  22. Reviews, 4 years

    obviously like your website but you need to take a look at the spelling on several of
    your posts. Several of them are rife with spelling issues and I
    in finding it very troublesome to inform the reality on the other hand I will certainly come
    back again.

  23. Quality articles is the secret to attract the people
    to pay a quick visit the web page, that’s what this web page is providing.

  24. ngopi 4 years

    I have fun with, result in I discovered just what I was taking
    a look for. You have ended my four day long hunt!
    God Bless you man. Have a nice day. Bye

  25. keluaran hk 3 years

    advanced poker calculator v4 advanced poker strategy articles aed vs pkr af poker definition after dark poker nottingham agen poker qq online indonesia agile poker deck agro poker definition ai and poker ai factory poker keluaran hk

  26. Birutoto 3 years

    what up, nice sir

  27. moscabianca 3 years

    il miglior sito per trovare e vendere in italia il migliori annunci economici”

  28. il miglior sito per comprare e vendere casa in italia

  29. DonneCercaUomo 3 years

    my dream is one day study in this university

  30. MoscaRossa 3 years

    NIce blog dude, you use wordpress?

  31. Thank you for sharing this content, it was really use full. keep doing it

  32. crepe 3 years

    su moscabianca ce qualcuno cercando quello che vuoi vendere

  33. masosn 3 years

    in oikia puoi vendere la tua casa

  34. Heitor 2 years

    Great blog

  35. DonnacercaUomo 2 years

    Thanks for sharings. Great Article!

  36. This is a very good tip especially to those new to the blogosphere.
    Brief but very accurate info… Thanks for sharing this
    one. A must read article!

  37. A must read article!

  38. very interesting , good job and thanks for sharing

  39. vicky 2 years

    Informative article. Thanks for sharing.

  40. Thanks for sharing.

  41. TechBloggerPro 2 years

    very interesting. Thanks for sharing.

  42. Kwabit 2 years

    am an upcoming dj and audio producer, i love to make my own beats and listen to great music produced by other great dj’s and producers, that’s why i decided to become a producer… i also started my own podcast at mp3paw which helps other people search and download podcasts from youtube. I know there are many music streaming sites like Spotify, Apple Music, and Google Play Music. But these a credit card and a very reliable internet connection. sometimes you wish to download mp3 to your phone. 

  43. thanks for sharing this information boss…

  44. MP3 Paw 1 year

    am an upcoming dj and audio producer, i love to make my own beats and listen to great music produced by other great dj’s and producers, that’s why i decided to become a producer… i also started my own podcast at mp3 paw which helps other people search and download podcasts from youtube. I know there are many music streaming sites like Spotify, Apple Music, and Google Play Music. But these a credit card and a very reliable internet connection. sometimes you wish to download mp3 to your phone. 

  45. . 1 year

    Thanks for sharing information on fMRI thresholding methods…

  46. 免费色情影片 12 months

    Thank you for sharing this article, very interesting!

  47. Richi Machinery 10 months

    The cost of the biomass pellet production line is identified by the resources made use of.

  48. Richi Machinery 10 months

    The great selling in Indonesia biomass pellet production line rate depends upon your basic materials as well as outcome dimension. Firstly, you require to know what your resources are.

  49. Richi Machinery 10 months

    Recognizing your resources is about the output. The outcome of the biomass pellet mill device is different and also the model is different, so the price is likewise various. The biomass pellet production line cost is not only a matter of reporting a number, yet it is needed to formulate a plan and also quotation according to your requirements.

  50. Richi Machinery 10 months

    For example, if the raw materials are branches and also trunks with high wetness content, they ought to be chipped – crushed – dried out – granulated – packed – shipped. It mainly depends on what basic materials are made use of. 350 type biomass pellet machine equipment 38,000, design: 350 type 300 ~ 500 kg per hour, 450 type 500 ~ 800 kg, 560 type 1 ~ 1.5 heaps, 600 kind 2 loads, 850 type 3 ~ 4 heaps. 5T/H wood pellet factory in Austria

  51. Richi Machinery 10 months

    The raw materials are prefabricated sawdust or wood chips as well as the wetness web content of the products is 10%-15%. A factory provided new technology timber pellet maker suffices. If your basic materials are unrefined logs, branches, etc need to be shattered by a timber chipper and biomass crusher, after that dried out in a clothes dryer (if the dampness exceeds 15%, it needs to be dried out), then the major equipment is granulated, and afterwards the product packaging device loads this collection of production lines. Richi Machinery

  52. Thank you for sharing this interesting piece of science!

  53. Emilee Amador 8 months

    Wow I really enjoyed to read that all i will always follow You thanks for the great article. It conatains so good knowledge. Full of informative article.

  54. kalyan final ank 6 months

    thanks for sharing this information sir…

  55. Celebs Pics 6 months

    thanks for sharing this information like a loot

  56. Agario 6 months

    this sort of mods, then maintain reading. We incorporate included everything on your own could have to have in direction of discover.
    Did You Try Our Agario Unblocked Mod?

  57. Incontri Trento 4 months

    That is a really informative article. Thanks for the share!

  58. Mp3download 4 months

    That is a really informative article. Thanks for the share!

  59. Mp3downloads 4 months

    That is a really informative article. Thanks for the share!

  60. Incontri Torino 3 months

    That is a really informative article. Thanks for the share!

  61. Emilee Amador 3 months

    Ihop Near Me is a Breakfast Giant That Offers Delicious Food At An Affordable Price. If You’re Looking For a Ihop Near Me
    Quick, Convenient Meal, Then Ihop is The Place For You. From Pancakes To Waffles and Everything In Between, Ihop Has You Covered.

  62. John imma 3 months

    Download mp3 at high quality from MP3 Juices music search engine.

  63. Moscarossa 2 months

    visit the best site for ads Escort moscarossa Milano

  64. Incontri Torino 2 months

    That is a really informative article. Thanks for the share!

  65. DonnaCerca 2 weeks

    Nice one, though it might take some time to get along with all of these things

  66. That is a really informative article. Thanks for the share!