The OHBM Replication Award

Goals

There is increasing concern about the reproducibility of research results.  Some aspects of brain imaging research, such as the relatively small sample sizes of many studies and the potentially context-dependent nature of brain mechanisms, raise particular concerns about the reproducibility of findings in the field.  One of the major avenues to enhance the reproducibility of a field is the replication of previous studies, but the incentives to perform and publish replications are much weaker compared to the incentives to publish novel research.

The OHBM Replication Award aims to recognize the best published replication study of the past year.  This award highlights the OHBM’s commitment to reproducibility in neuroimaging research and helps to begin to reshape the incentives towards replication.  The award competition was announced at the 2016 Annual Meeting, and then the first award will be given at the 2017 meeting.  The award is sponsored by the Laura and John Arnold Foundation.

What is a replication study?

A replication study is a repetition of a published study procedure with minor changes to variables assumed not to be important for the measured phenomena (this depends on the experiment, but could include demographics, scanner model, visual stimuli delivery system, analysis strategy, etc.). Replication studies usually (but not always) have a larger sample size than the original study for appropriate statistical power, and are performed by a different team than the original study (but planning of a replication study can benefit from involvement of the original researchers). Even though minor changes between the original study and its replication are inevitable they should be minimised as much as possible.

What makes a good replication study?

A good replication study is a study that addresses an important topic for the neuroimaging community. The study needs to show highest standards of experimental design, data collection and statistical analysis. Openness of experimental procedures, data processing and statistical analyses as well as public availability of collected data also speak in favor of a good replication study. Authors should also show maturity and thoughtfulness when discussing results of the replication and should give the original authors an appropriate platform to comment on the replication.
Pre-registration can also improve the quality of a replication study – especially when the pre-registration report has been reviewed by the authors of the original study. Pre-registration can lead to more predictable publishing process through the Registered Reports mechanism. Pre-registered studies can also be eligible for additional US$1000 provided through the Open Science Framework Preregistration Challenge.

Process/Criteria

  1. The award amounts to US$2000.
  2. Criteria for nomination would include:
    1. The study must meet the above definition of a replication study be relevant to the field of human neuroimaging
    2. The study must be published in a peer-reviewed journal or deposited into a recognized preprint archive (such as biorXiv and arXiv) before 1st of May 2017.
  3. A committee convened by OHBM would select the winner from the nominated papers. The award committee will include members of Program Committees from the past three years. Each member will have one vote. The number of votes counts be made publicly available.
  4. The initiators of the award (Chris Gorgolewski, Russell Poldrack, Jean Baptiste Poline, David Kennedy and Thomas Nichols) are not eligible to receive the award.
  5. The award would be presented at the opening ceremony of the 2017 Annual Meeting. The check will be handed to the corresponding author on the winning paper.
  6. More can be found here