The NIH takes on the challenge of reproducibility: new requirements for 2016

Coming this fall, you will see changes to application instructions for NIH grant applications. Advances in science are built upon previous observations and findings: that’s why replication and corroboration of research results are essential to the scientific process. In response to continued failures in the reliability and reproducibility of published research, the NIH has been analyzing possible causes and exploring potential interventions for some time. A series of pilot projects were run over the past year, and the results of these projects were used to inform NIH’s new grant application instructions on Enhancing Reproducibility through Rigor and Transparency.

These new instructions will be published in the fall of 2015, and will apply to all applications submitted for the January 25, 2016 due date and beyond. The revised guidelines and review criteria will focus on four areas:

The scientific premise of the proposed research: Investigators will be expected to identify the strengths and weaknesses of prior research being cited as crucial support to the application and propose ways to address/improve them moving forward.

Rigorous experimental design for robust and unbiased results: Applicants will be required to provide the specifics of experimental design, methodology, analysis, interpretation and reporting results with full transparency and sufficient detail so that others can reproduce and extend the findings.

Consideration of relevant biological variables: Biological variables such as sex, age, weight, and underlying health conditions must now be considered in NIH-funded studies. Last week we reported on the new NIH guidelines on Consideration of Sex as a Biological Variable in NIH-funded Research. Considering sex as a biological variable is essential to achieve a full understanding of the sex-based differences in disease process and treatment response. Without this consideration, results of a single sex study cannot be reproduced when both sexes are examined, nor can results be used to build on future research that must consider sex. Both new sets of guidelines reference one another and are closely linked in terms of new study design and reporting requirements.

Authentication of key biological and/or chemical resources: Investigators will be required to regularly verify the identity and validity of key biological and/or chemical resources used in proposed studies to ensure quality and reproducibility. These resources include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics. The NIH is asking the research community for their participation in developing guidelines on authenticating different types of resources by transparently reporting on their authentication methods so a consensus can emerge.

Of particular note: Consideration of scientific premise, rigorous experimental design, and consideration of sex and other relevant biological variables must be included in the Research Strategy section. Page limits for this section will not change. Reviewers will evaluate scientific premise as part of the Significance section, and rigorous experimental design and consideration of sex and other biological variables as part of the Approach criteria. As such, evaluation of these three areas will be included in the assessment of overall impact. Authentication of key resources will be incorporated as a new attachment under the other research plan sections, and reviewers will be asked to comment on the plan but not consider it when scoring overall impact.

Read more about the evolution of these guidelines and the challenges in irreproducibility.