As you are likely well aware, new review criteria and application instructions concerning rigor and reproducibility for many NIH and AHRQ grants go into effect May 25th. As the deadline nears for the June R01 cycle, we are getting a lot of questions about how to manage these changes. While this is still very much unsettled territory, NIH has released some clarifications and resources (as well as a graphical representation). The following represents our best intelligence to date.
First, let’s look at the new formal review criteria:
- Is there a strong scientific premise?
- Have the investigators demonstrated adequate plans to address relevant biological variables (in particular sex) in vertebrate animal and human research?
- Is the research rigorous: Are there strategies presented to ensure that the approach is robust and unbiased?
To a degree, your research has always been evaluated on these criteria, but now they are explicit whereas before they were implicit. Reviewers will be asked to look at these criteria throughout the application as well as in a new one-page attachment, “Authentication of Key Biological and/or Chemical Resources.”
So, if you’re writing a grant, what are you supposed to do differently? The two primary changes are these:
- address the new review criteria into your Significance and Approach sections
- fill out the attachment.
Reviewers will be paying attention to the scientific premise when they evaluate your supporting data—both in the Significance section and in your preliminary data (whether you include these data in your Approach or earlier in the application). They will focus on scientific rigor and the consideration of biological variables in the Approach section.
Let’s dive a little deeper into the review criteria:
The scientific premise of the application
Here, you’re being asked to describe the strengths and weaknesses of the prior research that you’re citing—for example, an evaluation of the rigor of previous experimental design (yours or those of others). If you wrote R01 applications back in the days of the 25-page application, this requirement may remind you of the 3-page “Background and Significance” section. The reviewer instructions do state that this information will be sought after in the Significance section—but the page lengths and headings for applications have not changed. You may want to write a longer Significance section: Up to now, there’s been an evolving practice to make the Significance section—which should describe why your science and health problems are important to solve—comparatively short. The details were saved for the Approach section. The instructions for scientific premise, however, direct reviewers to evaluate this in the Significance section. Thus, we expect the form of the Significance section to evolve further.
The scientific rigor of the proposed experiments
How well controlled are your experiments? What is their statistical power? The goal here is to create reproducible and unbiased results, so transparency will also be important—with the caveat that different fields will have different standards for what counts as rigorous. Well-written grants usually have this kind of information, so this may not be qualitatively different from what you have done before. If you’re curious, NIH has provided some samples of what they consider rigor (scroll down to Resources).
Consideration of relevant biological variables
Your approach will also be evaluated for how well you explain the effects of biological variables such as sex, age, weight, and underlying health conditions for vertebrate animal and human subjects research. Reviewers are being asked to focus on sex, in particular, because it has to been overlooked in much published research (for many years, by design). You are being asked to incorporate consideration of sex into both research design and analyses. And if you’re just studying one sex, you’ll need to have a compelling justification (previous literature, preliminary data, and so on). This criterion will probably cause the most confusion and angst, among grant writers and reviewers alike.
Authentication of key biological and/or chemical resources
This is the one change to the formal process, and it consists of filling out a one-page document. Here, you will explain how your key biological and chemical resources will be evaluated for their quality.
These resources should be integral to your research, and they may have properties that affect the data. They may include cell lines, specialty chemicals, antibodies, and other biologics. They may be something you created or a cell line you purchased—and even these may differ in different labs or over time. So your task is to explain how you will ensure these resources are as consistent as possible. Ideally, you are doing this already—for example, checking antibodies via Western blotting—but the difference here is that you’ll list out the process of authentication in a separate document. Applicants are being counseled to plan to independently verify even commercial resources—don’t presume that a cell line you purchased from a vendor is free of problems.
One note of caution: reviewers are being asked to look for ‘overloading’ of applications, so don’t try to put experimental detail in this document. They don’t want the data demonstrating authentication–they want to know your plans for doing this. It may feel a bit redundant with proposed controls in the approach—the jury is out at this point in terms of whether this section will ultimately overlap with or supplement the Approach. Finally, the review of this section, at least for the time being, will not count in your overall significance score. If it’s inadequate, it will be handled administratively prior to the release of funds.
If you’re writing a career award or training grant, you can learn from the experience of others: similar changes in these applications are being drafted and are expected to go into effect in 2017. In the meantime, you can study the FAQs: there will no doubt be more questions.