<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.1 20151215//EN" "JATS-archivearticle1.dtd">
<article xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" dtd-version="1.1"><front><journal-meta><journal-id journal-id-type="nlm-ta">elife</journal-id><journal-id journal-id-type="publisher-id">eLife</journal-id><journal-title-group><journal-title>eLife</journal-title></journal-title-group><issn pub-type="epub" publication-format="electronic">2050-084X</issn><publisher><publisher-name>eLife Sciences Publications, Ltd</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="publisher-id">67173</article-id><article-id pub-id-type="doi">10.7554/eLife.67173</article-id><article-categories><subj-group subj-group-type="display-channel"><subject>Research Article</subject></subj-group><subj-group subj-group-type="heading"><subject>Computational and Systems Biology</subject></subj-group></article-categories><title-group><article-title>Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health</article-title></title-group><contrib-group><contrib contrib-type="author" corresp="yes" id="author-33346"><name><surname>Lauer</surname><given-names>Michael S</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0002-9217-8177</contrib-id><email>Michael.Lauer@nih.gov</email><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="fn" rid="con1"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" id="author-224955"><name><surname>Doyle</surname><given-names>Jamie</given-names></name><xref ref-type="aff" rid="aff2">2</xref><xref ref-type="fn" rid="con2"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" id="author-224956"><name><surname>Wang</surname><given-names>Joy</given-names></name><xref ref-type="aff" rid="aff3">3</xref><xref ref-type="fn" rid="con3"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" id="author-224957"><name><surname>Roychowdhury</surname><given-names>Deepshikha</given-names></name><xref ref-type="aff" rid="aff3">3</xref><xref ref-type="fn" rid="con4"/><xref ref-type="fn" rid="conf1"/></contrib><aff id="aff1"><label>1</label><institution>Office of the Director, National Institutes of Health</institution><addr-line><named-content content-type="city">Bethesda</named-content></addr-line><country>United States</country></aff><aff id="aff2"><label>2</label><institution>Division of Clinical Innovation, National Center for Advancing Translational Sciences</institution><addr-line><named-content content-type="city">Bethesda</named-content></addr-line><country>United States</country></aff><aff id="aff3"><label>3</label><institution>Office of Extramural Research, National Institutes of Health</institution><addr-line><named-content content-type="city">Bethesda</named-content></addr-line><country>United States</country></aff></contrib-group><contrib-group content-type="section"><contrib contrib-type="editor"><name><surname>Rosen</surname><given-names>Cliff J</given-names></name><role>Reviewing Editor</role><aff><institution>Maine Medical Center Research Institute</institution><country>United States</country></aff></contrib><contrib contrib-type="senior_editor"><name><surname>Zaidi</surname><given-names>Mone</given-names></name><role>Senior Editor</role><aff><institution>Icahn School of Medicine at Mount Sinai</institution><country>United States</country></aff></contrib></contrib-group><pub-date date-type="publication" publication-format="electronic"><day>13</day><month>04</month><year>2021</year></pub-date><pub-date pub-type="collection"><year>2021</year></pub-date><volume>10</volume><elocation-id>e67173</elocation-id><history><date date-type="received" iso-8601-date="2021-02-03"><day>03</day><month>02</month><year>2021</year></date><date date-type="accepted" iso-8601-date="2021-04-08"><day>08</day><month>04</month><year>2021</year></date></history><permissions><ali:free_to_read/><license xlink:href="http://creativecommons.org/publicdomain/zero/1.0/"><ali:license_ref>http://creativecommons.org/publicdomain/zero/1.0/</ali:license_ref><license-p>This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/">Creative Commons CC0 public domain dedication</ext-link>.</license-p></license></permissions><self-uri content-type="pdf" xlink:href="elife-67173-v2.pdf"/><abstract><p>A previous report found an association of topic choice with race-based funding disparities among R01 applications submitted to the National Institutes of Health (‘NIH’) between 2011 and 2015. Applications submitted by African American or Black (‘AAB’) Principal Investigators (‘PIs’) skewed toward a small number of topics that were less likely to be funded (or ‘awarded’). It was suggested that lower award rates may be related to topic-related biases of peer reviewers. However, the report did not account for differential funding ecologies among NIH Institutes and Centers (‘ICs’). In a re-analysis, we find that 10% of 148 topics account for 50% of applications submitted by AAB PIs. These applications on ‘AAB Preferred’ topics were funded at lower rates, but peer review outcomes were similar. The lower rate of funding for these topics was primarily due to their assignment to ICs with lower award rates, not to peer-reviewer preferences.</p></abstract><kwd-group kwd-group-type="author-keywords"><kwd>funding</kwd><kwd>government</kwd><kwd>policy</kwd><kwd>disparities</kwd><kwd>peer review</kwd></kwd-group><kwd-group kwd-group-type="research-organism"><title>Research organism</title><kwd>None</kwd></kwd-group><funding-group><funding-statement>No external funding was received for this work.</funding-statement></funding-group><custom-meta-group><custom-meta specific-use="meta-only"><meta-name>Author impact statement</meta-name><meta-value>An analysis of peer review and funding outcomes of NIH research applications shows that funding disparities of topics preferred by African American Black investigators are not due to peer review preferences or biases.</meta-value></custom-meta></custom-meta-group></article-meta></front><body><sec id="s1" sec-type="intro"><title>Introduction</title><p>Data recently reported by <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> from the National Institutes of Health ('NIH') suggest that part of the well-documented funding disparity (<xref ref-type="bibr" rid="bib2">Ginther et al., 2011</xref>) affecting African-American Black ('AAB') principal investigators ('PIs') may be related to the topic of their applications. The authors of that report (including the first author of this report) found that topic choice accounted for over 20% of the disparity and wondered whether biases on the part of peer reviewers might explain why some application topics fare less well when submitted to the NIH for consideration of funding. In other words, peer reviewers might prefer certain topics over others, and specifically may be biased against applications focused on topics preferred by AAB PI’s.</p><p>However, peer review outcomes are not the only determinant of funding. Across the agency, grants are not simply awarded in order of peer review scores (<xref ref-type="bibr" rid="bib13">Taffe and Gilpin, 2021</xref>). One reason is that applications submitted to the NIH are not just submitted to NIH; they are assigned to one of 24 grant-issuing institutes or centers ('ICs') that in turn make decisions about which applications to fund. The proportion of applications funded (or 'award rate') varies across ICs; therefore, we can think of the NIH process as not one competition hinging entirely on peer review but rather as 24 separate competitions. The variability of award rates relates to differences in number of applications each IC receives, available funds, and IC priorities.</p><p><xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> did not account for IC assignment or variation in IC-specific award rates. It is possible that the apparent link between topic choice and funding disparities may reflect differences in IC assignment, since ICs receive applications according to alignment with their stated mission. For example, applications focusing on cancer epidemiology are more likely to be assigned to the National Cancer Institute while those focusing on basic human biology are more likely to be assigned to the National Institute of General Medical Sciences. If award rates at the National Institutes of General Medical Sciences are higher than at the National Cancer Institute, it might appear that NIH 'prefers' basic human biology over cancer epidemiology. While the former topic does fare better with a higher likelihood of funding, this may be largely because of different IC award rates as opposed to differences in how the topics are received by peer reviewers.</p><p>We therefore re-analyzed the data from <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> focusing on topic-specific peer review outcomes according to AAB PI preferences and on the possible role of IC assignment in application outcomes. To minimize well-documented biases related to repeated looks by the peer review system on individual proposals (from resubmissions [<xref ref-type="bibr" rid="bib6">Lauer, 2017</xref>] or competing renewals [<xref ref-type="bibr" rid="bib5">Lauer, 2016</xref>]) we focus on de novo applications submitted to the NIH for the first time.</p></sec><sec id="s2" sec-type="materials|methods"><title>Materials and methods</title><p>These analyses are based on R01 applications submitted to NIH between 2011 and 2015. <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> described in detail NIH peer review processes and the 'Word2vec' algorithm <xref ref-type="bibr" rid="bib8">Mikolov et al., 2013</xref> used to designate a topic for each application. Briefly, each application is assigned to a peer review group; approximately 2% of applications are withdrawn prior to review for administrative reasons. After a preliminary pre-meeting review, approximately half are deemed to be potentially meritorious and are therefore discussed during a formally convened meeting. After the meeting, each discussed application receives a priority score ranging from 10 (best) to 90 (worst); many, but not all, applications also receive a 'percentile ranking' to account for differences in how individual peer review groups calibrate their scores.</p><p>Applications are not only assigned to peer review groups; they are also assigned to ICs. The Center of Scientific Review (CSR) Division of Receipt and Referral makes assignments (<xref ref-type="bibr" rid="bib1">CSR, 2021</xref>) based on IC interests, specific Funding Opportunity Announcements (as appropriate), and sometimes on applicant request. Some applications receive a dual assignment (that is, two ICs with one primary and one secondary), but changes in primary IC assignment occur less than 2% of the time. ICs ultimately make decisions about which applications to fund, with funding decisions based on peer review scores, strategic priorities, and availability of funds (<xref ref-type="bibr" rid="bib11">NIH, 2021c</xref>).</p><sec id="s2-1"><title>Data and descriptive measures</title><p>To eliminate biases due to prior reviews and to maximize the likelihood that the analysis data frame reflects comparable observations with independently distributed variables, we focused on applications coming to the NIH for the first time; in other words, we excluded resubmissions (<xref ref-type="bibr" rid="bib6">Lauer, 2017</xref>) and competing renewals (<xref ref-type="bibr" rid="bib5">Lauer, 2016</xref>) which are known to systematically fare better on peer review. Each application was designated to one of 148 topics identified by the Word2vec algorithm (<xref ref-type="bibr" rid="bib8">Mikolov et al., 2013</xref>). <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> stated there were 150 topics, but subsequently discovered data processing errors for two topics yielded 148 for the re-analysis.</p><p>For each topic, we counted the number of applications of submitted by AAB PIs. We assessed AAB preference for each topic <italic>t</italic> by calculating the proportion of all AAB applications within that topic, that is<disp-formula id="equ1"><label>(1)</label><mml:math id="m1"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mrow><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>B</mml:mi></mml:mrow></mml:msub><mml:mo>=</mml:mo><mml:mrow><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:munderover><mml:mo largeop="true" movablelimits="false" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>n</mml:mi></mml:munderover><mml:mrow><mml:mi>a</mml:mi><mml:mo>⁢</mml:mo><mml:mi>a</mml:mi><mml:mo>⁢</mml:mo><mml:mi>b</mml:mi></mml:mrow></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>/</mml:mo><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:munderover><mml:mo largeop="true" movablelimits="false" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>N</mml:mi></mml:munderover><mml:mrow><mml:mi>a</mml:mi><mml:mo>⁢</mml:mo><mml:mi>a</mml:mi><mml:mo>⁢</mml:mo><mml:mi>b</mml:mi></mml:mrow></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow></mml:mrow></mml:mrow></mml:math></disp-formula>where <italic>i</italic> refers to individual applications, <italic>n</italic> is the number of applications in topic <italic>t</italic>, <italic>aab</italic> is an indicator of whether application <italic>i</italic> had an African American or Black contact PI (1 if yes, 0 if no), and <italic>N</italic> is total number of applications (across all topics). For each IC, we calculated award rates <italic>A</italic> as number of applications funded divided by number of applications assigned, that is<disp-formula id="equ2"><label>(2)</label><mml:math id="m2"><mml:mrow><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>I</mml:mi><mml:mo>⁢</mml:mo><mml:mi>C</mml:mi></mml:mrow></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:msubsup><mml:mo largeop="true" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>m</mml:mi></mml:msubsup><mml:mi>a</mml:mi></mml:mrow><mml:mi>m</mml:mi></mml:mfrac></mml:mrow></mml:math></disp-formula>where <italic>m</italic> is the number of applications received by the IC and <italic>a</italic> is an indicator of whether an application <italic>i</italic> was awarded (1 if yes, 0 if no). Similarly, for each topic we calculated awards rates as<disp-formula id="equ3"><label>(3)</label><mml:math id="m3"><mml:mrow><mml:msub><mml:mi>A</mml:mi><mml:mi>t</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:msubsup><mml:mo largeop="true" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>n</mml:mi></mml:msubsup><mml:mi>a</mml:mi></mml:mrow><mml:mi>n</mml:mi></mml:mfrac></mml:mrow></mml:math></disp-formula>where again <italic>a</italic> is an indicator of whether an application <italic>i</italic> was awarded (1 if yes, 0 if no) and <italic>n</italic> is the number of applications in the topic. We estimated a predicted award rate for each topic by designating the probability for funding for each application <italic>i</italic> as its IC award rate (in other words, if an application <italic>i</italic> was assigned to NEI, the probability of funding was 0.15). Thus, the predicted topic award rate was<disp-formula id="equ4"><label>(4)</label><mml:math id="m4"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>A</mml:mi><mml:mo stretchy="false">^</mml:mo></mml:mover><mml:mi>t</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:msubsup><mml:mo largeop="true" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>n</mml:mi></mml:msubsup><mml:mover accent="true"><mml:mi>a</mml:mi><mml:mo stretchy="false">^</mml:mo></mml:mover></mml:mrow><mml:mi>n</mml:mi></mml:mfrac></mml:mrow></mml:math></disp-formula>where <inline-formula><mml:math id="inf1"><mml:mover accent="true"><mml:mi>a</mml:mi><mml:mo stretchy="false">^</mml:mo></mml:mover></mml:math></inline-formula> is the award rate for the IC to which each application <italic>i</italic> is assigned and <italic>n</italic> is the number of applications in the topic.</p><p>For each topic, we calculated two peer review outcomes. The proportion of applications making it to discussion (or 'discussed') was<disp-formula id="equ5"><label>(5)</label><mml:math id="m5"><mml:mrow><mml:msub><mml:mi>D</mml:mi><mml:mi>t</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:msubsup><mml:mo largeop="true" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>n</mml:mi></mml:msubsup><mml:mi>d</mml:mi></mml:mrow><mml:mi>n</mml:mi></mml:mfrac></mml:mrow></mml:math></disp-formula>where <italic>d</italic> is an indicator of whether an application <italic>i</italic> was discussed (1 if yes, 0 if no) and <italic>n</italic> is the number of applications in the topic. For those applications that were discussed and therefore received a priority score, we calculated a topic-specific mean score as<disp-formula id="equ6"><label>(6)</label><mml:math id="m6"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo>¯</mml:mo></mml:mover><mml:mi>t</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:msubsup><mml:mo largeop="true" symmetric="true">∑</mml:mo><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>l</mml:mi></mml:msubsup><mml:mi>s</mml:mi></mml:mrow><mml:mi>l</mml:mi></mml:mfrac></mml:mrow></mml:math></disp-formula>where <italic>l</italic> is the number of applications in that topic that made it to discussion and <italic>s</italic> is the priority score of each application <italic>i</italic>.</p></sec><sec id="s2-2"><title>Regression analyses</title><p><xref ref-type="fig" rid="fig1">Figure 1</xref> shows the association of cumulative percentage of AAB applications by the cumulative percentage of topics. Consistent with <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref>, there was a nonrandom distribution whereby 10% of topics (or 15 topics) accounted for 50% of AAB applications, while 25% of the topics accounted for 70% of AAB applications. We designted a topic as 'AAB Preferred' if it was among the 15 topics that accounted for 50% of AAB applications.</p><fig id="fig1" position="float"><label>Figure 1.</label><caption><title>Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.</title><p>Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 8% of all applications submitted by an AAB PI. Dashed vertical lines show that 10% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators and that 25% of the topics accounted for 70% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/fig1.jpg"/></fig><p>To further assess the association of topic choice with peer review outcomes, we focused on applications that were discussed and therefore received a priority score. We constructed a plot of topic-specific mean peer review scores according to number of applications in each topic. We expected to find a greater variance of mean scores for topics receiving fewer applications ('regression to the mean'). We generated a linear regression model to estimate a predicted mean score for each topic based on topic size (namely the number of submitted applications that were discussed for each topic), and calculated a residual for each topic by subtracting from each topic-specific mean score the model-based predicted mean score. We compared the distribution of residuals for AAB preferred topics and other topics.</p><p>We next performed a series of probit regression analyses of application funding as<disp-formula id="equ7"><label>(7)</label><mml:math id="m7"><mml:mstyle displaystyle="true" scriptlevel="0"><mml:mrow><mml:msub><mml:mi>Y</mml:mi><mml:mrow><mml:mi>i</mml:mi></mml:mrow></mml:msub><mml:mo>=</mml:mo><mml:msub><mml:mi>β</mml:mi><mml:mrow><mml:mn>0</mml:mn></mml:mrow></mml:msub><mml:mo>+</mml:mo><mml:msub><mml:mi>β</mml:mi><mml:mrow><mml:mn>1</mml:mn></mml:mrow></mml:msub><mml:mrow><mml:mi>P</mml:mi><mml:msub><mml:mi>I</mml:mi><mml:mrow><mml:mi>A</mml:mi><mml:mi>A</mml:mi><mml:mi>B</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo>+</mml:mo><mml:msub><mml:mi>β</mml:mi><mml:mrow><mml:mn>2</mml:mn></mml:mrow></mml:msub><mml:mrow><mml:msub><mml:mi>T</mml:mi><mml:mrow><mml:mi>A</mml:mi><mml:mi>A</mml:mi><mml:mi>B</mml:mi><mml:mi>P</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo>+</mml:mo><mml:msub><mml:mi>β</mml:mi><mml:mrow><mml:mn>3</mml:mn></mml:mrow></mml:msub><mml:mrow><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>I</mml:mi><mml:mi>C</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo>+</mml:mo><mml:mrow><mml:mi>β</mml:mi></mml:mrow><mml:mrow><mml:mi mathvariant="bold">X</mml:mi></mml:mrow><mml:mo>+</mml:mo><mml:mi>ϵ</mml:mi></mml:mrow></mml:mstyle></mml:math></disp-formula>where <inline-formula><mml:math id="inf2"><mml:mrow><mml:mi>P</mml:mi><mml:mo>⁢</mml:mo><mml:msub><mml:mi>I</mml:mi><mml:mrow><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>B</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula> is an indicator variable for whether application <italic>i</italic> had an African American or Black PI (yes or no), <inline-formula><mml:math id="inf3"><mml:msub><mml:mi>T</mml:mi><mml:mrow><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>A</mml:mi><mml:mo>⁢</mml:mo><mml:mi>B</mml:mi><mml:mo>⁢</mml:mo><mml:mi>P</mml:mi></mml:mrow></mml:msub></mml:math></inline-formula> is an indicator variable for whether application <italic>i</italic> focused on an AAB Preferred topic (yes or no), <inline-formula><mml:math id="inf4"><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>I</mml:mi><mml:mo>⁢</mml:mo><mml:mi>C</mml:mi></mml:mrow></mml:msub></mml:math></inline-formula> is the IC award rate (<xref ref-type="disp-formula" rid="equ2">Equation 2</xref>), and <inline-formula><mml:math id="inf5"><mml:mi mathvariant="bold">𝐗</mml:mi></mml:math></inline-formula> corresponds to variables including 'Early Stage Investigator' status, whether application <italic>i</italic> designated more than one PI, and whether the proposed research in application <italic>i</italic> involved animal and/or human subjects. Early Stage Investigators were those PIs within 10 years of their terminal research degree or completion of clinical training. Akaike Information Criteria, Bayesian Information Criteria, and Log Likelihood values informed model strength.</p><p>All analyses used R (<xref ref-type="bibr" rid="bib12">R Foundation, 2021</xref>) packages, including tidyverse (<xref ref-type="bibr" rid="bib16">Wickham et al., 2019</xref>), ggplot2 (<xref ref-type="bibr" rid="bib15">Wickham, 2016</xref>), finalfit (<xref ref-type="bibr" rid="bib3">Harrison, 2020</xref>), and texreg (<xref ref-type="bibr" rid="bib7">Leifeld, 2013</xref>).</p></sec></sec><sec id="s3" sec-type="results"><title>Results</title><p>Of 157,405 applications received, there were, after exclusion of pre-review withdrawals, resubmissions and competing renewals, 96,697 applications considered and reviewed by NIH for the first time. Of these 8381 were funded, for an overall award rate of 9%. The vast majority of applications (88%) were reviewed in the NIH Center for Scientific Review (or CSR). There were 1643 applications, or 2%, submitted by AAB PIs. <xref ref-type="table" rid="table1">Table 1</xref> shows IC-specific values for applications received, applications funded, award rates (<xref ref-type="disp-formula" rid="equ2">Equation 2</xref>), and percent applications coming from AAB PIs. Of note, award rates varied from 6% to 15%, while the proportion of applications with AAB PIs ranged from <1% to nearly 15%. We also show 2015 appropriations for each IC (<xref ref-type="bibr" rid="bib10">NIH, 2021b</xref>).</p><table-wrap id="table1" position="float"><label>Table 1.</label><caption><title>Application characteristics according to Institute or Center (IC), as ranked by award rate (<xref ref-type="disp-formula" rid="equ2">Equation 2</xref>, highest to lowest).</title><p>AAB = African American or Black; PI = Principal Investigator; EY = National Eye Institute; DC = National Institute of Deafness and Communications Disorders; GM = National Institute of General Medical Sciences; DE = National Institute of Dental and Craniofacial Research; MH = National Institute of Mental Health; DA = National Institute on Drug Abuse; NS = National Institute of Neurological Disorders and Stroke; NR = National Institute of Nursing Research; HL = National Heart, Lung, and Blood Institute; AI = National Institute of Allergy and Infectious Diseases; ES = National Institute of Environmental Health Sciences; DK = National Institute of Diabetes and Digestive and Kidney Disease; AA = National Institute on Alcohol Abuse and Alcoholism; AG = National Institute on Aging; EB = National Institute of Biomedical Imaging and Bioengineering; CA = National Cancer Institute; HD = Eunice Kennedy Shriver National Institute of Child Health and Human Development; MD = National Institute on Minority Health and Health Disparities; AR = National Institute of Arthritis and Musculoskeletal and Skin Diseases. Data for ICs with cell sizes not exceeding 11 are not shown due to privacy concerns. Final columns shows 2015 appropriations in billions of dollars.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>IC</th><th>Applications(N)</th><th>Awards(N)</th><th>Award rate(%)</th><th>AAB PI</th><th>AAB(%)</th><th>Appropriations($B)</th></tr></thead><tbody><tr><td>EY</td><td>2026</td><td>315</td><td>15.5</td><td>14</td><td>0.69</td><td>0.684</td></tr><tr><td>GM</td><td>9781</td><td>1266</td><td>12.9</td><td>113</td><td>1.16</td><td>2.371</td></tr><tr><td>DC</td><td>1145</td><td>141</td><td>12.3</td><td>13</td><td>1.14</td><td>0.405</td></tr><tr><td>DE</td><td>1282</td><td>140</td><td>10.9</td><td>26</td><td>2.03</td><td>0.400</td></tr><tr><td>MH</td><td>5067</td><td>536</td><td>10.6</td><td>87</td><td>1.72</td><td>1.463</td></tr><tr><td>DA</td><td>3606</td><td>359</td><td>10.0</td><td>66</td><td>1.83</td><td>1.029</td></tr><tr><td>NS</td><td>6786</td><td>668</td><td>9.8</td><td>77</td><td>1.13</td><td>1.605</td></tr><tr><td>NR</td><td>1180</td><td>107</td><td>9.1</td><td>55</td><td>4.66</td><td>0.141</td></tr><tr><td>HL</td><td>11,325</td><td>995</td><td>8.8</td><td>171</td><td>1.51</td><td>2.998</td></tr><tr><td>ES</td><td>2193</td><td>188</td><td>8.6</td><td>44</td><td>2.01</td><td>0.745</td></tr><tr><td>AI</td><td>9421</td><td>791</td><td>8.4</td><td>196</td><td>2.08</td><td>4.359</td></tr><tr><td>DK</td><td>6986</td><td>538</td><td>7.7</td><td>110</td><td>1.57</td><td>1.900</td></tr><tr><td>AA</td><td>1423</td><td>105</td><td>7.4</td><td>16</td><td>1.12</td><td>0.447</td></tr><tr><td>EB</td><td>1777</td><td>131</td><td>7.4</td><td>15</td><td>0.84</td><td>0.330</td></tr><tr><td>AG</td><td>4211</td><td>302</td><td>7.2</td><td>52</td><td>1.23</td><td>1.199</td></tr><tr><td>CA</td><td>15,906</td><td>1045</td><td>6.6</td><td>206</td><td>1.30</td><td>4.950</td></tr><tr><td>HD</td><td>5454</td><td>340</td><td>6.2</td><td>168</td><td>3.08</td><td>1.287</td></tr><tr><td>MD</td><td>829</td><td>51</td><td>6.2</td><td>123</td><td>14.84</td><td>0.269</td></tr><tr><td>AR</td><td>2656</td><td>161</td><td>6.1</td><td>30</td><td>1.13</td><td>0.522</td></tr></tbody></table></table-wrap><p><xref ref-type="table" rid="table2">Table 2</xref> shows review and funding outcomes for applications according to whether the assignment was to an IC in the top quartile of proportion of applications with AAB PIs ('Higher AAB'). These ICs were the National Institute of Allergy and Infectious Diseases, the National Institute of Dental and Craniofacial Research, the National Institute of Child Health and Human Development, the National Institute of Minority Health and Disparities, the National Institute of Nursing Research, and the Fogarty International Center. Applications assigned to Higher AAB ICs were three times more likely to come from AAB PIs. Review outcomes – proportion discussed and, for those applications that were discussed at peer review meetings, priority scores and percentile rankings – were similar in both groups. Despite the similar review outcomes, these applications were 10% less likely to be funded. These descriptive data show that two IC-defined cohorts of applications with similar review outcomes can have substantively different funding outcomes.</p><table-wrap id="table2" position="float"><label>Table 2.</label><caption><title>Application review and funding outcomes according to whether Institute or Center received a higher or lower proportion of applications from AAB principal investigators.</title><p>AAB = African American or Black; PI = Principal Investigator.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>Characteristic or outcome</th><th/><th>Higher AAB</th><th>Lower AAB</th></tr></thead><tbody><tr><td>Total N (%)</td><td/><td>18262 (18.9)</td><td>78435 (81.1)</td></tr><tr><td>PI AAB</td><td>Yes</td><td>579 (3.2)</td><td>1064 (1.4)</td></tr><tr><td>Discussed</td><td>Yes</td><td>8809 (48.2)</td><td>35845 (45.7)</td></tr><tr><td>Priority Score</td><td>Median (IQR)</td><td>39.0 (30.0 to 48.0)</td><td>40.0 (30.0 to 48.0)</td></tr><tr><td>Percentile Ranking</td><td>Median (IQR)</td><td>34.0 (21.0 to 45.0)</td><td>33.0 (21.0 to 44.0)</td></tr><tr><td>Awarded</td><td>Yes</td><td>1440 (7.9)</td><td>6941 (8.8)</td></tr></tbody></table></table-wrap><sec id="s3-1"><title>Review outcomes according to topic</title><p><xref ref-type="fig" rid="fig2">Figure 2</xref> shows the topic-based peer review outcomes according to the proportion of total AAB applications linked to each topic. There was a non-significant trend whereby applications focusing on topics that accounted for a greater proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) were more likely to make it to discussion (<xref ref-type="disp-formula" rid="equ5">Equation 5</xref>)(Panel A, p=0.12). When we focused on applications that were discussed and therefore received a priority score, there was no association of mean score for each topic (<xref ref-type="disp-formula" rid="equ6">Equation 6</xref>) with proportion of total AAB applications linked to each topic (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) (Panel B, p=0.87).</p><fig id="fig2" position="float"><label>Figure 2.</label><caption><title>Topic peer review outcomes according to proportion of AAB applications linked to specific topics.</title><p>AAB = African American or Black. Panel <bold>A</bold>: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.12). Panel <bold>B</bold>: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.87).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/fig2.jpg"/></fig><p>To gain greater insights into possible peer review biases against topics preferred by AAB PIs, <xref ref-type="fig" rid="fig3">Figure 3</xref>, Panel A, shows the mean priority score by topic (<xref ref-type="disp-formula" rid="equ6">Equation 6</xref>; again, only discussed applications receive priority scores) according to the topic size, namely the number of submitted applications that were discussed for each topic. As would be expected topics of smaller size showed greater variability, a manifestation of regression to the mean.</p><fig id="fig3" position="float"><label>Figure 3.</label><caption><title>Topic peer review scores according to number of applications received ('Topic Size') and topic type (AAB Preferred or Other).</title><p>Panel <bold>A</bold>: Scatter plot of topic-specific mean peer review scores according to topic size. Each dot refers to a topic, with orange dots AAB preferred topics and green dots all others. The line is based on a linear regression of mean peer review scores on topic size. The slope of the line was not significant (p=0.63). Panel <bold>B</bold>: Distribution of weighted residuals of topic-specific mean review scores. Residuals are calculated as the distance between the dots in Panel A and the regression line, and are then weighted by topic size.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/fig3.jpg"/></fig><p>The line in <xref ref-type="fig" rid="fig3">Figure 3</xref>, Panel A, is based on a linear regression of mean score according to topic size. Although the slope was slightly negative (coeffecient −0.0002634), the association was not significant (p=0.63). Among AAB preferred topics (orange dots), there were 5 more than 1 point above the line (meaning with scores worse than predicted), while there were 3 more than 1 point below the line (meaning with scores better than predicted). The remaining seven topics had mean scores that were within 1 point of the predicted value.</p><p>For each topic, we calculated a residual by subtracting from the topic-specific mean priority score (<xref ref-type="disp-formula" rid="equ6">Equation 6</xref>) the predicted mean priority score; we weighted the residuals by the topic size, as the larger topics contribute more information. <xref ref-type="fig" rid="fig3">Figure 3</xref>, Panel B, shows the distribution of the weighted residuals according to topic type. Residuals were more positive (i.e. worse) for AAB preferred topics. However, the absolute differences were small, much less than one priority score point (over a possible range of 10–90, with topic-specific mean values ranging from 35 to 45).</p></sec><sec id="s3-2"><title>Funding outcomes according to topic: descriptive findings</title><p><xref ref-type="table" rid="table3">Table 3</xref> shows peer review and funding outcomes according to whether applications were focused on the 15 (or 10% of) topics that made up 50% of all applications with AAB PIs ('AAB Preferred' topics). Consistent with the absence of association with proportion of AAB applications linked to each topic (<xref ref-type="fig" rid="fig2">Figure 2</xref>), peer review outcomes were similar in the two groups. However, applications focusing on AAB Preferred topics were 7% less likely to be funded. <xref ref-type="table" rid="table4">Table 4</xref> shows similar findings when focusing on the 37 (or 25% of) topics that made up 70% of all applications with AAB PIs.</p><table-wrap id="table3" position="float"><label>Table 3.</label><caption><title>Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.</title><p>Abbreviations as in <xref ref-type="table" rid="table2">Table 2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>Characteristic or outcome</th><th/><th>AAB preferred</th><th>Other</th></tr></thead><tbody><tr><td>Total N (%)</td><td/><td>24416 (25.3)</td><td>72281 (74.7)</td></tr><tr><td>PI AAB</td><td>Yes</td><td>824 (3.4)</td><td>819 (1.1)</td></tr><tr><td>Discussed</td><td>Yes</td><td>11515 (47.2)</td><td>33139 (45.8)</td></tr><tr><td>Priority Score</td><td>Median (IQR)</td><td>40.0 (31.0 to 48.0)</td><td>39.0 (30.0 to 48.0)</td></tr><tr><td>Percentile Ranking</td><td>Median (IQR)</td><td>34.0 (22.0 to 45.0)</td><td>33.0 (20.0 to 44.0)</td></tr><tr><td>Assigned IC AAB Proportion</td><td>Higher AAB</td><td>5925 (24.3)</td><td>12337 (17.1)</td></tr><tr><td>Awarded</td><td>Yes</td><td>1992 (8.2)</td><td>6389 (8.8)</td></tr></tbody></table></table-wrap><table-wrap id="table4" position="float"><label>Table 4.</label><caption><title>Application review and funding outcomes according to whether topic was among those that accounted for 70% of all AAB applications.</title><p>Abbreviations as in <xref ref-type="table" rid="table2">Table 2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>Characteristic or outcome</th><th/><th>AAB preferred</th><th>Other</th></tr></thead><tbody><tr><td>Total N (%)</td><td/><td>46108 (47.7)</td><td>50589 (52.3)</td></tr><tr><td>PI AAB</td><td>Yes</td><td>1152 (2.5)</td><td>491 (1.0)</td></tr><tr><td>Discussed</td><td>Yes</td><td>21485 (46.6)</td><td>23169 (45.8)</td></tr><tr><td>Priority Score</td><td>Median (IQR)</td><td>40.0 (31.0 to 48.0)</td><td>39.0 (30.0 to 48.0)</td></tr><tr><td>Percentile Ranking</td><td>Median (IQR)</td><td>34.0 (21.0 to 45.0)</td><td>33.0 (20.0 to 44.0)</td></tr><tr><td>Assigned IC AAB Proportion</td><td>Higher AAB</td><td>11424 (24.8)</td><td>6838 (13.5)</td></tr><tr><td>Awarded</td><td>Yes</td><td>3852 (8.4)</td><td>4529 (9.0)</td></tr></tbody></table></table-wrap><p>Why do applications on AAB Preferred topics have worse funding outcomes despite similar peer review assessments? <xref ref-type="table" rid="table3">Table 3</xref> shows that applications on AAB Preferred topics (those 10% of topics that accounted for 50% of all AAB applications) were 42% more likely to be assigned to Higher AAB ICs. The scatter plot in <xref ref-type="fig" rid="fig4">Figure 4</xref> shows IC award rate according to the proportion of applications assigned to it that focus on AAB Preferred topics. ICs with receiving a higher percentage of AAB Preferred topic applications tended to have lower award rates (r = −0.35, p=0.08).</p><fig id="fig4" position="float"><label>Figure 4.</label><caption><title>Scatter plot of IC-specific award rates according to proportion of IC applications that focus on AAB Preferred topics.</title><p>In other words, the X-axis value would be 33% if one-third of all IC applications focused on AAB Preferred topics, namely those 15 topics that together accounted for 50% of all applications with AAB PIs. The orange-colored ICs are those in which the proportion of applications from AAB PIs were in the top quartile. AAB = African American or Black; IC = Institute or Center; EY = National Eye Institute; DC = National Institute of Deafness and Communications Disorders; GM = National Institute of General Medical Sciences; DE = National Institute of Dental and Craniofacial Research; MH = National Institute of Mental Health; DA = National Institute on Drug Abuse; NS = National Institute of Neurological Disorders and Stroke; NR = National Institute of Nursing Research; HL = National Heart, Lung, and Blood Institute; AI = National Institute of Allergy and Infectious Diseases; ES = National Institute of Environmental Health Sciences; DK = National Institute of Diabetes and Digestive and Kidney Disease; AA = National Institute on Alcohol Abuse and Alcoholism; AG = National Institute on Aging; EB = National Institute of Biomedical Imaging and Bioengineering; CA = National Cancer Institute; HD = Eunice Kennedy Shriver National Institute of Child Health and Human Development; MD = National Institute on Minority Health and Health Disparities; AR = National Institute of Arthritis and Musculoskeletal and Skin Diseases. ICs that receive relatively more applications on AAB Preferred topics tended to have lower award rates (r = −0.35, p=0.08). Data for ICs with cell sizes not exceeding 11 are not shown due to privacy concerns.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/fig4.jpg"/></fig><p>The association between IC award rates and application success is demonstrated in <xref ref-type="fig" rid="fig5">Figure 5</xref>, Panel A, which shows that observed topic-specific award rates (<xref ref-type="disp-formula" rid="equ3">Equation 3</xref>) were strongly correlated with predicted award rates (<xref ref-type="disp-formula" rid="equ4">Equation 4</xref>), that is award rates that would be expected based solely on IC-specific award rates (r = 0.62, p<0.0001). <xref ref-type="fig" rid="fig5">Figure 5</xref>, Panel B shows a tendency toward lower predicted topic-specific award rates (<xref ref-type="disp-formula" rid="equ4">Equation 4</xref>) with increasing proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) linked to each topic (p=0.06). Panel C shows that the actual topic-specific award rates (<xref ref-type="disp-formula" rid="equ3">Equation 3</xref>) were non-significantly negatively associated with increasing proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) (p=0.76).</p><fig id="fig5" position="float"><label>Figure 5.</label><caption><title>Descriptive analyses of actual (<xref ref-type="disp-formula" rid="equ3">Equation 3</xref>) and predicted (<xref ref-type="disp-formula" rid="equ4">Equation 4</xref>) topic-specific award rates.</title><p>AAB = African American or Black. Panel <bold>A</bold>: Association of actual vs predicted award rates. Each dot refers to a topic. The line is based on a linear regression of actual award rate on predicted award rate (r = 0.62, p<0.0001). Panel <bold>B</bold>: Association of predicted award rate (<xref ref-type="disp-formula" rid="equ4">Equation 4</xref>) with proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>). Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the 3 topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of predicted award rate on percent of all AAB applications; the negative slope was borderline signifiant (p=0.06). Panel <bold>C</bold>: Association of actual award rate (<xref ref-type="disp-formula" rid="equ3">Equation 3</xref>) with proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>). Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of actual award rate on percent of all AAB applications; the slope was non-signficant (p=0.76).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/fig5.jpg"/></fig></sec><sec id="s3-3"><title>Funding outcomes according to topic: probit regression models</title><p><xref ref-type="table" rid="table5">Table 5</xref> shows the association of an application with an AAB PI with the probability of funding. Consistent with <xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> and prior literature (<xref ref-type="bibr" rid="bib2">Ginther et al., 2011</xref>), AAB PI applications had a lower likelihood of funding (Model 1, negative regression coefficient for AAB Principal Investigator, p<0.001). Adjusting for the topic (AAB Preferred or Other) reduced the absolute value of the regression coefficient for race by 5% (Model 2); similarly adjusting for IC assignment (Higher or Lower AAB) reduced the absolute value of the regression coefficient by 5% (Model 3). However, adjusting for the award rate of the assigned IC reduced the absolute value of the regression coefficient for race by 14% (Model 4).</p><table-wrap id="table5" position="float"><label>Table 5.</label><caption><title>Probit Regression Models (regression coefficients and standard errors) with focus on the PI.</title><p>Model 1 shows the univariable association of funding success according to whether the PI is AAB. The negative non-zero cofficient indicates that applications with AAB PIs are less likely to be funded; the p-value refers to the likelihood that the difference from zero would be due to chance were the null hypothesis of no association to be true. Model 2 adjusts for topic, Model 3 adjusts for IC assignment, and Model 4 adjusts for IC award rate. Note that the absolute value for the regression coefficient linking AAB PI to funding outcome decreases with each of these adjustments, with the greatest reduction after adjusting for IC award rate. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in <xref ref-type="table" rid="table1">Tables 1</xref> and <xref ref-type="table" rid="table2">2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th/><th>Model 1</th><th>Model 2</th><th>Model 3</th><th>Model 4</th></tr></thead><tbody><tr><td>Intercept</td><td>−1.359***</td><td>−1.349***</td><td>−1.348***</td><td>−1.903***</td></tr><tr><td/><td>(0.006)</td><td>(0.007)</td><td>(0.006)</td><td>(0.020)</td></tr><tr><td>AAB Principal Investigator</td><td>−0.184***</td><td>−0.174***</td><td>−0.174***</td><td>−0.158**</td></tr><tr><td/><td>(0.049)</td><td>(0.049)</td><td>(0.049)</td><td>(0.050)</td></tr><tr><td>AAB Preferred Topic</td><td/><td>−0.040**</td><td/><td/></tr><tr><td/><td/><td>(0.013)</td><td/><td/></tr><tr><td>IC Higher AAB Applications</td><td/><td/><td>−0.060***</td><td/></tr><tr><td/><td/><td/><td>(0.015)</td><td/></tr><tr><td>IC Award Rate</td><td/><td/><td/><td>0.061***</td></tr><tr><td/><td/><td/><td/><td>(0.002)</td></tr><tr><td>AIC</td><td>56996.279</td><td>56989.156</td><td>56982.092</td><td>56175.409</td></tr><tr><td>BIC</td><td>57015.238</td><td>57017.594</td><td>57010.530</td><td>56203.847</td></tr><tr><td>Log Likelihood</td><td>−28496.140</td><td>−28491.578</td><td>−28488.046</td><td>−28084.705</td></tr><tr><td>Deviance</td><td>56992.279</td><td>56983.156</td><td>56976.092</td><td>56169.409</td></tr><tr><td>Num. obs.</td><td>96697</td><td>96697</td><td>96697</td><td>96697</td></tr></tbody></table><table-wrap-foot><fn><p><inline-formula><mml:math id="inf6"><mml:mrow><mml:mmultiscripts><mml:mi mathsize="70%">p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo>⁣</mml:mo><mml:mrow><mml:mi/><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo mathsize="70%" stretchy="false">*</mml:mo></mml:mrow></mml:mrow></mml:mmultiscripts><mml:mo mathsize="70%" stretchy="false"><</mml:mo><mml:mn mathsize="70%">0.001</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf7"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mi/><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.01</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf8"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mo>*</mml:mo></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.05</mml:mn></mml:mrow></mml:math></inline-formula>.</p></fn></table-wrap-foot></table-wrap><p><xref ref-type="table" rid="table6">Table 6</xref> focuses on topic and funding outcomes. Without consideration of other variables, an AAB preferred topic was associated with a lower probability of funding (Model 1, negative regression coefficient for AAB preferred topic, p<0.01). However, after adjusting for IC award rate alone (Model 2, regression coefficient for AAB preferred topic close to zero, p=NS) as well as IC award rate and other characteristics (whether the PI is an early stage investigator, whether the application included more than one PI, and whether the proposed research included human and/or animal subjects), there was no association between AAB preferred topics and funding (Model 3, regression coefficient for AAB preferred topic close to zero, p=NS). The IC award rate was strongly associated with likelihood of funding (Models 2 and 3, regression coefficient positive, p<0.001).</p><table-wrap id="table6" position="float"><label>Table 6.</label><caption><title>Probit Regression Models (regression coefficients and standard errors) with focus on topic type.</title><p>Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in <xref ref-type="table" rid="table1">Tables 1</xref> and <xref ref-type="table" rid="table2">2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th/><th>Model 1</th><th>Model 2</th><th>Model 3</th></tr></thead><tbody><tr><td>Intercept</td><td>−1.351***</td><td>−1.904***</td><td>−1.869***</td></tr><tr><td/><td>(0.007)</td><td>(0.021)</td><td>(0.027)</td></tr><tr><td>AAB Preferred Topic</td><td>−0.044**</td><td>−0.005</td><td>−0.013</td></tr><tr><td/><td>(0.013)</td><td>(0.014)</td><td>(0.015)</td></tr><tr><td>IC Award Rate</td><td/><td>0.061***</td><td>0.057***</td></tr><tr><td/><td/><td>(0.002)</td><td>(0.002)</td></tr><tr><td>AAB Principal Investigator</td><td/><td/><td>−0.170***</td></tr><tr><td/><td/><td/><td>(0.050)</td></tr><tr><td>Early Stage Investigator</td><td/><td/><td>0.149***</td></tr><tr><td/><td/><td/><td>(0.015)</td></tr><tr><td>Multi-PI Application</td><td/><td/><td>0.099***</td></tr><tr><td/><td/><td/><td>(0.015)</td></tr><tr><td>Human Subjects</td><td/><td/><td>−0.056</td></tr><tr><td/><td/><td/><td>(0.014)</td></tr><tr><td>Animal Subjects</td><td/><td/><td>−0.052***</td></tr><tr><td/><td/><td/><td>(0.014)</td></tr><tr><td>AIC</td><td>57000.243</td><td>56185.926</td><td>56047.521</td></tr><tr><td>BIC</td><td>57019.201</td><td>56214.364</td><td>56123.355</td></tr><tr><td>Log Likelihood</td><td>−28498.121</td><td>−28089.963</td><td>−28015.760</td></tr><tr><td>Deviance</td><td>56996.243</td><td>56179.926</td><td>56031.521</td></tr><tr><td>Num. obs.</td><td>96697</td><td>96697</td><td>96697</td></tr></tbody></table><table-wrap-foot><fn><p><inline-formula><mml:math id="inf9"><mml:mrow><mml:mmultiscripts><mml:mi mathsize="70%">p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo>⁣</mml:mo><mml:mrow><mml:mi/><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo mathsize="70%" stretchy="false">*</mml:mo></mml:mrow></mml:mrow></mml:mmultiscripts><mml:mo mathsize="70%" stretchy="false"><</mml:mo><mml:mn mathsize="70%">0.001</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf10"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mi/><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.01</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf11"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mo>*</mml:mo></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.05</mml:mn></mml:mrow></mml:math></inline-formula>.</p></fn></table-wrap-foot></table-wrap></sec><sec id="s3-4"><title>Resubmission and single-PI applications</title><p>We repeated the main analyses, but this time focusing on resubmission applications, which as a general rule are assigned to the same IC as the original submission. The findings were similar, except that, as expected, the absolute award rates were higher. We also conducted a separate series of analyses which repeated our primarily analyses but focusing solely on single-PI applications. Again, findings were similar (see Appendix 1 and Appendix 2).</p></sec></sec><sec id="s4" sec-type="discussion"><title>Discussion</title><p>Among over 96,000 R01 applications submitted to NIH between 2011 and 2015, 2% were submitted by AAB PIs. Their applications were skewed towards a relatively small group of 'AAB Preferred' topics (<xref ref-type="fig" rid="fig1">Figure 1</xref>); 10% of 148 topics accounted for 50% of AAB applications. Applications on AAB Preferred topics had similar review outcomes as those on other topics (<xref ref-type="fig" rid="fig2">Figure 2</xref> and <xref ref-type="table" rid="table3">Table 3</xref>) but were less likely to be funded. The lower award rates for applications on AAB Preferred applications (<xref ref-type="table" rid="table5">Table 5</xref>, Model 1) were no longer seen after accounting for IC award rates (<xref ref-type="table" rid="table6">Table 6</xref>, Models 2 and 3).</p><p>These observations reflect that there are two factors at play in determining whether an application submitted to NIH will be funded. The first, well known to all involved with NIH system, is peer review; those applications that receive better scores are more likely to be funded. But there is a second factor, namely the funding ecology of the IC to which the application is assigned. As shown in <xref ref-type="table" rid="table2">Table 2</xref> applications with similar peer review outcomes are less likely to be funded if they are assigned to ICs with lower overall award rates. AAB PIs are more likely to submit applications to ICs with lower award rates, and applications (whether submitted by AAB or other PIs) that focus on AAB Preferred topics tended to be assigned to ICs with lower award rates (<xref ref-type="fig" rid="fig4">Figure 4</xref>).</p><p><xref ref-type="bibr" rid="bib4">Hoppe et al., 2019</xref> found that topic choice accounted for over 20% of funding disparities that adversely effect AAB PIs. We confirm this, but find that IC assignment (which, of course, is linked to topic) explains the disparities just as well, and that IC award rates explain the disparities even better (<xref ref-type="table" rid="table5">Table 5</xref>, 14% vs 5% reduction in absolute value of regression coefficient for AAB PI). Furthermore, after accounting for IC award rate, we found no association between topic choice and funding outcome (<xref ref-type="table" rid="table6">Table 6</xref>).</p><p>There is variability in how well different topics fare at peer review, but inspection of <xref ref-type="fig" rid="fig3">Figure 3</xref> suggests that much of this variability reflects instability of estimates stemming from smaller sample sizes. Many topics that are not favored by AAB PIs receive better (lower) priority scores than the overall average, but many other such topics receive worse scores (<xref ref-type="fig" rid="fig3">Figure 3</xref>, Panel A). An inspection of weighted residuals suggest that AAB Preferred topics may fare a bit worse (<xref ref-type="fig" rid="fig3">Figure 3</xref>, Panel B), but to a lower degree than the difference of award rates among assigned ICs (<xref ref-type="table" rid="table1">Table 1</xref>). Furthermore, it should be noted that applications on these topics were <italic>more likely</italic> to make it past the first hurdle of peer review, that is reaching the point of formal discussion (<xref ref-type="fig" rid="fig2">Figure 2</xref> and <xref ref-type="table" rid="table3">Table 3</xref>, see line 'Discussed'); thus, if anything, peer reviewers may be slightly biased in favor of AAB-preferred topics.</p><p>Our primary analysis focused on first-time submissions of de novo applications in which the award rates were low (<10%). Nonetheless, our analyses of resubmission applications yielded similar findings (Appendix 1). It is also important to note that this was an analysis of applications, not persons. This is an issue when considering multi-PI applications, since all PI’s, not just the contact PI, plays a role in choosing the topic of their proposal. Nonetheless, an analysis confined to single PI applications yielded similar findings (Appendix 2).</p><p>Only 2% of PIs in this study were African American or Black and thus it is not possible that this small percentage of scientists cover the entire breadth of what NIH funds. The small number of AAB PIs is a source of error for describing AAB scientific interests as reflected in grant applications. Nonetheless, in recent years, NIH has increasingly recognized the importance of enhancing research in targeted topics, as exemplified by a recently announced Common Fund program to enhance transformative research in health disparities and health equity (<xref ref-type="bibr" rid="bib9">NIH, 2021a</xref>) and by the large investment in RADx-UP, a COVID-19 diagnostic testing program in underserved communities (<xref ref-type="bibr" rid="bib14">Tromberg et al., 2020</xref>).</p><sec id="s4-1"><title>Conclusion</title><p>The lower rate of funding for applications focused on AAB Preferred topics is likely primarily due to their assignment to ICs with lower award rates. These applications have similar peer review outcomes as those focused on other topics (<xref ref-type="table" rid="table3">Table 3</xref>). When AAB preference for each topic (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) was considered as a continuous variable, there was similarly no association between AAB preference and peer review outcomes (<xref ref-type="fig" rid="fig2">Figure 2</xref>). Topic choice does partially explain race-based funding disparities (<xref ref-type="table" rid="table5">Table 5</xref>, Model 2, absolute value of regression coefficient reduced by 5%), but IC-specific award rates explain the disparities to an even greater degree (<xref ref-type="table" rid="table5">Table 5</xref>, Model 4, absolute value of regression coefficient reduced by 14%). After accounting for IC-specific award rates, we find no association between topic choice and funding outcomes (<xref ref-type="table" rid="table6">Table 6</xref>, Models 2 and 3).</p></sec></sec></body><back><sec id="s5" sec-type="additional-information"><title>Additional information</title><fn-group content-type="competing-interest"><title>Competing interests</title><fn fn-type="COI-statement" id="conf1"><p>No competing interests declared</p></fn></fn-group><fn-group content-type="author-contribution"><title>Author contributions</title><fn fn-type="con" id="con1"><p>Conceptualization, Formal analysis, Supervision, Methodology, Writing - original draft, Project administration</p></fn><fn fn-type="con" id="con2"><p>Conceptualization, Formal analysis, Methodology, Writing - review and editing</p></fn><fn fn-type="con" id="con3"><p>Formal analysis, Methodology, Writing - review and editing</p></fn><fn fn-type="con" id="con4"><p>Formal analysis, Methodology, Writing - review and editing</p></fn></fn-group></sec><sec id="s6" sec-type="supplementary-material"><title>Additional files</title><supplementary-material id="sdata1"><label>Source data 1.</label><caption><title>csr_anon_id.RData.</title></caption><media mime-subtype="zip" mimetype="application" xlink:href="elife-67173-data1-v2.zip"/></supplementary-material><supplementary-material id="scode1"><label>Source code 1.</label><caption><title>Main Manuscript.Rmd.</title></caption><media mime-subtype="zip" mimetype="application" xlink:href="elife-67173-code1-v2.zip"/></supplementary-material><supplementary-material id="scode2"><label>Source code 2.</label><caption><title>Appendix 1.Rmd.</title></caption><media mime-subtype="zip" mimetype="application" xlink:href="elife-67173-code2-v2.zip"/></supplementary-material><supplementary-material id="scode3"><label>Source code 3.</label><caption><title>Appendix 2.Rmd.</title></caption><media mime-subtype="zip" mimetype="application" xlink:href="elife-67173-code3-v2.zip"/></supplementary-material><supplementary-material id="transrepform"><label>Transparent reporting form</label><media mime-subtype="docx" mimetype="application" xlink:href="elife-67173-transrepform-v2.docx"/></supplementary-material></sec><sec id="s7" sec-type="data-availability"><title>Data availability</title><p>The authors have provided the de-identified data frame (in. RData format) along with three R markdown files that will make it possible for interested readers to reproduce the main paper and the two appendices (including all tables, figures, and numbers in the text).</p></sec><ref-list><title>References</title><ref id="bib1"><element-citation publication-type="web"><person-group person-group-type="author"><collab>CSR</collab></person-group><year iso-8601-date="2021">2021</year><article-title>The assignment process</article-title><ext-link ext-link-type="uri" xlink:href="https://public.csr.nih.gov/ForApplicants/SubmissionAndAssignment/DRR/assignmentprocess">https://public.csr.nih.gov/ForApplicants/SubmissionAndAssignment/DRR/assignmentprocess</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib2"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ginther</surname> <given-names>DK</given-names></name><name><surname>Schaffer</surname> <given-names>WT</given-names></name><name><surname>Schnell</surname> <given-names>J</given-names></name><name><surname>Masimore</surname> <given-names>B</given-names></name><name><surname>Liu</surname> <given-names>F</given-names></name><name><surname>Haak</surname> <given-names>LL</given-names></name><name><surname>Kington</surname> <given-names>R</given-names></name></person-group><year iso-8601-date="2011">2011</year><article-title>Race, ethnicity, and NIH research awards</article-title><source>Science</source><volume>333</volume><fpage>1015</fpage><lpage>1019</lpage><pub-id pub-id-type="doi">10.1126/science.1196783</pub-id><pub-id pub-id-type="pmid">21852498</pub-id></element-citation></ref><ref id="bib3"><element-citation publication-type="web"><person-group person-group-type="author"><name><surname>Harrison</surname> <given-names>E</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Tom Drake, and ots Quickly Create Elegant Regression Results Tables and Plots when Modelling</article-title><ext-link ext-link-type="uri" xlink:href="https://finalfit.org/index.html">https://finalfit.org/index.html</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib4"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hoppe</surname> <given-names>TA</given-names></name><name><surname>Litovitz</surname> <given-names>A</given-names></name><name><surname>Willis</surname> <given-names>KA</given-names></name><name><surname>Meseroll</surname> <given-names>RA</given-names></name><name><surname>Perkins</surname> <given-names>MJ</given-names></name><name><surname>Hutchins</surname> <given-names>BI</given-names></name><name><surname>Davis</surname> <given-names>AF</given-names></name><name><surname>Lauer</surname> <given-names>MS</given-names></name><name><surname>Valantine</surname> <given-names>HA</given-names></name><name><surname>Anderson</surname> <given-names>JM</given-names></name><name><surname>Santangelo</surname> <given-names>GM</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Topic choice contributes to the lower rate of NIH awards to African-American/black scientists</article-title><source>Science Advances</source><volume>5</volume><elocation-id>eaaw7238</elocation-id><pub-id pub-id-type="doi">10.1126/sciadv.aaw7238</pub-id><pub-id pub-id-type="pmid">31633016</pub-id></element-citation></ref><ref id="bib5"><element-citation publication-type="web"><person-group person-group-type="author"><name><surname>Lauer</surname> <given-names>MS</given-names></name></person-group><year iso-8601-date="2016">2016</year><article-title>Are attempts at renewal successful?</article-title><ext-link ext-link-type="uri" xlink:href="https://nexus.od.nih.gov/all/2016/02/16/are-attempts-at-renewal-successful/">https://nexus.od.nih.gov/all/2016/02/16/are-attempts-at-renewal-successful/</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib6"><element-citation publication-type="web"><person-group person-group-type="author"><name><surname>Lauer</surname> <given-names>MS</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Resubmissions revisited: funded resubmission applications and their initial peer review scores</article-title><ext-link ext-link-type="uri" xlink:href="https://nexus.od.nih.gov/all/2017/02/17/resubmissions-revisited-funded-resubmission-applications-and-their-initial-peer-review-scores/">https://nexus.od.nih.gov/all/2017/02/17/resubmissions-revisited-funded-resubmission-applications-and-their-initial-peer-review-scores/</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib7"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Leifeld</surname> <given-names>P</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Texreg: conversion of statistical model output in R to LaTeX and HTML tables</article-title><source>Journal of Statistical Software</source><volume>55</volume><fpage>1</fpage><lpage>24</lpage><pub-id pub-id-type="doi">10.18637/jss.v055.i08</pub-id></element-citation></ref><ref id="bib8"><element-citation publication-type="preprint"><person-group person-group-type="author"><name><surname>Mikolov</surname> <given-names>T</given-names></name><name><surname>Chen</surname> <given-names>K</given-names></name><name><surname>Corrado</surname> <given-names>G</given-names></name><name><surname>Dean</surname> <given-names>J</given-names></name></person-group><year iso-8601-date="2013">2013</year><article-title>Efficient estimation of word representations in vector space</article-title><source>arXiv</source><ext-link ext-link-type="uri" xlink:href="https://arxiv.org/abs/1301.3781">https://arxiv.org/abs/1301.3781</ext-link></element-citation></ref><ref id="bib9"><element-citation publication-type="web"><person-group person-group-type="author"><collab>NIH</collab></person-group><year iso-8601-date="2021">2021a</year><article-title>ACD meetings</article-title><ext-link ext-link-type="uri" xlink:href="https://www.acd.od.nih.gov/meetings.html">https://www.acd.od.nih.gov/meetings.html</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib10"><element-citation publication-type="web"><person-group person-group-type="author"><collab>NIH</collab></person-group><year iso-8601-date="2021">2021b</year><article-title>NIH — Office of Budget —Appropriations History by Institute/Center (1938 to Present)</article-title><ext-link ext-link-type="uri" xlink:href="https://officeofbudget.od.nih.gov/approp_hist.html">https://officeofbudget.od.nih.gov/approp_hist.html</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib11"><element-citation publication-type="web"><person-group person-group-type="author"><collab>NIH</collab></person-group><year iso-8601-date="2021">2021c</year><article-title>NIH strategic plans and visions | RePORT</article-title><ext-link ext-link-type="uri" xlink:href="https://report.nih.gov/reports/strategic-plans">https://report.nih.gov/reports/strategic-plans</ext-link><date-in-citation iso-8601-date="2021-04-25">April 25, 2021</date-in-citation></element-citation></ref><ref id="bib12"><element-citation publication-type="software"><person-group person-group-type="author"><collab>R Foundation</collab></person-group><year iso-8601-date="2021">2021</year><source>R: The R Project for Statistical Computing</source><ext-link ext-link-type="uri" xlink:href="https://www.r-project.org/">https://www.r-project.org/</ext-link></element-citation></ref><ref id="bib13"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Taffe</surname> <given-names>MA</given-names></name><name><surname>Gilpin</surname> <given-names>NW</given-names></name></person-group><year iso-8601-date="2021">2021</year><article-title>Racial inequity in grant funding from the US National Institutes of Health</article-title><source>eLife</source><volume>10</volume><elocation-id>e65697</elocation-id><pub-id pub-id-type="doi">10.7554/eLife.65697</pub-id></element-citation></ref><ref id="bib14"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Tromberg</surname> <given-names>BJ</given-names></name><name><surname>Schwetz</surname> <given-names>TA</given-names></name><name><surname>Pérez-Stable</surname> <given-names>EJ</given-names></name><name><surname>Hodes</surname> <given-names>RJ</given-names></name><name><surname>Woychik</surname> <given-names>RP</given-names></name><name><surname>Bright</surname> <given-names>RA</given-names></name><name><surname>Fleurence</surname> <given-names>RL</given-names></name><name><surname>Collins</surname> <given-names>FS</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Rapid scaling up of Covid-19 diagnostic testing in the united states — The NIH RADx Initiative</article-title><source>New England Journal of Medicine</source><volume>383</volume><fpage>1071</fpage><lpage>1077</lpage><pub-id pub-id-type="doi">10.1056/NEJMsr2022263</pub-id></element-citation></ref><ref id="bib15"><element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Wickham</surname> <given-names>H</given-names></name></person-group><year iso-8601-date="2016">2016</year><source>GGplot2: Elegant Graphics for Data Analysis</source><publisher-loc>New York</publisher-loc><publisher-name>Springer-Verlag</publisher-name><pub-id pub-id-type="doi">10.1007/978-0-387-98141-3</pub-id></element-citation></ref><ref id="bib16"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wickham</surname> <given-names>H</given-names></name><name><surname>Averick</surname> <given-names>M</given-names></name><name><surname>Bryan</surname> <given-names>J</given-names></name><name><surname>Chang</surname> <given-names>W</given-names></name><name><surname>McGowan</surname> <given-names>L</given-names></name><name><surname>François</surname> <given-names>R</given-names></name><name><surname>Grolemund</surname> <given-names>G</given-names></name><name><surname>Hayes</surname> <given-names>A</given-names></name><name><surname>Henry</surname> <given-names>L</given-names></name><name><surname>Hester</surname> <given-names>J</given-names></name><name><surname>Kuhn</surname> <given-names>M</given-names></name><name><surname>Pedersen</surname> <given-names>T</given-names></name><name><surname>Miller</surname> <given-names>E</given-names></name><name><surname>Bache</surname> <given-names>S</given-names></name><name><surname>Müller</surname> <given-names>K</given-names></name><name><surname>Ooms</surname> <given-names>J</given-names></name><name><surname>Robinson</surname> <given-names>D</given-names></name><name><surname>Seidel</surname> <given-names>D</given-names></name><name><surname>Spinu</surname> <given-names>V</given-names></name><name><surname>Takahashi</surname> <given-names>K</given-names></name><name><surname>Vaughan</surname> <given-names>D</given-names></name><name><surname>Wilke</surname> <given-names>C</given-names></name><name><surname>Woo</surname> <given-names>K</given-names></name><name><surname>Yutani</surname> <given-names>H</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Welcome to the tidyverse</article-title><source>Journal of Open Source Software</source><volume>4</volume><elocation-id>1686</elocation-id><pub-id pub-id-type="doi">10.21105/joss.01686</pub-id></element-citation></ref></ref-list><app-group><app id="appendix-1"><title>Appendix 1</title><boxed-text><sec id="s8" sec-type="appendix"><title>Resubmission applications</title><p>In our primary analyses, we excluded resubmission applications and competing renewals. Since many applications are not funded on the first try, we repeated our analyses focusing on resubmissions. Our findings are similar.</p><p>Of 31,900 resubmission applications received and reviewed, 9605 were funded, for an overall award rate of 30%. There were 448 applications, or 1%, submitted by AAB PIs.</p><p><xref ref-type="fig" rid="app1fig1">Appendix 1—figure 1</xref> shows the association of cumulative percentage of AAB applications by the cumulative percentage of topics. There was a nonrandom distribution whereby 11 percent of topics (or 16 topics) accounted for 50 percent of AAB applications. We designted a topic as 'AAB Preferred' if it was among the 16 topics that accounted for 50% of AAB applications.</p><fig id="app1fig1" position="float"><label>Appendix 1—figure 1.</label><caption><title>Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.</title><p>Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 6% of all applications submitted by an AAB PI. Dashed vertical lines show that 11% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/app1-fig1.jpg"/></fig><p><xref ref-type="fig" rid="app1fig2">Appendix 1—figure 2</xref> shows the topic-based peer review outcomes according to the proportion of total AAB applications linked to each topic. There was a non-significant trend whereby applications focusing on topics that accounted for a greater proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) were more likely to make it to discussion (<xref ref-type="disp-formula" rid="equ5">Equation 5</xref>)(Panel A, p=0.34). When we focused on applications that were discussed and therefore received a priority score, there was no association of mean score for each topic (<xref ref-type="disp-formula" rid="equ6">Equation 6</xref>) with proportion of total AAB applications linked to each topic (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) (Panel B, p=0.14)</p><fig id="app1fig2" position="float"><label>Appendix 1—figure 2.</label><caption><title>Topic peer review outcomes according to proportion of AAB applications linked to specific topics.</title><p>AAB = African American or Black. Panel <bold>A</bold>: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.34). Panel <bold>B</bold>: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.14).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/app1-fig2.jpg"/></fig><p><xref ref-type="table" rid="app1table1">Appendix 1—table 1</xref> shows peer review and funding outcomes according to whether applications were focused on the 16 topics that made up 50% of all resubmission applications with AAB PIs ('AAB Preferred' topics). Peer review outcomes were similar in the two groups, but applications focusing on AAB Preferred topics were 3% less likely to be funded.</p><table-wrap id="app1table1" position="float"><label>Appendix 1—table 1.</label><caption><title>Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.</title><p>Abbreviations as in <xref ref-type="table" rid="table2">Table 2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>Characteristic or outcome</th><th/><th>AAB preferred</th><th>Other</th></tr></thead><tbody><tr><td>Total N (%)</td><td/><td>8193 (25.7)</td><td>23707 (74.3)</td></tr><tr><td>PI AAB</td><td>Yes</td><td>224 (2.7)</td><td>224 (0.9)</td></tr><tr><td>Discussed</td><td>Yes</td><td>6107 (74.5)</td><td>17300 (73.0)</td></tr><tr><td>Priority Score</td><td>Median (IQR)</td><td>31.0 (22.0 to 40.0)</td><td>31.0 (22.0 to 40.0)</td></tr><tr><td>Percentile Ranking</td><td>Median (IQR)</td><td>21.0 (9.8 to 36.0)</td><td>21.0 (10.0 to 35.0)</td></tr><tr><td>Awarded</td><td>Yes</td><td>2419 (29.5)</td><td>7186 (30.3)</td></tr></tbody></table></table-wrap><p><xref ref-type="table" rid="app1table2">Appendix 1—table 2</xref> shows probit regressions on topic and funding outcomes. Without consideration of other variables, an AAB preferred topic was associated with a non-significantly lower probability of funding (Model 1, negative regression coefficient for AAB preferred topic). However, after adjusting for IC award rate alone (Model 2, regression coefficient for AAB preferred topic close to zero, p=NS) as well as other IC award rate and other characteristics, there was no association between AAB preferred topics and funding (Model 3, regression coefficient for AAB preferred topic close to zero, p=NS). The IC award rate was strongly associated with likelihood of funding (Models 2 and 3, regression coefficient positive, p<0.001)</p><table-wrap id="app1table2" position="float"><label>Appendix 1—table 2.</label><caption><title>Probit Regression Models (regression coefficients and standard errors) with focus on topic type.</title><p>Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in <xref ref-type="table" rid="table1">Tables 1</xref> and <xref ref-type="table" rid="table2">2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th/><th>Model 1</th><th>Model 2</th><th>Model 3</th></tr></thead><tbody><tr><td>Intercept</td><td>−0.515***</td><td>−1.397***</td><td>−1.422***</td></tr><tr><td/><td>(0.009)</td><td>(0.065)</td><td>(0.068)</td></tr><tr><td>AAB Preferred Topic</td><td>−0.023</td><td>−0.001</td><td>−0.004</td></tr><tr><td/><td>(0.017)</td><td>(0.017)</td><td>(0.018)</td></tr><tr><td>IC Award Rate</td><td/><td>0.029***</td><td>0.029***</td></tr><tr><td/><td/><td>(0.002)</td><td>(0.002)</td></tr><tr><td>AAB Principal Investigator</td><td/><td/><td>−0.171**</td></tr><tr><td/><td/><td/><td>(0.065)</td></tr><tr><td>Early Stage Investigator</td><td/><td/><td>0.221***</td></tr><tr><td/><td/><td/><td>(0.019)</td></tr><tr><td>Multi-PI Application</td><td/><td/><td>0.035</td></tr><tr><td/><td/><td/><td>(0.020)</td></tr><tr><td>Human Subjects</td><td/><td/><td>−0.020</td></tr><tr><td/><td/><td/><td>(0.018)</td></tr><tr><td>Animal Subjects</td><td/><td/><td>−0.006</td></tr><tr><td/><td/><td/><td>(0.018)</td></tr><tr><td>AIC</td><td>39034.476</td><td>38849.763</td><td>38720.539</td></tr><tr><td>BIC</td><td>39051.217</td><td>38874.874</td><td>38787.502</td></tr><tr><td>Log Likelihood</td><td>−19515.238</td><td>−19421.881</td><td>−19352.270</td></tr><tr><td>Deviance</td><td>39030.476</td><td>38843.763</td><td>38704.539</td></tr><tr><td>Num. obs.</td><td>31900</td><td>31900</td><td>31900</td></tr></tbody></table><table-wrap-foot><fn><p><inline-formula><mml:math id="inf12"><mml:mrow><mml:mmultiscripts><mml:mi mathsize="70%">p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo>⁣</mml:mo><mml:mrow><mml:mi/><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo mathsize="70%" stretchy="false">*</mml:mo></mml:mrow></mml:mrow></mml:mmultiscripts><mml:mo mathsize="70%" stretchy="false"><</mml:mo><mml:mn mathsize="70%">0.001</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf13"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mi/><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.01</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf14"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mo>*</mml:mo></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.05</mml:mn></mml:mrow></mml:math></inline-formula>.</p></fn></table-wrap-foot></table-wrap></sec></boxed-text></app><app id="appendix-2"><title>Appendix 2</title><boxed-text><sec id="s9" sec-type="appendix"><title>Single PI applications</title><p>In our primary analyses, we included all de novo applications, whether authored by one single PI or more than one PI ('multi-PI'). Here we focus on single-PI applications. Our findings are similar.</p><p>Of 76,976 single-PI applications received and reviewed, 6556 were funded, for an overall award rate of 9%. There were 1323 applications, or 2%, submitted by AAB PIs.</p><p><xref ref-type="fig" rid="app2fig1">Appendix 2—figure 1</xref> shows the association of cumulative percentage of AAB applications by the cumulative percentage of topics. There was a nonrandom distribution whereby 11% of topics (or 16 topics) accounted for 50% of AAB applications. We designted a topic as 'AAB Preferred' if it was among the 16 topics that accounted for 50% of AAB applications.</p><fig id="app2fig1" position="float"><label>Appendix 2—figure 1.</label><caption><title>Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.</title><p>Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 8% of all applications submitted by an AAB PI. Dashed vertical lines show that 11% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/app2-fig1.jpg"/></fig><p><xref ref-type="fig" rid="app2fig2">Appendix 2—figure 2</xref> shows the topic-based peer review outcomes according to the proportion of total AAB applications linked to each topic. There was a non-significant trend whereby applications focusing on topics that accounted for a greater proportion of AAB applications (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) were more likely to make it to discussion (<xref ref-type="disp-formula" rid="equ5">Equation 5</xref>)(Panel A, p=0.12). When we focused on applications that were discussed and therefore received a priority score, there was no association of mean score for each topic (<xref ref-type="disp-formula" rid="equ6">Equation 6</xref>) with proportion of total AAB applications linked to each topic (<xref ref-type="disp-formula" rid="equ1">Equation 1</xref>) (Panel B, p=0.84)</p><fig id="app2fig2" position="float"><label>Appendix 2—figure 2.</label><caption><title>Topic peer review outcomes according to proportion of AAB applications linked to specific topics.</title><p>AAB = African American or Black. Panel <bold>A</bold>: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.12). Panel <bold>B</bold>: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of <xref ref-type="fig" rid="fig1">Figure 1</xref>. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.84).</p></caption><graphic mime-subtype="jpeg" mimetype="image" xlink:href="Lauer eLife paper.media/app2-fig2.jpg"/></fig><p><xref ref-type="table" rid="app2table1">Appendix 2—table 1</xref> shows peer review and funding outcomes according to whether applications were focused on the 16 topics that made up 50% of all resubmission applications with AAB PIs ('AAB Preferred' topics). Peer review outcomes were similar in the two groups, but applications focusing on AAB Preferred topics were 7% less likely to be funded.</p><table-wrap id="app2table1" position="float"><label>Appendix 2—table 1.</label><caption><title>Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.</title><p>Abbreviations as in <xref ref-type="table" rid="table2">Table 2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th>Characteristic or outcome</th><th/><th>AAB preferred</th><th>Other</th></tr></thead><tbody><tr><td>Total N (%)</td><td/><td>19134 (24.9)</td><td>57842 (75.1)</td></tr><tr><td>PI AAB</td><td>Yes</td><td>660 (3.4)</td><td>663 (1.1)</td></tr><tr><td>Discussed</td><td>Yes</td><td>8993 (47.0)</td><td>26342 (45.5)</td></tr><tr><td>Priority Score</td><td>Median (IQR)</td><td>40.0 (31.0 to 48.0)</td><td>40.0 (30.0 to 48.0)</td></tr><tr><td>Percentile Ranking</td><td>Median (IQR)</td><td>35.0 (22.0 to 45.0)</td><td>34.0 (21.0 to 44.0)</td></tr><tr><td>Awarded</td><td>Yes</td><td>1541 (8.1)</td><td>5015 (8.7)</td></tr></tbody></table></table-wrap><p><xref ref-type="table" rid="app2table2">Appendix 2—table 2</xref> shows probit regressions on topic and funding outcomes. Without consideration of other variables, an AAB preferred topic was associated with a lower probability of funding (Model 1, negative regression coefficient for AAB preferred topic, p<0.01). However, after adjusting for IC award rate alone (Model 2, regression coefficient for AAB preferred topic close to zero, p=NS) as well as other IC award rate and other characteristics, there was no association between AAB preferred topics and funding (Model 3, regression coefficient for AAB preferred topic close to zero, p=NS). The IC award rate was strongly associated with likelihood of funding (Models 2 and 3, regression coefficient positive, p<0.001)</p><table-wrap id="app2table2" position="float"><label>Appendix 2—table 2.</label><caption><title>Probit Regression Models (regression coefficients and standard errors) with focus on topic type.</title><p>Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in <xref ref-type="table" rid="table1">Tables 1</xref> and <xref ref-type="table" rid="table2">2</xref>.</p></caption><table frame="hsides" rules="groups"><thead><tr><th/><th>Model 1</th><th>Model 2</th><th>Model 3</th></tr></thead><tbody><tr><td>Intercept</td><td>−1.361***</td><td>−1.903***</td><td>−1.856***</td></tr><tr><td/><td>(0.007)</td><td>(0.023)</td><td>(0.030)</td></tr><tr><td>AAB Preferred Topic</td><td>−0.040**</td><td>0.002</td><td>0.008</td></tr><tr><td/><td>(0.015)</td><td>(0.015)</td><td>(0.017)</td></tr><tr><td>IC Award Rate</td><td/><td>0.060***</td><td>0.056***</td></tr><tr><td/><td/><td>(0.002)</td><td>(0.002)</td></tr><tr><td>AAB Principal Investigator</td><td/><td/><td>−0.206***</td></tr><tr><td/><td/><td/><td>(0.057)</td></tr><tr><td>Early Stage Investigator</td><td/><td/><td>0.160***</td></tr><tr><td/><td/><td/><td>(0.016)</td></tr><tr><td>Human Subjects</td><td/><td/><td>−0.070***</td></tr><tr><td/><td/><td/><td>(0.016)</td></tr><tr><td>Animal Subjects</td><td/><td/><td>−0.038*</td></tr><tr><td/><td/><td/><td>(0.016)</td></tr><tr><td>AIC</td><td>44830.312</td><td>44152.773</td><td>44026.919</td></tr><tr><td>BIC</td><td>44848.814</td><td>44180.526</td><td>44091.678</td></tr><tr><td>Log Likelihood</td><td>−22413.156</td><td>−22073.386</td><td>−22006.460</td></tr><tr><td>Deviance</td><td>44826.312</td><td>44146.773</td><td>44012.919</td></tr><tr><td>Num. obs.</td><td>76976</td><td>76976</td><td>76976</td></tr><tr><td colspan="4"><inline-formula><mml:math id="inf15"><mml:mrow><mml:mmultiscripts><mml:mi mathsize="70%">p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo>⁣</mml:mo><mml:mrow><mml:mi/><mml:mo mathsize="70%" stretchy="false">*</mml:mo><mml:mo mathsize="70%" stretchy="false">*</mml:mo></mml:mrow></mml:mrow></mml:mmultiscripts><mml:mo mathsize="70%" stretchy="false"><</mml:mo><mml:mn mathsize="70%">0.001</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf16"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mrow><mml:mi/><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.01</mml:mn></mml:mrow></mml:math></inline-formula>; <inline-formula><mml:math id="inf17"><mml:mrow><mml:mmultiscripts><mml:mi>p</mml:mi><mml:mprescripts/><mml:none/><mml:mo>*</mml:mo></mml:mmultiscripts><mml:mo><</mml:mo><mml:mn>0.05</mml:mn></mml:mrow></mml:math></inline-formula></td></tr></tbody></table></table-wrap></sec></boxed-text></app></app-group></back><sub-article article-type="decision-letter" id="sa1"><front-stub><article-id pub-id-type="doi">10.7554/eLife.67173.sa1</article-id><title-group><article-title>Decision letter</article-title></title-group><contrib-group><contrib contrib-type="editor"><name><surname>Rosen</surname><given-names>Cliff J</given-names></name><role>Reviewing Editor</role><aff><institution>Maine Medical Center Research Institute</institution><country>United States</country></aff></contrib></contrib-group><contrib-group><contrib contrib-type="reviewer"><name><surname>Isales</surname><given-names>Carlos</given-names> </name><role>Reviewer</role><aff><institution>Medical College of Georgia at Augusta University</institution><country>United States</country></aff></contrib><contrib contrib-type="reviewer"><name><surname>Rosen</surname><given-names>Cliff J</given-names></name><role>Reviewer</role><aff><institution>Maine Medical Center Research Institute</institution><country>United States</country></aff></contrib></contrib-group></front-stub><body><boxed-text><p>Our editorial process produces two outputs: i) <ext-link ext-link-type="uri" xlink:href="https://sciety.org/articles/activity/10.1101/2020.12.27.424490">public reviews</ext-link> designed to be posted alongside <ext-link ext-link-type="uri" xlink:href="https://www.biorxiv.org/content/10.1101/2020.12.27.424490v2">the preprint</ext-link> for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.</p></boxed-text><p><bold>Acceptance summary:</bold></p><p>This is an important paper that nicely addresses a topical and timely issue relevant to extramural NIH funding. The knowledge gained from your analyses will undoubtedly add to the vigorous discussion about how to move forward. We believe you have revised the paper according to the comments of the reviewers. It was clear from your edits and explanation that some of the comments were outside of the scope of the paper, and we understand that. In the meantime, the other responses were noted to be thoughtful and thorough.</p><p><bold>Decision letter after peer review:</bold></p><p>Thank you for submitting your article "National Institutes of Health Institute and Center Award Rates and Funding Disparities" for consideration by <italic>eLife</italic>. Your article has been reviewed by 4 peer reviewers, including Cliff J Rosen as the Reviewing Editor and Reviewer #2, and the evaluation has been overseen by Mone Zaidi as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Carlos Isales (Reviewer #1).</p><p>The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.</p><p>In general there was enthusiasm for the topical nature of the subject and the importance of defining inequities in NIH funding for African Americans. However there were issues raised about the analysis, the absence of significant data sets in 50% of the topics, and approach to the analysis. Notwithstanding these potentially resolvable issues, there was consensus that a revised paper will be critical to enhance discussion of this important area and move toward possible solutions.</p><p>Essential Revisions</p><p>Please refer to individual Recommendations for the Authors, and specifically focus on:</p><p>1. Further defending the overall approach to the analysis by providing new data and/or offering explanations to issues raised in the four reviews.</p><p>2. Further refine statistical analysis particularly as noted by Reviewer 3.</p><p><italic>Reviewer #1 (Recommendations for the authors):</italic></p><p>Please respond to the following questions by explaining further and/or providing data.</p><p>1. The authors don't discuss the details about how Institute assignments are made. Agreed that this is done by CSR, a number of variables go into this: (a) the π may attempt to direct the application to a specific Institute or Center by including keywords in abstract or title; (b) the π may request a specific Institute or Center.</p><p>2. Is part of the issue related to having an Institute named Minority Health, but then have that Institute be relatively underfunded? Would you suggest increased funding for specific Institutes?</p><p>3. Not clear if the numbers included are all the R01's received or just the ones received by CSR. Some R01's go directly to Institutes (e.g. RFA's); as do P01's and training grants. If the analysis is for total grants submitted, in sub analysis, were there any differences in AAB funding rates for those applications that went directly to a specific Institute or Center and were of a different topic that the 10% of 148 topics, i.e were there differences in outcomes in π assigned vs CSR assigned Institutes?</p><p>4. For resubmitted R01's do 100% go back to the same Institute-Center or what % are reassigned to a different Institute?</p><p>5. How was an "Early-Stage investigator Table 5 defined? Did seniority (# of publications, h-index) and funding success correlate with specific Institutes?</p><p>6. Do we know if there were differences between number of applications submitted before successful funding for AAB investigators? Or whether AAB investigators were less likely to submit an A1 application after initial triage?</p><p><italic>Reviewer #2 (Recommendations for the authors):</italic></p><p>Please address the following issues.</p><p>1. It is still problematic why topic choice of community engagement or population studies fare worse overall and particularly at an Institute such as AI compared to GM if both have the relatively same proportion of preferred topics, and both have relatively high budgets compared to other institutes. Please provide data if available.</p><p>2. Are there other factors imbedded within ICs or their leadership that determine priorities independent of funding scores?</p><p>3. Also, are there one or more ICs that drive the correlations between IC funding and preferred topics or PIs? Please provide any data.</p><p>4. In addition it should be noted that only 2% of all PIs are AABs so that represents a potential source of error.</p><p>5. The data are only up to 2015; it has now been five years and things have changed dramatically at NIH and in society. Do the authors have more recent data? There are now many more multiple π applications including AABs that may not be the contact π yet are likely to choose to be in a preferred topic area.</p><p>6. Please discuss potential resolutions to this issue; In other words we now know that the disparity in funding is such that AAB RO1s do worse than white PIs because they are selecting topics that end up at institutes with lower funding rates. Should the institutes devote a set aside for these topic choices to balance the portfolio of the IC and equal the playing field for AABs?</p><p><italic>Reviewer #3 (Recommendations for the authors):</italic></p><p>1. The limited introductory material, and the Discussion do not appropriately place this important, indeed critical, work in the context it deserves. In general, the expectation for a general science journal such as <italic>eLife</italic> demands greater attention to scholarship and contextualization for the anticipated broad audience. Indeed, the Cover Letter argues that this work is relevant to the "NIH extramural research community" and that it is correcting a "mistaken belief" related to their Hoppe et al. 2019 paper. Furthermore, the authors reference the Taffe and Gilpin <italic>eLife</italic> Feature Article which explains at some length why these matters are of such pressing importance. This is indeed a very important study to publish, although perhaps it should be understood, and written, more as an extension of the ongoing inquiry into NIH funding practices and less as a correction of "mistaken belief" related to the prior paper. It is suggested that at the very least the Introduction and Discussion should address the impact of the funding disparity on (1) AA/B researchers and (2) research on the health concerns of Black Americans, in a scholarly way, including reference to recent materials on this topic to guide the interested reader.</p><p>2. Introduction Ln26-41: This section, currently the bulk of the Introduction, fails to describe why applications in some topics would be assigned to one IC over another. The "for example" section is a misdirection and distracting. The key question, given the data in Figure 1 and Table 1, is why a given grant would be assigned to NIMHD or NINR instead of the disease-domain parent institute.</p><p>2. The Methods indicate that the 15 topics that accounted for 50% of the AA/B applications were designated as AA/B-preferred for analysis. There is no rationale or justification provided for this choice. Why not the topics that accounted for 66% or 33% or 75%? It would appear critical to provide a strong justification, and some indication of what impact other choices for what is an apparently arbitrary cutoff would have on the central analyses.</p><p>3. The Conclusion section presages what will be takeaway message for many readers, i.e. "likely primarily due to". Yet, from this analysis, this conclusion which is likely to be assumed to apply to all AA/B application leaves any disparity of funding for the topics associated with half of the applications with AA/B PIs ignored.</p><p>4. The Methods indicate that the definition of AA/B-preferred ICs was the top quartile (6/24 ICs), but there is no rationale or justification provided for this choice. In the absence of a strong justification, it would appear critical to provide some indication of what impact other essentially arbitrary cutoff choices would have on the central analyses. For example, of the five (Fogerty International Center appears to be omitted due to "privacy concerns") ICs with the highest percentage of AA/B applications shown in Table 1, the two lowest are 2.13% (AI) and 2.06% (ES) which are much closer to 12 remaining ICs in AA/B percentage than they are to the third lowest of the AA/B upper quartile. Three of these (DE, 1.99% AA/B; DA, 1.81% AA/B; MH, 1.69% AA/B) are more obviously grouped with AI and ES in terms of AA/B award rate than are the three most extreme AA/B-preferred ICs (HD, NR, MD). This differs from the choices made in Hoppe et al. 2019 where it is fairly obvious from Figure 3A why the 8 topic clusters receiving a high percentage of AA/B applications, and the 8 with zero, were selected for analysis. From Table 1 of this manuscript, the top quartile of ICs account for only 27.3% of AA/B applications. Inclusion of DE, DA and MH would increase this to 48.5% of the AA/B applications, more consistent with the 50% threshold used by the authors to designate AA/B-preferred topics. These choices are not explained or justified and it is critical to give some indication of whether the central conclusions depend on such arbitrary choices.</p><p>5. Table 1 is missing a statistic which is key to the authors' conclusions. The award rate for applications with AA/B versus white PIs should be listed per IC, instead of only the aggregate award rate. The entire argument hinges on the presumption that there is no difference in the white/AA/B success rate ratio across the ICs, yet we have seen no analysis of this in prior papers nor does the information occur here.</p><p>6. Page 3, Line 102: The phrase "submitted [applications] to" in the context of IC assignment is an improper misrepresentation. This requires correcting throughout, including the Abstract and Line 163 of the Discussion. While applicants can request assignment to a given I or C, this is not always respected and thus this phrasing should reflect NIH assignment, not π choice. This aspect of application assignment (and any dual assignments and selection/refusal to pick up as a secondary IC) should be incorporated into the analysis. Are applications with AA/B PIs more or less likely to be assigned by the NIH's receipt and referral process to the lower-success institutes instead of the parent IC? Are applications with AA/B PIs more likely to request those low-success rate ICs? Are assignment requests equivalently respected? Are applications with AA/B PIs more or less likely to be dual-assigned and/or selected for funding by the secondary IC? What about the role of shared funding between ICs?</p><p>7. Page 5, Line 123: While the regression coefficients are useful for a sophisticated audience, it is essential to report the results in a more easily understandable manner for a more general audience. Most importantly the results should be expressed in a way that is comparable with the prior analyses provide by Ginther, Hoppe and their colleagues and indeed is mentioned in the first paragraph of the Introduction. The reader should be informed clearly what percentage of the disparity is accounted for here, similar to the ~20-25% of the racial bias effect accounted for by topic in Hoppe et al. 2019 and by CV metrics in Ginther et al. 2018. This will then help to make the Abstract and Conclusion statements more precise.</p><p>8. Figure 1 is confusing in juxtaposition with Table 1 and in context with the overall thrust of the paper. It is suggested that an additional panel be added in which the X axis is the AA/B percentage of applications to that IC, as in Table 1. That is what the reader is expecting at first, and it takes some close reading to figure out these are rankings of the ICs based on submissions by topic. In addition, the six AA/B-preferred ICs should be clearly indicated on-figure by means of colored text, bolded text, a circle or something similar.</p><p>9. Discussion, page 10: The second paragraph refers to the funding ecology and ICs with lower award rates, reflecting language used throughout the manuscript. There does not appear to be any description of this ecology and the general science reader will not understand the tremendous disparity in IC funding (e.g., NIMHD and NINR, the extreme outliers in % of AA/B π applications, received 0.8% and 0.4% of the FY2000 budget) without this information. It is essential to add columns in Table 1 that show the average amount of money assigned to each IC across the relevant FYs and to also represent this as a percentage to assist the reader. This context then needs to be mentioned in both Introduction and Discussion where points about IC award rates and funding "ecology" are being made. By way of analogy, the readership of <italic>eLife</italic> expects an extensive mechanistic explanation without key omissions to accompany empirical results and this is similar.</p><p>10. Discussion, page 10: The second paragraph is an outline of the illogic captured in this entire investigation. It cleverly draws indirect lines from the overall AA/B disparity of award to the success rate of ICs while seemingly ignoring the fact the analysis reflects only the topics preferred by half of the AA/B applications and reports on only 27.3% of AA/B applications. It is unclear why the primary analysis approach for this inquiry is not simply to examine white vs AA/B success rates across each of the ICs. This would be a much simpler and intuitive answer to what appear to be the authors originating question. The Hoppe report admitted, but tried to undersell, the critically important finding that even in the least-successful quintile of topics, the AA/B applications faired more poorly and it would be critical to understand if that is also true in this analysis of IC assignment.</p><p>11. Discussion, page 10: The third and fourth paragraphs are written in imprecise language that does a disservice to the reader understanding of the size of the effects that are being reported. The Hoppe sentence should just include the ~20% number. The comments about "just as well", "even better", "fare a bit worse" and "to a much lower degree" need quantitation.</p><p>12. Discussion, page 11, Line 179: The statement is utterly speculative and should be removed or backed with better evidence.</p><p>13. Conclusion, page 11: The use of "explain disparities to an even greater degree" needs to be accompanied by an estimate of the effect size, similar to the data in Hoppe underlying the "topic choice does partially explain" beginning of the sentence. In Hoppe, it was something on the order of accounting for only 21% of the white/AAB disparity of grants that were discussed. In this manuscript the authors need to be clearer about what amount of which racial disparity does "an even greater degree" reflect?</p><p>14. Conclusion, page 11: The third sentence again inappropriately draws conclusions from the indirect logic available from the design. The authors need to show the IC-specific award rates for white and AA/B applicants within each IC and contrast these by IC-specific award rate if they wish to make this statement.</p><p>15. Supplementary Materials: As mentioned above, there is not a rationale provided for excluding the competing continuation applications from the main analysis. This is peculiar given that these were included in Hoppe and the current manuscript is framed as an explanation of those prior results. If the outcome is the same, it would make more sense to make the full sample the subject of the main paper and the more selected sub-sample the supplement.</p><p>Reviewer #4 (Recommendations for the authors):</p><p>The paper is well written and uses appropriate methods in the described analysis. While this analysis does contribute to understanding the differential outcomes that have been previously described in the funding of AAB investigators, there are still unanswered questions about the success rate differentials. The topics examined only account for 50% of the applications. If there is little or no biases as is suggested by the authors, the other 50% should be able to be accounted for using the analysis performed in this paper. The authors should consider adding a bit of conditional conclusion of these findings.</p></body></sub-article><sub-article article-type="reply" id="sa2"><front-stub><article-id pub-id-type="doi">10.7554/eLife.67173.sa2</article-id><title-group><article-title>Author response</article-title></title-group></front-stub><body><disp-quote content-type="editor-comment"><p>Essential Revisions</p></disp-quote><p>Before going into the Editors’ and Reviewers’ recommendations in detail, I want to summarize a discussion we had regarding some of Reviewer 3’s comments:</p><p>– Reviewer 3 states: “The second paragraph is an outline of the illogic captured in this entire investigation. It cleverly draws indirect lines from the overall AA/B disparity of award to the success rate of ICs while seemingly ignoring the fact the analysis reflects only the topics preferred by half of the AA/B applications and reports on only 27.3% of AA/B applications. It is unclear why the primary analysis approach for this inquiry is not simply to examine white vs AA/B success rates across each of the ICs.”</p><p>– We did not focus on white vs AA/B because that is not the question we were attempting to answer.</p><p>– In the original Hoppe paper there is a sentence in the discussion which states, “Our analysis shows that all three of the factors that underlie the funding gap—preference for some topics over others, assignment of poorer scores, and decision to discuss an application—revolve around decisions made by reviewers.”</p><p>– The point of this analysis is that preference for some topics over others do <italic>not</italic> revolve around decisions made by reviewers. Instead, the funding preferences are due to IC award rates.</p><p>– We agreed that we needed to clarify the purpose of our analyses, and we feel have done that in the revised paper.</p><p>– You agreed that an analysis focusing on how peer review and IC’s approach topics (not PI’s) would be of interest to eLife. Therefore we decided to revise the paper and submit this revision to you.</p><disp-quote content-type="editor-comment"><p>Please refer to individual Recommendations for the Authors, and specifically focus on:</p><p>1. Further defending the overall approach to the analysis by providing new data and/or offering explanations to issues raised in the four reviews.</p></disp-quote><p>We have made a number of changes, including more detailed explanations and additional analyses.</p><disp-quote content-type="editor-comment"><p>2. Further refine statistical analysis particularly as noted by Reviewer 3.</p></disp-quote><p>Notwithstanding our discussion above regarding Reviewer 3’s comments, we have refined and extended our analyses in response to a number of comments with which we completely agree:</p><p>– New Figure 1 shows in quantitative terms the skewed nature of AAB π preferences. The Figure will help the reader understand why the 50% cut-off was reasonable.</p><p>– Reviewer 3 takes issue with our categorical approach (e.g. identifying 10% of topics that account for 50% of AAB π applications). Therefore, we calculated AAB preference for each topic as a continuous variable (equation 1) and show peer review and funding outcomes according to AAB preference in new Figures 2 and 5. These Figures reinforce the absence of an association of peer review outcomes with AAB preference for certain topics.</p><p>– We also present an extended descriptive analysis in new Table 4 showing that using a 70% cut-point, instead of 50%, yields similar findings.</p><p>– We took steps to better explain our terms and regression analyses. In the Methods section we present a series of equations to explain our terms, and we refer to those equations in other parts of the manuscript. In the legend to Table 5 we explain what the regression coefficients and p-values mean. In the text in the Results and Discussion sections we attempt to “walk the reader through” each of the models presented in the regression table.</p><disp-quote content-type="editor-comment"><p>Reviewer #1 (Recommendations for the authors):</p><p>Please respond to the following questions by explaining further and/or providing data.</p><p>1. The authors don't discuss the details about how Institute assignments are made. Agreed that this is done by CSR, a number of variables go into this: (a) the π may attempt to direct the application to a specific Institute or Center by including keywords in abstract or title; (b) the π may request a specific Institute or Center.</p></disp-quote><p>We provide information, as well as a citation, on Institute assignments.</p><disp-quote content-type="editor-comment"><p>2. Is part of the issue related to having an Institute named Minority Health, but then have that Institute be relatively underfunded? Would you suggest increased funding for specific Institutes?</p></disp-quote><p>This is an important question, but it is related to policy and legislative priorities. Therefore, we feel this is not the right venue for NIH staff to publish such a discussion. However, we did add text in the Discussion on how NIH is prioritizing funding for specific topics.</p><disp-quote content-type="editor-comment"><p>3. Not clear if the numbers included are all the R01's received or just the ones received by CSR. Some R01's go directly to Institutes (e.g. RFA's); as do P01's and training grants. If the analysis is for total grants submitted, in subanalysis, were there any differences in AAB funding rates for those applications that went directly to a specific Institute or Center and were of a different topic that the 10% of 148 topics, i.e were there differences in outcomes in π assigned vs CSR assigned Institutes?</p></disp-quote><p>The vast majority (88%) of the applications were reviewed by CSR. Given that only 12% of applications were reviewed outside CSR, we did not conduct stratified analyses.</p><disp-quote content-type="editor-comment"><p>4. For resubmitted R01's do 100% go back to the same Institute-Center or what % are reassigned to a different Institute?</p></disp-quote><p>As a rule, resubmitted applications go back to the same IC.</p><disp-quote content-type="editor-comment"><p>5. How was an "Early-Stage investigator Table 5 defined? Did seniority (# of publications, h-index) and funding success correlate with specific Institutes?</p></disp-quote><p>We provide a definition for Early-Stage Investigator. The second question is about PI’s, not topics, and therefore beyond the scope of this paper.</p><disp-quote content-type="editor-comment"><p>6. Do we know if there were differences between number of applications submitted before successful funding for AAB investigators? Or whether AAB investigators were less likely to submit an A1 application after initial triage?</p></disp-quote><p>This question is about PI’s, not topics, and therefore beyond the scope of this paper. However, it was addressed in the original Hoppe paper.</p><disp-quote content-type="editor-comment"><p>Reviewer #2 (Recommendations for the authors):</p><p>Please address the following issues.</p><p>1. It is still problematic why topic choice of community engagement or population studies fare worse overall and particularly at an Institute such as AI compared to GM if both have the relatively same proportion of preferred topics, and both have relatively high budgets compared to other institutes. Please provide data if available.</p></disp-quote><p>Different IC’s do not have the same proportion of preferred topics. Tables 3 and 4 (which is a new table) shows the data. For example, in new Table 4, using a 70% cut-off, AAB preferred topics were disproportionately assigned to the 6 IC’s that received the highest proportion of applications with AAB PIs (25% vs. 14%).</p><disp-quote content-type="editor-comment"><p>2. Are there other factors imbedded within ICs or their leadership that determine priorities independent of funding scores?</p></disp-quote><p>We explain briefly how ICs make funding decisions and cite IC-specific strategic plans.</p><disp-quote content-type="editor-comment"><p>3. Also, are there one or more ICs that drive the correlations between IC funding and preferred topics or PIs? Please provide any data.</p></disp-quote><p>These data are shown in Figure 4, but we can see how this was not clear. We relabeled the X axis as “Proportion of Applications on AAB Preferred Topic (%).” In the Figure legend we added, “In other words, the X-axis value would be 33 percent if one-third of all IC applications focused on AAB Preferred topics, namely those 15 topics that together accounted for 50% of all applications with AAB PIs.” As suggested by Reviewer 3, we highlight the ICs that received the highest proportion of applications with AAB PIs; Table 2 shows that the award rates for these ICs were lower than for all other ICs.</p><disp-quote content-type="editor-comment"><p>4. In addition it should be noted that only 2% of all PIs are AABs so that represents a potential source of error.</p></disp-quote><p>We agree and added text in the Discussion.</p><disp-quote content-type="editor-comment"><p>5. The data are only up to 2015; it has now been five years and things have changed dramatically at NIH and in society. Do the authors have more recent data? There are now many more multiple π applications including AABs that may not be the contact π yet are likely to choose to be in a preferred topic area.</p></disp-quote><p>Since this is a re-analysis of the Hoppe project, we focused on those data. We agree that an analysis of more recent data would be of value, particularly to look for secular trends, but this is beyond the scope of this paper. We agree with the concerns about multi-PI applications; therefore we include a supplemental analysis on single-PI applications.</p><disp-quote content-type="editor-comment"><p>6. Please discuss potential resolutions to this issue; In other words we now know that the disparity in funding is such that AAB RO1s do worse than white PIs because they are selecting topics that end up at institutes with lower funding rates. Should the institutes devote a set aside for these topic choices to balance the portfolio of the IC and equal the playing field for AABs?</p></disp-quote><p>This is an important question, but it is related to policy and legislative priorities. Therefore, we feel this is not the right venue for NIH staff to publish such a discussion. However, we did add text in the Discussion on how NIH is prioritizing funding for specific topics. Furthermore, we hope that the paper communicates the message that the problem is not purported peer review biases against certain topics (Figures 2 and 3, Table 6 models 2 and 3).</p><p>This paper is not stating that the entire disparity in NIH R01 funding between AAB and whites is due to the topics they choose. IC funding ecology is one of many factors that contribute to the funding disparity. This has been clarified in the Discussion section.</p><disp-quote content-type="editor-comment"><p>Reviewer #3 (Recommendations for the authors):</p><p>1. The limited introductory material, and the Discussion do not appropriately place this important, indeed critical, work in the context it deserves. In general, the expectation for a general science journal such as eLife demands greater attention to scholarship and contextualization for the anticipated broad audience. Indeed, the Cover Letter argues that this work is relevant to the "NIH extramural research community" and that it is correcting a "mistaken belief" related to their Hoppe et al. 2019 paper. Furthermore, the authors reference the Taffe and Gilpin eLife Feature Article which explains at some length why these matters are of such pressing importance. This is indeed a very important study to publish, although perhaps it should be understood, and written, more as an extension of the ongoing inquiry into NIH funding practices and less as a correction of "mistaken belief" related to the prior paper. It is suggested that at the very least the Introduction and Discussion should address the impact of the funding disparity on (1) AA/B researchers and (2) research on the health concerns of Black Americans, in a scholarly way, including reference to recent materials on this topic to guide the interested reader.</p></disp-quote><p>As noted previously, we have clarified that this analysis focuses on topics, not PIs.</p><disp-quote content-type="editor-comment"><p>2. Introduction Ln26-41: This section, currently the bulk of the Introduction, fails to describe why applications in some topics would be assigned to one IC over another. The "for example" section is a misdirection and distracting. The key question, given the data in Figure 1 and Table 1, is why a given grant would be assigned to NIMHD or NINR instead of the disease-domain parent institute.</p></disp-quote><p>We provide information, as well as a citation, on Institute assignments.</p><disp-quote content-type="editor-comment"><p>2. The Methods indicate that the 15 topics that accounted for 50% of the AA/B applications were designated as AA/B-preferred for analysis. There is no rationale or justification provided for this choice. Why not the topics that accounted for 66% or 33% or 75%? It would appear critical to provide a strong justification, and some indication of what impact other choices for what is an apparently arbitrary cutoff would have on the central analyses.</p></disp-quote><p>This is an excellent point. We added analyses on AAB Preference as a continuous variable (equation 1, Figures 2 and 5), present a cumulative distribution plot (Figure 1), and test a different cut-point (Figure 1 and Table 4).</p><disp-quote content-type="editor-comment"><p>3. The Conclusion section presages what will be takeaway message for many readers, i.e. "likely primarily due to". Yet, from this analysis, this conclusion which is likely to be assumed to apply to all AA/B application leaves any disparity of funding for the topics associated with half of the applications with AA/B PIs ignored.</p></disp-quote><p>This is an analysis of topics, not AAB PIs. As noted, we considered topic preferences as continuous and categorical variables (equation 1; Figures 1, 2, and 5; new Table 4). We clarify in the re-written conclusion where our interpretations are based (mainly from the probit regressions shown in Tables 5 and 6).</p><disp-quote content-type="editor-comment"><p>4. The Methods indicate that the definition of AA/B-preferred ICs was the top quartile (6/24 ICs), but there is no rationale or justification provided for this choice. In the absence of a strong justification, it would appear critical to provide some indication of what impact other essentially arbitrary cutoff choices would have on the central analyses. For example, of the five (Fogerty International Center appears to be omitted due to "privacy concerns") ICs with the highest percentage of AA/B applications shown in Table 1, the two lowest are 2.13% (AI) and 2.06% (ES) which are much closer to 12 remaining ICs in AA/B percentage than they are to the third lowest of the AA/B upper quartile. Three of these (DE, 1.99% AA/B; DA, 1.81% AA/B; MH, 1.69% AA/B) are more obviously grouped with AI and ES in terms of AA/B award rate than are the three most extreme AA/B-preferred ICs (HD, NR, MD). This differs from the choices made in Hoppe et al. 2019 where it is fairly obvious from Figure 3A why the 8 topic clusters receiving a high percentage of AA/B applications, and the 8 with zero, were selected for analysis. From Table 1 of this manuscript, the top quartile of ICs account for only 27.3% of AA/B applications. Inclusion of DE, DA and MH would increase this to 48.5% of the AA/B applications, more consistent with the 50% threshold used by the authors to designate AA/B-preferred topics. These choices are not explained or justified and it is critical to give some indication of whether the central conclusions depend on such arbitrary choices.</p></disp-quote><p>We agree with the reviewer that we should beyond one categorical cut-off. See comments to Editor; Figures 1, 2, and 5; Table 4.</p><disp-quote content-type="editor-comment"><p>5. Table 1 is missing a statistic which is key to the authors' conclusions. The award rate for applications with AA/B versus white PIs should be listed per IC, instead of only the aggregate award rate. The entire argument hinges on the presumption that there is no difference in the white/AA/B success rate ratio across the ICs, yet we have seen no analysis of this in prior papers nor does the information occur here.</p></disp-quote><p>This paper is an analysis of topics, not PIs.</p><disp-quote content-type="editor-comment"><p>6. Page 3, Line 102: The phrase "submitted [applications] to" in the context of IC assignment is an improper misrepresentation. This requires correcting throughout, including the Abstract and Line 163 of the Discussion. While applicants can request assignment to a given I or C, this is not always respected and thus this phrasing should reflect NIH assignment, not π choice. This aspect of application assignment (and any dual assignments and selection/refusal to pick up as a secondary IC) should be incorporated into the analysis. Are applications with AA/B PIs more or less likely to be assigned by the NIH's receipt and referral process to the lower-success institutes instead of the parent IC? Are applications with AA/B PIs more likely to request those low-success rate ICs? Are assignment requests equivalently respected? Are applications with AA/B PIs more or less likely to be dual-assigned and/or selected for funding by the secondary IC? What about the role of shared funding between ICs?</p></disp-quote><p>We agree, and changed the language to “assigned to”. We address dual assignments.</p><disp-quote content-type="editor-comment"><p>7. Page 5, Line 123: While the regression coefficients are useful for a sophisticated audience, it is essential to report the results in a more easily understandable manner for a more general audience. Most importantly the results should be expressed in a way that is comparable with the prior analyses provide by Ginther, Hoppe and their colleagues and indeed is mentioned in the first paragraph of the Introduction. The reader should be informed clearly what percentage of the disparity is accounted for here, similar to the ~20-25% of the racial bias effect accounted for by topic in Hoppe et al. 2019 and by CV metrics in Ginther et al. 2018. This will then help to make the Abstract and Conclusion statements more precise.</p></disp-quote><p>In the table legends for Tables 5 and 6, equation 7, Results text, and Discussion / Conclusion text, we take steps to be precise in our explanations, including walking the reader through the tables.</p><disp-quote content-type="editor-comment"><p>8. Figure 1 is confusing in juxtaposition with Table 1 and in context with the overall thrust of the paper. It is suggested that an additional panel be added in which the X axis is the AA/B percentage of applications to that IC, as in Table 1. That is what the reader is expecting at first, and it takes some close reading to figure out these are rankings of the ICs based on submissions by topic. In addition, the six AA/B-preferred ICs should be clearly indicated on-figure by means of colored text, bolded text, a circle or something similar.</p></disp-quote><p>The ranking of ICs is by award rate; we added language to the Table legend. We added a color-legend to specific ICs in Figure 4.</p><disp-quote content-type="editor-comment"><p>9. Discussion, page 10: The second paragraph refers to the funding ecology and ICs with lower award rates, reflecting language used throughout the manuscript. There does not appear to be any description of this ecology and the general science reader will not understand the tremendous disparity in IC funding (e.g., NIMHD and NINR, the extreme outliers in % of AA/B π applications, received 0.8% and 0.4% of the FY2000 budget) without this information. It is essential to add columns in Table 1 that show the average amount of money assigned to each IC across the relevant FYs and to also represent this as a percentage to assist the reader. This context then needs to be mentioned in both Introduction and Discussion where points about IC award rates and funding "ecology" are being made. By way of analogy, the readership of eLife expects an extensive mechanistic explanation without key omissions to accompany empirical results and this is similar.</p></disp-quote><p>We added a column in Table 1 on the 2015 appropriations to each IC.</p><disp-quote content-type="editor-comment"><p>10. Discussion, page 10: The second paragraph is an outline of the illogic captured in this entire investigation. It cleverly draws indirect lines from the overall AA/B disparity of award to the success rate of ICs while seemingly ignoring the fact the analysis reflects only the topics preferred by half of the AA/B applications and reports on only 27.3% of AA/B applications. It is unclear why the primary analysis approach for this inquiry is not simply to examine white vs AA/B success rates across each of the ICs. This would be a much simpler and intuitive answer to what appear to be the authors originating question. The Hoppe report admitted, but tried to undersell, the critically important finding that even in the least-successful quintile of topics, the AA/B applications faired more poorly and it would be critical to understand if that is also true in this analysis of IC assignment.</p></disp-quote><p>See above at the beginning of this letter.</p><disp-quote content-type="editor-comment"><p>11. Discussion, page 10: The third and fourth paragraphs are written in imprecise language that does a disservice to the reader understanding of the size of the effects that are being reported. The Hoppe sentence should just include the ~20% number. The comments about "just as well", "even better", "fare a bit worse" and "to a much lower degree" need quantitation.</p></disp-quote><p>In the text descriptions of the probit regression analyses, we provided quantitative terms.</p><disp-quote content-type="editor-comment"><p>12. Discussion, page 11, Line 179: The statement is utterly speculative and should be removed or backed with better evidence.</p></disp-quote><p>We agree; new Figure 2 provides that evidence.</p><disp-quote content-type="editor-comment"><p>13. Conclusion, page 11: The use of "explain disparities to an even greater degree" needs to be accompanied by an estimate of the effect size, similar to the data in Hoppe underlying the "topic choice does partially explain" beginning of the sentence. In Hoppe, it was something on the order of accounting for only 21% of the white/AAB disparity of grants that were discussed. In this manuscript the authors need to be clearer about what amount of which racial disparity does "an even greater degree" reflect?</p></disp-quote><p>We have rewritten the conclusion to be more quantitative and to indicate where in the probit regression models we base our interpretations.</p><disp-quote content-type="editor-comment"><p>14. Conclusion, page 11: The third sentence again inappropriately draws conclusions from the indirect logic available from the design. The authors need to show the IC-specific award rates for white and AA/B applicants within each IC and contrast these by IC-specific award rate if they wish to make this statement.</p></disp-quote><p>See above at the beginning of this letter.</p><disp-quote content-type="editor-comment"><p>15. Supplementary Materials: As mentioned above, there is not a rationale provided for excluding the competing continuation applications from the main analysis. This is peculiar given that these were included in Hoppe and the current manuscript is framed as an explanation of those prior results. If the outcome is the same, it would make more sense to make the full sample the subject of the main paper and the more selected sub-sample the supplement.</p></disp-quote><p>We try to explain better why it is inappropriate to combine first submissions with resubmissions and why to exclude competing renewals. Both resubmissions and competing renewals receive systematically different peer review outcomes.</p><disp-quote content-type="editor-comment"><p>Reviewer #4 (Recommendations for the authors):</p><p>The paper is well written and uses appropriate methods in the described analysis. While this analysis does contribute to understanding the differential outcomes that have been previously described in the funding of AAB investigators, there are still unanswered questions about the success rate differentials. The topics examined only account for 50% of the applications. If there is little or no biases as is suggested by the authors, the other 50% should be able to be accounted for using the analysis performed in this paper. The authors should consider adding a bit of conditional conclusion of these findings.</p></disp-quote><p>As noted, we added AAB Preference as a continuous variable (equation 1, Figure 2 and 5) and tested a different cut-point (Figure 1, Table 4).</p></body></sub-article></article>