The National Institutes of Health established career development (K) awards to promote the careers of promising new investigators by providing mentorship, salary support, resources, and protected academic time.Reference Carlson, Wang and Scott 1 , Reference Kontaridis 2 The training objectives for the K awards are designed to support junior investigators as they transition to independent funding, but successful candidates are often those who have already had early exposure to research and, at minimum, have a faculty appointment of instructor. The National Heart, Lung, and Blood Institute is the primary funder of the training grants in cardiovascular disease with substantial amounts for their K awards. The priority scores for funding these K awards have risen slightly from 30 in 2015 and 2016 to 32 in 2017. 3 However, because the prevalence of paediatric heart disease is well below that of adult-onset cardiovascular disease, the public health impact may be perceived to be less during review and the applicant studying paediatric heart disease may be disadvantaged in attaining a fundable priority score.Reference Hill, Chiswell and Califf 4 Currently, the number of paediatric cardiovascular trials registered in ClinicalTrials.gov is substantially below the number of adult cardiovascular trials and also below the number of trials in other paediatric sub-specialties.Reference Hill, Chiswell and Califf 4 – Reference Califf, Zarin and Kramer 6 Given the increasing competition for available funding, the numbers of paediatric clinician scientists have declined.Reference Kontaridis 2 , Reference Houser 7 , Reference Yin, Gabrilove and Jackson 8
History of the Pediatric Heart Network training
Since 2001, the Pediatric Heart Network funded by National Heart, Lung, and Blood Institute has sought to train the next generation of clinician scientists in the conduct of multicentre clinical research as one of its stated objectives.Reference Burns, Pemberton and Pearson 9 Initially, the classic National Institutes of Health model for a Clinical Research Skills Development Core 10 – Reference Schulman, Suncion and Karantalis 13 was used and two centres, Children’s Hospital of Philadelphia and Children’s Hospital of New York-Presbyterian, successfully competed for funds. These centres received support for providing education, training, and mentoring to their own candidates and faculty using locally developed programmes. Although their early career investigators were exposed to multicentre research, they had no individual projects to facilitate “hands-on” training or focussed multidisciplinary networking for research development. To “spread the wealth” and provide opportunities for more new researchers,Reference Lai, Vetter and Richmond 11 the Pediatric Heart Network embedded a novel pre-career development scholar award supported by funds built into the data coordinating centre (New England Research Institutes) budget. To prepare promising clinician scientists, who have less dedicated time for research, for successful competition for the K award or alternative sources of extramural funding, the goals of this unique training programme were to provide an established network structure for the scholars/mentors to conduct their projects and execute their career development plan; facilitate cross-germination of ideas and collaborative multidisciplinary mentoring across all Pediatric Heart Network core centres; encourage engagement in Pediatric Heart Network activities; leverage the use of Pediatric Heart Network resources; teach regulatory documentation; and promote presentations and publications for academic advancement.
Pediatric Heart Network Scholar Award application
The application process for the Scholar Award mirrored the Pediatric Heart Network’s established procedures for evaluating and prioritising protocols. 10 – Reference Peterson, Winchester and Park 12 A request for applications – new pilot studies or ancillary studies related to existing or planned Pediatric Heart Network studies – was sent to the nine core centres. Innovative translational, clinical, or epidemiologic paediatric cardiovascular research was eligible for an award of up to $75,000 for 1 year of support. Basic science applications were not accepted. Junior faculty members who were <5 years from completion of their last fellowship at the time of funding, fellows, and doctoral/pre-doctoral nurses were eligible. Each core centre was responsible for the local process that would identify up to two candidates – total of 18 potential applicants – who would develop a two-page concept proposal for review and scoring by the three chairs of the Scholars Award Committee. The top-scoring applicants were then invited to submit expanded applications for scoring by the review panel that included representatives from each core centre: National Heart, Lung, and Blood Institute, the data coordinating centre, and the Pediatric Heart Network Protocol Chair. Applications were assigned primary and secondary reviewers and preliminary scores were obtained from all eligible panellists; members were recused from the discussion and scoring of applications from their own centres. Applicants were evaluated on relevance, scientific merit, public health significance, and feasibility within the Pediatric Heart Network environment using standard National Institutes of Health peer review scoring forms. Departmental and institutional support, as well as mentorship plans and mentoring team communication strategies, were required to be clearly described in the application. After discussion, an overall priority score was assigned and the top applicants received awards. All applicants (whether funded or not) received a summary paragraph of their review without actual numeric scores.
Conduct of the scholarship
Pediatric Heart Network Scholars have been funded for two award cycles. After receiving funding, the scholars and their mentors determined the conduct of their projects and the level of involvement of Pediatric Heart Network and non-Pediatric Heart Network centres. Each scholar gave two formal presentations at the Pediatric Heart Network steering committee meetings: their preliminary results at the end of the 1st year of funding and their final results after project completion. Progress reports were submitted annually. Although funding was awarded for 1 year of support, all scholars in both groups applied for and received a 1-year unfunded extension to complete their projects.
Study purpose
To inform the greater research community regarding the lessons learned and the benefits/challenges of a mentored award embedded in a multicentre clinical research network, we sought to determine patient satisfaction with the Pediatric Heart Network Scholar Award and to describe the academic products of the funded scholars.
Methods
Survey design
The study team developed survey questions designed to evaluate the application, selection, and feedback processes, as well as the overall challenges and benefits encountered in the Scholar Award programme. Survey questions were customised into two separate questionnaires aimed at two groups involved in the Pediatric Heart Network Scholar Award, as appropriate: all Scholar Award applicants, regardless of whether they received funding or not, and the project mentors of the funded scholars and the Pediatric Heart Network centre principal investigator – in some cases, this was the same individual. Study questions were constructed on a Likert scale, which is a three-point scale from “not helpful”, “somewhat helpful”, to “very helpful” or a five-point scale from “strongly agree” to “strongly disagree” – or posed as “Yes/No/Not applicable” (Supplemental materials A and B). All survey data were collected and stored at the University of Utah, and the study was deemed exempt by the University of Utah Institutional Review Board.
Web questionnaires were constructed using the REDCap electronic data capture toolsReference Harris, Taylor and Thielke 14 hosted at the University of Utah. In January, 2017, a pre-notification message was sent to all funded scholars, unfunded applicants, their mentors, and the centre principal investigators to inform them of the upcoming survey request before sending the official e-mail invitation request, which contained the survey link. Non-respondents were sent two additional reminder messages via REDCap over the following weeks. In March, 2017, a final e-mail request was sent directly to the principal investigators of centres with non-responding individuals, requesting their help in encouraging responses. All responses were anonymous and unattached to the survey response tracking system.
Academic achievements
To evaluate the academic success of funded scholars, we requested a detailed list of their academic achievements and scholarly product from the time they received the award until November, 2017 – 4–5 years after the scholarships were awarded. Specifically, they were asked to include abstracts and manuscripts that were published or in press; funded extramural grants and the amount of each award; and participation in Pediatric Heart Network writing groups, standing committees, and/or as centre principal investigator for a study.
Analysis
We performed descriptive analyses of the survey responses using Microsoft Excel and Stata MP Version 13. We calculated response distributions for all survey questions seeking to assess the application process, selection process, and benefits and challenges of the programme. To assess differences in ratings of the application and selection process between funded applicants (scholars), unfunded applicants, and mentors/principal investigators, we conducted χ2 tests of the association between respondent role, such as funded scholar, unfunded applicant, or mentor/principal investigator, and responses to three questions: rating the clarity of evaluation criteria, fairness of selection process, and overall rating of the process. Repetitive responses to open-ended questions regarding strengths and weaknesses of the programme and suggestions for improvement are reported as themes. Isolated comments are not reported.
Results
Survey results
All 13 funded scholars responded, which shows a 100% response rate, and 15/19 unfunded applicants completed the questionnaire (79%), resulting in a response rate of 88% for all applicants. All 20 of the mentors/centre principal investigators responded to the survey, which shows a 100% response rate.
Application and selection processes
We asked all funded scholars, unfunded applicants, and mentors/principal investigators to rate the scholarship application and selection process. Nearly all respondents (98%) reported that the criteria used to evaluate applications were either “somewhat” or “very clear” (Fig 1a). However, ratings of the evaluation criteria varied among funded scholars, unfunded applicants, and mentors/principal investigators (χ2=10.85, p=0.028). The mentors/principal investigators were most likely to say that the review criteria were “very clear” (74%), whereas 54% of funded scholars and only 20% of unfunded applicants rated the criteria as “very clear”. Although a few respondents rated the criteria as “not clear”, these individuals were all unfunded applicants. There was also significant variation across the three response groups (Fig 1b) in their ratings of the fairness of the selection process (χ2=16.97, p=0.002). The majority of funded scholars and mentors/principal investigators rated the selection process as “very fair” (85 and 89%, respectively), and no respondent in these groups rated it as “unfair”. In contrast, 14% of unfunded applicant respondents reported that the process was “unfair”. Comments made by the unfunded respondents indicated a lack of transparency and a need for better communication of the award scoring criteria and selection process at their centre.
Ratings of the clarity of the evaluation criteria ( a ), fairness of the selection process ( b ), and overall perception of the application/selection process ( c ) by the funded scholars, unfunded applicants, and funded scholars’ mentors/principal investigators. Applicants were asked to rate the application process, whereas mentors/principal investigators were asked to rate both the application and selection process.
All respondents were asked to provide an overall rating of the application/selection processes (Fig 1c) on a scale consisting of “poor”, “fair”, “good”, and “excellent”. Responses differed for funded scholars, unfunded applicants, and mentors/principal investigators (χ2=16.83, p=0.002). Only 7% of unfunded respondents rated the processes as “excellent” and 40% found it was merely “fair” compared with the funded scholars and the mentors/principal investigators, more of whom rated it as “excellent” (38 and 63%, respectively) and fewer as “fair” (0 and 11%, respectively). None of the respondents in any group rated the processes as “poor”.
Ratings of the feedback process by unfunded applicants
We assessed the helpfulness of feedback provided to the unfunded applicants (Table 1) and found that nearly half (47%) of the respondents in this group found the feedback was “not helpful” and only 13% found it “very helpful” overall. Only one respondent reported that the feedback they received was “very helpful” for moving their proposal forward, and a third of respondents found the feedback “not helpful” in this regard. A majority of unfunded applicants reported the feedback was “somewhat helpful” for refining research questions, developing writing skills, and learning to apply for a grant, whereas the remaining were nearly equally split between the most extreme responses of “not helpful” and “very helpful” (~20% each) for each of these areas. Overall, nearly half of the unfunded respondents did not find the feedback useful for connecting with potential collaborators or mentors (47% for both collaborators and mentors). Comments from the unfunded applicants centred on the paucity of information they received and the lack of detail to improve their proposal.
Unfunded applicants ratings of the helpfulness of reviewer feedback on proposals (n=15)Footnote *.
* Of the 19 unfunded applicants, 15 responded
Perceived benefits and challenges of the award by funded scholars
Most of the specific benefits of the Scholar Award were rated on a four- or five-point scale from “strongly agree” to “strongly disagree” (Table 2). More than half of the funded scholars (54%) found the award was not useful for finding new mentors. There were no other negative ratings, such as “strongly disagree” or “somewhat disagree”, for any category. The majority of Pediatric Heart Network scholars (77%) found the award to be very beneficial overall. The group was nearly equally split between “somewhat” and “strongly” agree that the award improved research skills, and established their career trajectory. Funded scholars “strongly” agreed that the award prepared them to be a principal investigator and increased their publication potential. The highest-rated benefit of the award was in expanding the scholar’s collaborative network outside his/her centre and expanding their involvement in the Pediatric Heart Network. All 13 scholars indicated that the award programme met their expectations.
Funded scholars ratings of the benefits of the Pediatric Heart Network Scholar AwardFootnote * (n=13).
PI=principal investigator
* Full range of responses to agree/disagree items included: strongly disagree, somewhat disagree, neither agree nor disagree, somewhat agree, strongly agree; response fields not selected by any respondent are excluded from this table
Over 90% of funded scholars thought the Pediatric Heart Network Scholar Committee members should identify and address barriers for the scholars and facilitate multicentre enrolment. About half (54%) of the scholars found that obtaining data use agreements and delays in obtaining data from other participating centres were barriers, yet only one scholar found obtaining Institutional Review Board approval to be a barrier (Table 3). When asked whether more funding would improve their ability to conduct the study, 54% of scholars responded “not at all” and 15% responded “greatly improve”. Nearly all scholars (92.3%) agreed that the ideal award period was 2 years. Fewer than half of the scholars thought the Pediatric Heart Network Scholar Committee members should be involved in study implementation. Although 54% of scholars favoured the Pediatric Heart Network Scholar Award Committee members liaising with mentors/principal investigators, a higher proportion (69%) indicated that the committee members should serve as liaisons between the scholars and others. The funded scholars’ recommendations were as follows: provide clear guidelines for the roles of the mentors, develop criteria that require the mentor be more active in the planning and implementation phases, identify mentors across centres who can guide junior investigators in their roles both as principal investigator and as co-investigators, and offer mentorship programmes for all core centres, regardless of whether the centre has funded scholars or not.
Funded scholars ratings of the role of the Pediatric Heart Network Scholar Award Committee (n=13).
Perceived benefits and challenges of the award by mentors/principal investigators
The mentors/principal investigators favourably evaluated the qualifications of the review committee, with 17/19 finding them “very qualified” and no respondent finding them “unqualified”. Respondents suggested including ad hoc reviewers where a particular expertise was needed. All mentors/principal investigators rated the usefulness of including more senior-level faculty on the review committee as only “somewhat helpful”. The seven questions aimed at assessing the benefits of the award allowed one of three graded responses, and no mentor/principal investigator respondent scored the award in the lowest grade for any of these categories (Table 4). All respondents found the programme “very” valuable for the scholars, and nearly 75% found it beneficial for the Pediatric Heart Network. Responses were nearly equal between “somewhat” and “very” beneficial for the mentors, principal investigators, and the centre as a whole. The goals of the award were perceived as being clear for 14 mentors/principal investigators, but only 11 felt the programme “completely” achieved its goals. The mentors/principal investigators suggested requiring clear, well-written mentoring plans (modelled after T32 or K awards) and leveraging the Pediatric Heart Network expertise to allow scholars to work with multiple experts in diverse fields. For the question of whether one should change the Scholar Award to increase funding amounts by funding fewer scholars, 85% of mentors/principal investigators voted “no”. Most mentors/principal investigators (95%) would recommend the programme to early investigators (one was “unsure”).
Mentors/principal investigators ratings of the benefits of the Pediatric Heart Network Scholar AwardFootnote * (n=20Footnote **).
* Full range of responses included: not at all, somewhat, very much; response fields not selected by any respondent are excluded from this table
** Two respondents did not answer all items
Scholarly product for the funded scholars
The Pediatric Heart Network Scholar Award programme has completed two complete funding cycles (commencing in 2013 and 2014). The funded scholars’ research covered a wide variety of areas of CHD: predictive modelling (1), costs (1), and outcomes (2) in the single-ventricle population; development of apps to improve remote care of infants (1) and transition of care for adults (1); investigation of fetal brain development (2); exploration of academic outcomes (1); improving prenatal diagnosis (1); impact of early nutrition on glucose control after cardiac surgery (1); inappropriate shocks in pacemaker patients (1); and percutaneous valve implantation (1). At the data collection deadline (1 November, 2017), all awardees successfully completed their projects, and all currently have academic appointments. Of the 13 funded scholars, two (one in each funding year) had a PhD; the remaining 11 were MDs. For the 2013 cohort, 5/6 scholars had advanced degrees in research compared with 2/6 from the 2014 cohort. In addition to differences in the proportion with advanced degrees, the two groups of scholars differed in Pediatric Heart Network participation and academic productivity (Table 5).
Academic benchmarks and Pediatric Heart Network activity reported by funded Pediatric Heart Network Scholars (as of 11/1/2017).
* First or senior author articles that are published or in press
** Includes writing committees, standing committees, and centre principal investigator on a project subcommittee
*** PhD
All 13 scholars participated in at least two Pediatric Heart Network steering committee meetings where they networked with their peers and senior investigators. In addition, nine scholars attended at least one Pediatric Heart Network Career Day with expert didactic workshops and discussions of networking, multicentre study design and conduct, mentoring, grant writing, and career development. Eight scholars were members of 18 Pediatric Heart Network manuscript writing committees, two served on standing Pediatric Heart Network committees, and three were the local centre principal investigators for four Pediatric Heart Network studies. Seven scholars leveraged Pediatric Heart Network expertise for obtaining co-investigators, collaborators, and mentors for four multicentre Pediatric Heart Network scholar projects and seven applications for career development awards. Overall, the 13 scholars were first or senior authors for 97 abstracts and 109 manuscripts. They were awarded funding for 34 grants from a wide variety of sources (Table 6) for a total of $9,673,660. Although many factors are ultimately involved to achieve successful funding, the return on investment was calculated by dividing the total amount obtained in subsequent awards by the total spent for all 13 funded scholars [($9,673,660÷(13×$75,000)] or $9.92 for each Pediatric Heart Network scholar dollar spent. Currently, an additional nine grant applications are under review: six from National Institutes of Health – four R01 and two K awards; two from the Department of Defense; and one from Woodruff Health Sciences Center.
Extramural funding status reported by funded Pediatric Heart Network Scholars (as of 11/1/2017).
Government includes National Institutes of Health and Canadian Institutes of Health Research; Foundation includes the American Heart Association, Thrasher Research Fund, Saving Tiny Hearts Society; Pharmaceutical includes Roche, Actelion, Janssen, and Genzyme; Institutional includes local funding agencies.
Discussion
To train junior investigators, National Institutes of Health Networks have historically used Research Skills Development Core funds provided to selected centres for their candidates who then receive intensive local training, education, and mentoring in focused areas. 10 – Reference Schulman, Suncion and Karantalis 13 The Pediatric Heart Network Scholar programme took a novel, more inclusive, team science approach and embedded a scholarship programme into the framework of a successful multicentre research network. Although there are many promising investigators, this programme was considered a benefit of successful competition as a network core centre and was limited to those sites. Novice investigators were selected by peer review and funded to foster research career development through completion of their proposals and through multicentre networking, collaboration, and mentoring. Although there were no changes made in the selection process for the 2013 and the 2014 scholars, the work product difference between these groups probably cannot be explained by the extra year. Therefore, we used feedback from the first two funding cycles to identify important strengths and challenges of the programme and instituted strategies to improve selection of the strongest applicants.
Lessons learned and changes made to the Pediatric Heart Network Scholar Award programme
Clear strengths of the programme varied among the groups surveyed, but, in general, included developing research skills, networking, and increasing the scholars’ publication potential. Although overall ratings of the processes, benefits, and goals of the programme were generally high, feedback to the applicant was identified as a deficiency, particularly for unfunded applicants. In response, future award cycles have been expanded so that feedback includes not only the written reviews by the primary and secondary reviewers but also a detailed summary of the review committee’s discussion. A Scientific Review Officer who is familiar with the policies, goals, and conduct of the Pediatric Heart Network was recruited from outside the Pediatric Heart Network to ensure that the strengths and weakness of the proposal, candidate, and mentoring environment/team were included in this discussion summary. The scholars with the highest-scored projects in the upcoming grant review will be approached for permission to provide their applications as examples for future candidates.
Compared with the mentors/principal investigators, the funded scholars found the application and review process less clear and indicated that instructions and evaluation criteria needed to be more transparent to ensure fairness of the selection process. Although this difference in clarity may reflect the scholars’ inexperience rather than a failure of the process itself, the Pediatric Heart Network Scholar Committee members responded by adapting selection criteria for the new award cycle based on those specified for the National Institutes of Health K awards. 15 The pre-review and scoring of the preliminary concept proposal by the Scholar Committee co-chairs was abandoned in favour of permitting each centre to select and submit their two best candidates and projects. The selected candidates were asked to provide a Letter of Intent describing the project, mentoring team, participating centres, and any special review expertise needed to allow the recruitment of ad hoc members, if needed.
In preparation for the new funding cycle, the centre principal investigators nominated representatives for the Pediatric Heart Network Scholar Committee to act as champions of the award at their centre. Requirements for the position included a clear interest in serving, guaranteed time commitment to fulfil the duties of the committee, and agreement to serve as the point person for communication of all requirements and aspects of the award at their centre.Reference Tsen, Borus and Nadelson 16 The nine core centre representatives included three assistant professors, three associate professors, and three professors who also served as the selection panel for the award. The equal distribution in rank allowed input on the process and review across the spectrum of junior to senior investigators. To ensure consistency, the committee members participated in an in-person mock review of a previously submitted Pediatric Heart Network application. The independent review chair was recruited for her extensive experience in the review of mentored awards. As part of the upcoming review process, all members with a conflict will be recused. Scores will be compiled by the Scientific Review Officer and Chair and funding decisions made by the Pediatric Heart Network leadership. As the vast majority of respondents recommended, the funding period was extended to 2 years.
Although nearly all funded scholars strongly agreed that the award expanded networking outside their centre, many did not find new mentors either at their centre or at another centre, and the ratings indicated a need for improvement in expanding the scholar’s involvement in the Pediatric Heart Network and in multicentre research. Studies have shown that mentored faculty are more successful and more satisfied with their careers, often citing it as the first or second most important factor in career development. 15 – Reference Cen, Sandborg and Hudgins 18 The complexity of current research typically requires a multidisciplinary team approach with a variety of mentors needed outside of the scholar’s centre.Reference Robinson, Schwartz and Dimeglio 19 , Reference Kaltman, Burns and Pearson 20 Therefore, in the new award announcement for 2018 funding, we adopted specific criteria from both the Pediatric Heart Network aims for multicentre researchReference Burns, Pemberton and Pearson 9 and the K award specifications 15 for detailed plans for the mentoring principal investigator and multicentre, multidisciplinary mentoring, and protocol teams.
The funded scholars and mentors/principal investigators recommended that the Scholar Award Committee members facilitate multicentre enrolment. A statement was added to the award announcement encouraging collaborative proposals involving multiple Pediatric Heart Network centres. To address the perceived need for more involvement of the Scholar Committee members in identifying and addressing barriers for the scholars and liaising with others, the first Pediatric Heart Network steering committee presentation for funded scholars was redesigned to allow the scholars, mentors, and Network Center PI to garner support, receive feedback from the larger group of steering committee members, and facilitate the development of collaboration with experts and mentors outside their own centre before launching their study. Biannual team meetings with the centre’s Pediatric Heart Network Scholar Award Committee member, funded scholar, mentoring principal investigator, Pediatric Heart Network centre principal investigator, and the Scholar Committee chairs are planned to identify and remove barriers that delay completion of the project and monitor the scholar’s progress in accomplishing their career development plan.
Limitations
Only two grant cycles have been completed thus far; hence, the sample size is small and follow-up times are relatively short, precluding inferences regarding long-term success of the funded scholars. As the survey was distributed after funding decisions were made, we could not evaluate the effect funding may have had on responses. We did not have sufficient data to compare the success of the Pediatric Heart Network scholars with junior investigators supported by Research Skills Development Cores or other early career awards because of the limited sample size, but this is a goal for the future. The time interval between the start of the first cycle and the distribution of the survey, which was close to 4 years, may have affected the recall of the finer details of the process for some of the individuals surveyed, and probably the response rate for unfunded applicants. We did not, as part of the survey announcement, send out any supplemental material as reminders of the original process, but instead chose to rely upon the recall of each individual. Work products, such as abstracts, manuscripts, grant funding, and so on, were not collected for unfunded applicants, so we cannot say whether these individuals had more, the same, or less success than their funded scholar peers. Assessing this in future cycles may allow a more in-depth understanding of the role the Pediatric Heart Network Scholar Award plays in securing extramural funding or propelling a research career for both funded and unfunded applicants.
Conclusion
The Pediatric Heart Network scholars met or exceeded many markers of academic success, and the return on investment, in terms of grant dollars, publications, and workforce trained and committed to paediatric cardiovascular research, was rewarding. Despite this, the survey of patient satisfaction with the Scholar Award identified areas for improvement and strategies were developed to meet these challenges for future funding cycles. Our programme of embedding a mentored award within a successful network may be used as a model for other multicentre networks.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1047951118000483
Acknowledgement
None.
Financial Support
The study was supported by grants – HL 109673, HL109741, HL109743, HL109778, HL109781, HL109816, HL109818, HL109777, HL068270, HL109737 – from the National Heart, Lung, and Blood Institute, National Institutes of Health. The contents of this work are solely the responsibility of the authors and do not necessarily represent the official views of the National Heart, Lung, and Blood Institute.
Conflicts of Interest
None.
