Introduction
The Health Technology Assessment (HTA) Consumer Consultative Committee (CCC) was established in 2017. It is a committee of health consumer representatives sitting within the Principal HTA Committees and associated subcommittees operating on behalf of the Australian Government and Department of Health. There are currently consumer representatives serving on the Pharmaceutical Benefits Advisory Committee (PBAC), its Economic Sub-committee (ESC), Drug Utilization Sub-committee (DUSC), the Medical Services Advisory Committee (MSAC) and its Evaluation Sub-committee, and Population, Intervention, Comparator, Outcomes Advisory Sub-committee (PASC). There is also the Prosthesis List Advisory Committee (PLAC). Consumer representatives’ role is to bring patient groups’ perspectives and experiences of new medicines, tests, or devices into HTA processes (1).
The terms “patient” and “patient representative” are employed in other countries, whereas Australia uses the term “healthcare consumer” which also covers carers and the broader community (2). As with UK's National Institute for Health and Care Excellence (NICE) and Canada's Canadian Agency for Drugs and Technology in Health (CADTH) (Reference Berglas, Vatour and Bell3;Reference Norburn and Thomas4), Australian HTA committees have consumer representatives as full members, contributing to reviewing applications for public funding of medicines, medical services, devices, prostheses, and tests. Consumer representatives may also seek to consult with patient groups and organizations for their input. Consumer representatives are appointed through formal nomination processes and come from diverse experiences and are not necessarily familiar with HTA. They have experience serving on other health-related committees. Consumer representatives upskill themselves on HTA matters. This is the case with the authors of this article. The current approach does not lead to consistent capability on HTA committees.
The CCC is a principal committee of the office of HTA and sits alongside PBAC, MSAC, and PLAC in the overall structure. HTA consumer representatives are members of the CCC. The CCC Chair works with other chairs regarding any items of interest and specific relevance to consumer matters. Key functions of the CCC include informing HTA committees and the department on consumer and patient matters in HTA, bringing consumer and community evidence and perspectives into HTA processes and assisting consumer representatives to participate in committee work (1).
HTA committee work can be perceived as daunting due to the volume of work and its technical nature (Reference Weeks, Polisena, Scott, Holtarf, Stanisewska and Facey5). Both CADTH and NICE have concentrated on improving the quality of patient and patient group input into submissions. CADTH provides information and training sessions by a staff member, whereas NICE provides resources and support (Reference Weeks, Polisena, Scott, Holtarf, Stanisewska and Facey5). Both CADTH and NICE provide templated submission forms.
In 2018, the HTA CCC identified the need to provide training to future consumer representatives joining any of the above HTA committees to assist to recruit, train, and therefore retain consumer representatives. Following discussion, it was decided the HTA CCC should explore mentoring. Mentoring, either informal or formal, is based on adult learning, which is self-directed, experience-based, and self-developmental for immediate specific purposes (Reference Chinnasamy6). Mentoring offers a means for experienced consumer representatives to share their knowledge and experiences within the context of complex work undertaken by specific HTA committees. Having an overarching framework based on principles of adult learning allows for formal evaluation. Thus, the aim of this project changed to explore the potential role of mentoring for HTA consumer representatives.
Methods
The mentoring project team, all members of the HTA CCC (CW, JD, and JW), considered a literature review would assist the CCC to develop a mentoring program. Due to resource constraints, work began as a “rapid review”. Rapid reviews take anywhere between 1 and 6 months, with a limited literature search and confine results and their analysis to qualitative synthesis (7). Such reviews still employ formal research steps: defining the research question, defining inclusion criteria, identifying biases, conducting the search in a manner that excludes as many biases as possible, and appraising the quality of documents (Reference Tricco, Antony, Wasifa, Strifler, Ghassemi and Ivory8). The project team participated in all the steps as well as consulting with CCC members.
The project team developed the the list of search terms given in Table 1.
Table 1.Search terms mentoring
PubMed, CINAHL, and PsycINFO and a search of “grey literature” through Google and Google Scholar were undertaken in late 2018 and into 2019. In this subject area, “gray literature” search was productive since many organizations undertake mentoring programs but do not publish their results in academic journals. The combined searches produced fifty-seven articles in English. Of these, the project team selected thirty-five articles most relevant to the review's aim appropriate to mentoring Australian consumer representatives working in the HTA assessment areas. The articles selected are largely from UK, US, Canadian, and Australian sources.
Identified articles were listed and discussed in terms of reporting of evidence and relevance to the project. A limitation of our review was identified early on. Many of the articles provided scant evidence to support the outcomes they claimed. Data were lacking to assess bias or appraise quality of mentoring. Thus, the “rapid review” became exploratory and pragmatic, seeking out relevant information on mentoring with a view to developing and evaluating a pilot program.
The results were discussed again and structured into themes. Finally, articles were placed within a framework to inform the contribution of mentoring in HTA (Reference Norburn and Thomas4;Reference Abelson, Wagner, DeJean, Boesveld, Gauvin and Bean9;Reference Kaltenthaler, Cooper, Pandor, Martyn-St James, Chatters and Wong10). Following deliberations on limitations around evidence, the research question became: What models of mentoring and their attributes will be applicable to a program for consumer representatives in the HTA area?
As the literature was heterogeneous in its origins and in presentations of data, the narrative synthesis approach meant that those articles that best described the parameters of mentoring applicable to developing a mentoring program in the health technology context were included. In this article, to demonstrate the overall themes relevant to HTA committees, we mainly focus on the examples of mentoring programs undertaken by academic or other educational institutions.
Results
Results appear below under the terms the project team identified from the literature as features of a mentoring program framework. Results will assist consumer representatives in the Australian Government Department of Health HTA committees to decide on the components of mentoring that assist them to perform effectively either as mentors or as mentees. Following discussion with CCC members, the framework should be subject to ongoing evaluation and improvement.
Definitions of Mentoring
Few articles defined mentoring. The most relevant to the collegiate committee structure of the CCC emphasized mutuality and a preparedness to help others, a quality already present in the CCC that can be captured in a mentoring program. Manchester Metropolitan University, Human Resources defines mentoring as a process in which an experienced individual helps another person develop his or her goals and skills (11) through a series of time-limited, confidential, one-on-one conversations, and other learning activities. The Centre for Health Leadership and Practice (12) defines mentoring somewhat differently, as helping people to develop more effectively. Here, it is a relationship designed to build confidence and support the mentee to take control of their own development and work.
Primary Health Care Research Information Service (PHCRIS) “Mentoring Matters” (13) considers mentoring involves multiple roles depending on the needs of the mentees and defines it by its component parts:
(1) a mutual relationship where participants share experiences, knowledge, and information;
(2) a developmental process where the mentee grows in skills, knowledge, and confidence;
(3) a strategy to share intellectual and other resources;
(4) informal or formal support by a more experienced and skilled person;
(5) guided learning by the mentor; and
(6) traditionally a one-to-one relationship but group mentoring may be a beneficial option.
Mentoring Attributes or Qualities
These definitions emphasize positive relationships. The qualities of mentors to progress these relationships are important.
Mentoring Guidelines from Manchester Metropolitan University, Human Resources (11) and PHCRIS (13) suggest that a mentor should display some of the following attributes:
(1) Listening
(2) Asking questions to help develop mentor's and mentee's understanding of a situation or problem
(3) Providing information and knowledge and share informal networks
(4) Providing advice on career development
(5) Offering different perspectives
(6) Providing support and encouragement
(7) Offering guidance and advice in regard to qualifications
(8) Being a sounding board
(9) Being a critical friend
(10) Encouraging self-reflection
(11) Helping mentees identify areas for development
(12) Being available, open, respectful, accepting, willing to share and learn
(13) Answering questions, gives guidance and constructive feedback
(14) Motivating by setting an example
(15) Promoting the development of contacts and networks
(16) Advocating for the mentee.
These definitions and qualities emphasize that mentors will use their experience and knowledge in a facilitative manner to support the development of the mentee. However, the mentee is responsible for applying his/her learning and putting plans into action. With regard to HTA consumer representatives, almost of all of these qualities except perhaps those concerned with career development and qualifications will be of value.
Benefits of Training in Mentoring
While the above qualities are foundational, some programs recommend mentor training to ensure effective and consistent mentorship. Lindsey et al. (Reference Lindsay, Hartman and Fellin14) discussing mentoring youth argue that a well-trained, confident, and supported mentor is more likely to continue mentoring.
Similarly, the Center for Public Service and Engaged Scholarship at Harvard University (15) offers a 4-hour training program for all people mentoring at the university. Training is certificated and mandatory for all mentoring volunteers. The Victorian Institute of Teaching (VIT) (16) offers a two-day training program for experienced teachers from all Victorian schools and early childhood centers wishing to mentor teachers new to the profession to improve their skills.
Resources
An additional approach is to provide a mentoring handbook. The Postgraduate Medical Council of Victoria (PMCV) (17) provides handbooks to both mentors and mentees, so that both members of the relationship are aware of their roles, responsibilities, and expectations. The handbook provides templates of documents such as roles and responsibilities of the mentor which includes certification, a mentor preference form, where mentees can also specify with whom they prefer not to be paired, an activity log, a goal-setting template, and a mentor/mentee agreement form.
Both training of mentors and a handbook are being considered by the HTA CCC; a handbook may reduce administrative work, while training promotes consistency among mentors.
Other Considerations in Mentoring Programs
Articles were chosen from diverse workplaces such as academia, schools, professional organizations, and commercial workplaces. They demonstrated that mentors can be used in various functions such as induction, changing roles, and continuous professional development. As mentoring takes place within a defined situation such as the workplace, an institution, and among colleagues with similar professional qualifications, it assumes that at some stage, both mentor and mentee will be on some level of equality. It is an approach that is consensual, flexible, undertaken in work hours, relates to the job, is generally individual, people-centered, and provides a feedback system to enhance and embed learning.
Pairing of mentor/mentee may be done through an administrative process, which is based on arbitrary criteria, where mentor and mentee choose from one another based on sets of attributes, formal programs tend to adopt an administrative process while informal programs rely on personal choices. There is some evidence that where mentors/mentees have some choice, better outcomes result (Reference Meagher, Taylor, Probsfield and Fleming18).
Evaluations of Mentoring Training Programs
Evaluating Outcomes and Validated Tools for Measurement
Murdoch University evaluated a pilot program of mentoring new maths and science teachers at Western Australian schools (Reference McConney and Maor19). The evaluation found that mentees required a reduction in their workload of 20 percent to engage in mentorship; the program was best started at the beginning of the school year; both mentors and mentees required professional development to develop effective relationships (whereas most of the literature suggests only providing training to mentors); co-location in the same school saved time and was convenient. Additionally, a careful selection of mentors, perhaps based on their passions for teaching science and maths enhanced the value to mentees, as did building in regular milestone meetings throughout the year which improved accountability while also contributing to systematic evaluations to improve programs.
Most helpful to evaluating a mentoring program are validated tools to measure the benefit of mentoring programs. One is Mentoring Competency Assessment (MCA) (Reference Fleming, House, Shewakramani Hanson, Yu, Garbutt and McGee20). Fleming et al. (Reference Fleming, House, Shewakramani Hanson, Yu, Garbutt and McGee20) described evaluating MCA to assess skills of research mentors in clinical and translational research. The aim was to have a tool that could measure the value of various mentor characteristics and skills in mentoring new researchers. MCA was trialed among 220 pairs of mentors and mentees at 16 US universities. They combined the Mentorship Effectiveness Scale (MES) with two other tools used to evaluate mentorship. The result was a twenty-six-item scale across six areas (maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, promoting professional development, and fostering independence). The University of Wisconsin Institute for Clinical and Translational Research provides a full list of these resources for all mentees and mentors to self-evaluate as well as evaluate the entire program (21).
The MES is used in medical and nursing schools in the US (Reference Berk, Berg, Mortimer, Walton-Moss and Yeo22). It is a twelve-item, six-point scale to evaluate behavioral characteristics of mentors. MES is accompanied by the Mentorship Profile Questionnaire which describes the mentoring relationship and then specifies the outcome measures produced from the relationship. Supporting documents demonstrate what the relationship has produced, such as conference papers, published articles, and other projects.
Another aspect of evaluation covers the benefits of investing in mentoring. A longitudinal study of mentoring doctoral students showed that mentoring did not lead to greater research output (Reference Paglis, Green and Bauer23). Mentoring in school teachers was found to be a cost-benefit. Villar and Strong (Reference Villar and Strong24) asked if mentoring was worth the money in regard to teacher retention. Villar and Strong (Reference Villar and Strong24) found: “increases in teacher effectiveness yielded greater savings than the reduction in costs associated with teacher attrition. Overall, the benefit–cost analysis showed that, after five years, an investment of $1 produces a positive return to society, the school district, the teachers, and the students, and the state almost recovers its expenses.”
Be A Mentor, a US-based foundation working with vulnerable children and young people to improve their lives conducted a cost–benefit analysis to demonstrate the long-term benefits of trained volunteer mentors (25). They found mentoring was “an outstanding success”: that, for example, “98% of all youth mentored complete their high school education. That alone raises their employability, income and subsequently their ability to support and strengthen the economy, rather than become a drain on it. And the cost is only $1,500 per mentored youth per year. As noted in the chart, it costs twenty times more for a youth to receive drug/alcohol treatment in a single year and over 130 times more for a youth in the juvenile justice system than it does to mentor.”
Best Practice in Mentoring
“Best practice” is operationally defined in an academic setting as those actions that produce the most desirable faculty outcomes, based on evidence and real-life experiences (Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26).
A formal mentoring program is best practice when it has appropriately matched pairs, there is a clear purpose for the mentoring relationship with stated goals, the relationship is developed, the mentor advocates on behalf of the mentee and is able to integrate the mentee into the organizational culture and assist them access resources, such as administrative support and workplace documents (Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26). The evaluation of the process is also essential for best practice (Reference Meagher, Taylor, Probsfield and Fleming18;Reference McConney and Maor19).
The Center for Health Leadership and Practice (12) speaks about best practices in mentoring to emphasize that it is in the practice or exercise of mentoring that effectiveness will be achieved. A mentor does not have all the answers and does not give advice. Mentors assist the mentee to think more broadly and deeply; share experiences and lessons learned; assist the mentee to see alternative interpretations; provide supportive feedback and encouragement; and help the mentee to apply their own successful strategies from their past work to the new challenges.
Similarly, Sherk (Reference Sherk27) of the EMT Group of Companies (USA) consulted both administrators and mentors of mentor programs in community groups to identify best practices. Best practice in community groups develops networking to ensure quality improvement and recruiting mentors to sustain the program. The supervision of mentors ensures that mentor/mentee relationships prosper (Reference Sherk27). Sherk's work also identified pitfalls of mentoring: inadequate personnel and/or resources to run the mentoring program; lack of commitment by a program manager; lack of integration of mission and goals, or poor integration of the program within the community; superficial support of mentors; and finally, a program that offers little that is unique. Sherk suggests starting out small, with passionate leadership, learning from mistakes, and slowly building up a program. Good resourcing, especially with regard to personnel, is important.
Applying standards for best practice can assist effectiveness (28). Elements of Effective Practice for Mentoring bases effectiveness on a set of standards the organization has researched and practitioners of mentoring have approved in mentoring youth (28). Standard One is recruitment based on realistic aims, objectives, and outcomes. Standard Two concerns screening of mentors to assess their suitability as mentors and have time to commit to mentoring. Standard Three argues for training of both mentors and mentees to ensure a culturally safe mentoring relationship. Matching mentors with mentees to produce sustainable relationships is Standard Four. Monitoring and support by the parent organization should be provided throughout and constitutes Standard Five. Finally, Standard Six relates to a fulfilling closure to mentoring.
Limitations
We acknowledge the limitations of “rapid reviews”; in this case, a level of bias in terms of selectivity as the project team explored articles that would inform the design of a mentoring program for consumers in the health technology areas (Reference Tricco, Antony, Wasifa, Strifler, Ghassemi and Ivory8). Much of the literature cited dealt with only one aspect of a mentoring program and related successful outcomes.
The literature was generally positive in its assessment of mentoring, with no reporting of any adverse events, though some programs provided opportunities for both mentors and mentees to do so, suggesting that such events occur.
The greatest limitation of this review relates to the lack of published peer-review evaluations, making it difficult to assess the robustness of any data. Acknowledging the lack provides the opportunity to address it through data-driven evaluations in future mentoring work in Australia.
Discussion
Applicability of Findings
This review was conducted to assist consumer representatives in the Australian Government Department of Health HTA committees decide on components of mentoring applicable to assist them to perform effectively, either as mentors or mentees. While the literature search revealed few articles that had an empirical evidence-base, it demonstrated that mentoring as a form of adult learning is widely applied. There was evidence that some mentoring programs had improved the quality of a program by training mentors or providing more resources, but little data to support these claims. Largely the project team “gleaned” the essential components to include in a mentoring program and concluded that evaluation should also be undertaken to demonstrate effectiveness.
Given that current HTA consumer representatives are established in their roles, mentoring new consumer representatives in HTA processes offers a more flexible approach than training. Training is largely didactic and would require multiple curricula to work across committee structures, where needs and roles differ. As mentoring is built on mutual respect, sharing, and listening, it should also serve to attract and sustain future consumer representatives in the HTA space. Mentees are also potential future mentors.
Applications to Assist Consumer Representatives Contribute Effectively to HTA Committees
The review serves to provide knowledge and understanding to construct a model suitable to the Australian HTA context. Currently, project consultants are using the review to develop a pilot model, the design of which needs to consider that mentoring will take place in participants’ own time before or after formal meetings. Unlike mentoring that takes place within employment, Australian CCC members reside in different time zones, across the continent. Online resources may help address potential issues related to undertaking mentoring in one's own time (17;Reference Meagher, Taylor, Probsfield and Fleming18;Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26). A handbook and resources could be tailored for CCC members as an adjunct to an online learning platform where people join one another online at convenient times. A handbook has the benefit of contributing to consistency across the mentoring process as does an online program; it should provide essential skills and direction on the HTA procedures to mentors and mentees in Australia.
Another possibility is that a group program could be developed, trialled, and made available online so that mentors and mentees can meet up at times convenient to themselves. Online handbooks and other resources would also be made available.
Other considerations are mentor burnout and difficulties in pairing. Over-burdening mentors will be monitored and evaluated in the initial pilot program.
Evaluation
Evaluating the effectiveness of mentoring may benefit from basic organizational structures and a set of evaluation tools. The organizational structure for a mentoring program and its evaluation could include a project officer, budget, resources for both mentors and mentees, and time allocation, providing essential skills and direction to participate in HTA procedures in Australia. Best practice in the literature suggests mentoring should include clearly stated goals (usually set by the mentee), that there be a timeframe established which includes preparation, enabling, and closure. Mentors/mentees recording these steps should show whether goals were achieved or if goals have changed. The literature review provided access to a number of validated tools that may prove extremely useful in evaluating the effectiveness of mentoring to consumer representatives in their respective committees. As best practice (Reference Meagher, Taylor, Probsfield and Fleming18;Reference McConney and Maor19), evaluation has an additional advantage of contributing to accreditation of a mentoring program.
Challenges
In exploring mentoring and its applicability to the HTA area, the project team found useful indicators on what constitutes best practice as well as measuring effectiveness, but little evidence of their widespread use. Additionally, both peer-reviewed and “gray” literature, provided little detail of any adverse events arising from mentoring programs. One reason given was that both mentors and mentees do not like reporting negative experiences in their relationships (17). The development of resources and handbooks to mitigate against adverse events suggests it is a problem requiring some attention (17).
A related issue for the HTA CCC is the need to recruit more consumer representatives to HTA committees as current ones retire. A mentoring program could assist recruitment by providing introductory resources to applicants considering applying, detailing HTA work, and available help, including mentoring through the CCC.
Conclusion
This review provides descriptions of a range of mentoring programs that informs development of a pilot mentoring program for consumer representatives engaged in committees in the complex areas of Australian HTA. The aim is to sustain a consistent level of capabilities in health consumer representation on HTA committees.
Mentoring was chosen as the most appropriate vehicle to create a consistently supportive program as those who would be either mentors or mentees are usually experienced consumer representatives already participating in other committees in the health system.
The review explored definitions, effectiveness of mentoring, best practice, evaluation including measures, program design, and both group mentorship and one-on-one mentorship.
While training programs are used to upskill patient input in some health technology assessments such as in Canada and the UK, there were no examples of mentoring in HTA in the identified literature. However, useful examples emerged from academic programs where mentoring is used to retain young researchers, first-year teachers, and assist newly qualified clinicians and nursing staff.
This review provides background material for the development of a pilot program specifically for new CCC members which includes evaluating the program in terms of its effectiveness of consumer representatives' contributions to HTA committees. Currently, there is an ongoing pilot with one mentor and one mentee. As consumer representatives complete their terms, they can offer to mentor their replacements. Greater choice for mentees may evolve over time.
You have
Access
Introduction
The Health Technology Assessment (HTA) Consumer Consultative Committee (CCC) was established in 2017. It is a committee of health consumer representatives sitting within the Principal HTA Committees and associated subcommittees operating on behalf of the Australian Government and Department of Health. There are currently consumer representatives serving on the Pharmaceutical Benefits Advisory Committee (PBAC), its Economic Sub-committee (ESC), Drug Utilization Sub-committee (DUSC), the Medical Services Advisory Committee (MSAC) and its Evaluation Sub-committee, and Population, Intervention, Comparator, Outcomes Advisory Sub-committee (PASC). There is also the Prosthesis List Advisory Committee (PLAC). Consumer representatives’ role is to bring patient groups’ perspectives and experiences of new medicines, tests, or devices into HTA processes (1).
The terms “patient” and “patient representative” are employed in other countries, whereas Australia uses the term “healthcare consumer” which also covers carers and the broader community (2). As with UK's National Institute for Health and Care Excellence (NICE) and Canada's Canadian Agency for Drugs and Technology in Health (CADTH) (Reference Berglas, Vatour and Bell3;Reference Norburn and Thomas4), Australian HTA committees have consumer representatives as full members, contributing to reviewing applications for public funding of medicines, medical services, devices, prostheses, and tests. Consumer representatives may also seek to consult with patient groups and organizations for their input. Consumer representatives are appointed through formal nomination processes and come from diverse experiences and are not necessarily familiar with HTA. They have experience serving on other health-related committees. Consumer representatives upskill themselves on HTA matters. This is the case with the authors of this article. The current approach does not lead to consistent capability on HTA committees.
The CCC is a principal committee of the office of HTA and sits alongside PBAC, MSAC, and PLAC in the overall structure. HTA consumer representatives are members of the CCC. The CCC Chair works with other chairs regarding any items of interest and specific relevance to consumer matters. Key functions of the CCC include informing HTA committees and the department on consumer and patient matters in HTA, bringing consumer and community evidence and perspectives into HTA processes and assisting consumer representatives to participate in committee work (1).
HTA committee work can be perceived as daunting due to the volume of work and its technical nature (Reference Weeks, Polisena, Scott, Holtarf, Stanisewska and Facey5). Both CADTH and NICE have concentrated on improving the quality of patient and patient group input into submissions. CADTH provides information and training sessions by a staff member, whereas NICE provides resources and support (Reference Weeks, Polisena, Scott, Holtarf, Stanisewska and Facey5). Both CADTH and NICE provide templated submission forms.
In 2018, the HTA CCC identified the need to provide training to future consumer representatives joining any of the above HTA committees to assist to recruit, train, and therefore retain consumer representatives. Following discussion, it was decided the HTA CCC should explore mentoring. Mentoring, either informal or formal, is based on adult learning, which is self-directed, experience-based, and self-developmental for immediate specific purposes (Reference Chinnasamy6). Mentoring offers a means for experienced consumer representatives to share their knowledge and experiences within the context of complex work undertaken by specific HTA committees. Having an overarching framework based on principles of adult learning allows for formal evaluation. Thus, the aim of this project changed to explore the potential role of mentoring for HTA consumer representatives.
Methods
The mentoring project team, all members of the HTA CCC (CW, JD, and JW), considered a literature review would assist the CCC to develop a mentoring program. Due to resource constraints, work began as a “rapid review”. Rapid reviews take anywhere between 1 and 6 months, with a limited literature search and confine results and their analysis to qualitative synthesis (7). Such reviews still employ formal research steps: defining the research question, defining inclusion criteria, identifying biases, conducting the search in a manner that excludes as many biases as possible, and appraising the quality of documents (Reference Tricco, Antony, Wasifa, Strifler, Ghassemi and Ivory8). The project team participated in all the steps as well as consulting with CCC members.
The project team developed the the list of search terms given in Table 1.
Search terms mentoring
PubMed, CINAHL, and PsycINFO and a search of “grey literature” through Google and Google Scholar were undertaken in late 2018 and into 2019. In this subject area, “gray literature” search was productive since many organizations undertake mentoring programs but do not publish their results in academic journals. The combined searches produced fifty-seven articles in English. Of these, the project team selected thirty-five articles most relevant to the review's aim appropriate to mentoring Australian consumer representatives working in the HTA assessment areas. The articles selected are largely from UK, US, Canadian, and Australian sources.
Identified articles were listed and discussed in terms of reporting of evidence and relevance to the project. A limitation of our review was identified early on. Many of the articles provided scant evidence to support the outcomes they claimed. Data were lacking to assess bias or appraise quality of mentoring. Thus, the “rapid review” became exploratory and pragmatic, seeking out relevant information on mentoring with a view to developing and evaluating a pilot program.
The results were discussed again and structured into themes. Finally, articles were placed within a framework to inform the contribution of mentoring in HTA (Reference Norburn and Thomas4;Reference Abelson, Wagner, DeJean, Boesveld, Gauvin and Bean9;Reference Kaltenthaler, Cooper, Pandor, Martyn-St James, Chatters and Wong10). Following deliberations on limitations around evidence, the research question became: What models of mentoring and their attributes will be applicable to a program for consumer representatives in the HTA area?
As the literature was heterogeneous in its origins and in presentations of data, the narrative synthesis approach meant that those articles that best described the parameters of mentoring applicable to developing a mentoring program in the health technology context were included. In this article, to demonstrate the overall themes relevant to HTA committees, we mainly focus on the examples of mentoring programs undertaken by academic or other educational institutions.
Results
Results appear below under the terms the project team identified from the literature as features of a mentoring program framework. Results will assist consumer representatives in the Australian Government Department of Health HTA committees to decide on the components of mentoring that assist them to perform effectively either as mentors or as mentees. Following discussion with CCC members, the framework should be subject to ongoing evaluation and improvement.
Definitions of Mentoring
Few articles defined mentoring. The most relevant to the collegiate committee structure of the CCC emphasized mutuality and a preparedness to help others, a quality already present in the CCC that can be captured in a mentoring program. Manchester Metropolitan University, Human Resources defines mentoring as a process in which an experienced individual helps another person develop his or her goals and skills (11) through a series of time-limited, confidential, one-on-one conversations, and other learning activities. The Centre for Health Leadership and Practice (12) defines mentoring somewhat differently, as helping people to develop more effectively. Here, it is a relationship designed to build confidence and support the mentee to take control of their own development and work.
Primary Health Care Research Information Service (PHCRIS) “Mentoring Matters” (13) considers mentoring involves multiple roles depending on the needs of the mentees and defines it by its component parts:
(1) a mutual relationship where participants share experiences, knowledge, and information;
(2) a developmental process where the mentee grows in skills, knowledge, and confidence;
(3) a strategy to share intellectual and other resources;
(4) informal or formal support by a more experienced and skilled person;
(5) guided learning by the mentor; and
(6) traditionally a one-to-one relationship but group mentoring may be a beneficial option.
Mentoring Attributes or Qualities
These definitions emphasize positive relationships. The qualities of mentors to progress these relationships are important.
Mentoring Guidelines from Manchester Metropolitan University, Human Resources (11) and PHCRIS (13) suggest that a mentor should display some of the following attributes:
(1) Listening
(2) Asking questions to help develop mentor's and mentee's understanding of a situation or problem
(3) Providing information and knowledge and share informal networks
(4) Providing advice on career development
(5) Offering different perspectives
(6) Providing support and encouragement
(7) Offering guidance and advice in regard to qualifications
(8) Being a sounding board
(9) Being a critical friend
(10) Encouraging self-reflection
(11) Helping mentees identify areas for development
(12) Being available, open, respectful, accepting, willing to share and learn
(13) Answering questions, gives guidance and constructive feedback
(14) Motivating by setting an example
(15) Promoting the development of contacts and networks
(16) Advocating for the mentee.
These definitions and qualities emphasize that mentors will use their experience and knowledge in a facilitative manner to support the development of the mentee. However, the mentee is responsible for applying his/her learning and putting plans into action. With regard to HTA consumer representatives, almost of all of these qualities except perhaps those concerned with career development and qualifications will be of value.
Benefits of Training in Mentoring
While the above qualities are foundational, some programs recommend mentor training to ensure effective and consistent mentorship. Lindsey et al. (Reference Lindsay, Hartman and Fellin14) discussing mentoring youth argue that a well-trained, confident, and supported mentor is more likely to continue mentoring.
Similarly, the Center for Public Service and Engaged Scholarship at Harvard University (15) offers a 4-hour training program for all people mentoring at the university. Training is certificated and mandatory for all mentoring volunteers. The Victorian Institute of Teaching (VIT) (16) offers a two-day training program for experienced teachers from all Victorian schools and early childhood centers wishing to mentor teachers new to the profession to improve their skills.
Resources
An additional approach is to provide a mentoring handbook. The Postgraduate Medical Council of Victoria (PMCV) (17) provides handbooks to both mentors and mentees, so that both members of the relationship are aware of their roles, responsibilities, and expectations. The handbook provides templates of documents such as roles and responsibilities of the mentor which includes certification, a mentor preference form, where mentees can also specify with whom they prefer not to be paired, an activity log, a goal-setting template, and a mentor/mentee agreement form.
Both training of mentors and a handbook are being considered by the HTA CCC; a handbook may reduce administrative work, while training promotes consistency among mentors.
Other Considerations in Mentoring Programs
Articles were chosen from diverse workplaces such as academia, schools, professional organizations, and commercial workplaces. They demonstrated that mentors can be used in various functions such as induction, changing roles, and continuous professional development. As mentoring takes place within a defined situation such as the workplace, an institution, and among colleagues with similar professional qualifications, it assumes that at some stage, both mentor and mentee will be on some level of equality. It is an approach that is consensual, flexible, undertaken in work hours, relates to the job, is generally individual, people-centered, and provides a feedback system to enhance and embed learning.
Pairing of mentor/mentee may be done through an administrative process, which is based on arbitrary criteria, where mentor and mentee choose from one another based on sets of attributes, formal programs tend to adopt an administrative process while informal programs rely on personal choices. There is some evidence that where mentors/mentees have some choice, better outcomes result (Reference Meagher, Taylor, Probsfield and Fleming18).
Evaluations of Mentoring Training Programs
Evaluating Outcomes and Validated Tools for Measurement
Murdoch University evaluated a pilot program of mentoring new maths and science teachers at Western Australian schools (Reference McConney and Maor19). The evaluation found that mentees required a reduction in their workload of 20 percent to engage in mentorship; the program was best started at the beginning of the school year; both mentors and mentees required professional development to develop effective relationships (whereas most of the literature suggests only providing training to mentors); co-location in the same school saved time and was convenient. Additionally, a careful selection of mentors, perhaps based on their passions for teaching science and maths enhanced the value to mentees, as did building in regular milestone meetings throughout the year which improved accountability while also contributing to systematic evaluations to improve programs.
Most helpful to evaluating a mentoring program are validated tools to measure the benefit of mentoring programs. One is Mentoring Competency Assessment (MCA) (Reference Fleming, House, Shewakramani Hanson, Yu, Garbutt and McGee20). Fleming et al. (Reference Fleming, House, Shewakramani Hanson, Yu, Garbutt and McGee20) described evaluating MCA to assess skills of research mentors in clinical and translational research. The aim was to have a tool that could measure the value of various mentor characteristics and skills in mentoring new researchers. MCA was trialed among 220 pairs of mentors and mentees at 16 US universities. They combined the Mentorship Effectiveness Scale (MES) with two other tools used to evaluate mentorship. The result was a twenty-six-item scale across six areas (maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, promoting professional development, and fostering independence). The University of Wisconsin Institute for Clinical and Translational Research provides a full list of these resources for all mentees and mentors to self-evaluate as well as evaluate the entire program (21).
The MES is used in medical and nursing schools in the US (Reference Berk, Berg, Mortimer, Walton-Moss and Yeo22). It is a twelve-item, six-point scale to evaluate behavioral characteristics of mentors. MES is accompanied by the Mentorship Profile Questionnaire which describes the mentoring relationship and then specifies the outcome measures produced from the relationship. Supporting documents demonstrate what the relationship has produced, such as conference papers, published articles, and other projects.
Another aspect of evaluation covers the benefits of investing in mentoring. A longitudinal study of mentoring doctoral students showed that mentoring did not lead to greater research output (Reference Paglis, Green and Bauer23). Mentoring in school teachers was found to be a cost-benefit. Villar and Strong (Reference Villar and Strong24) asked if mentoring was worth the money in regard to teacher retention. Villar and Strong (Reference Villar and Strong24) found: “increases in teacher effectiveness yielded greater savings than the reduction in costs associated with teacher attrition. Overall, the benefit–cost analysis showed that, after five years, an investment of $1 produces a positive return to society, the school district, the teachers, and the students, and the state almost recovers its expenses.”
Be A Mentor, a US-based foundation working with vulnerable children and young people to improve their lives conducted a cost–benefit analysis to demonstrate the long-term benefits of trained volunteer mentors (25). They found mentoring was “an outstanding success”: that, for example, “98% of all youth mentored complete their high school education. That alone raises their employability, income and subsequently their ability to support and strengthen the economy, rather than become a drain on it. And the cost is only $1,500 per mentored youth per year. As noted in the chart, it costs twenty times more for a youth to receive drug/alcohol treatment in a single year and over 130 times more for a youth in the juvenile justice system than it does to mentor.”
Best Practice in Mentoring
“Best practice” is operationally defined in an academic setting as those actions that produce the most desirable faculty outcomes, based on evidence and real-life experiences (Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26).
A formal mentoring program is best practice when it has appropriately matched pairs, there is a clear purpose for the mentoring relationship with stated goals, the relationship is developed, the mentor advocates on behalf of the mentee and is able to integrate the mentee into the organizational culture and assist them access resources, such as administrative support and workplace documents (Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26). The evaluation of the process is also essential for best practice (Reference Meagher, Taylor, Probsfield and Fleming18;Reference McConney and Maor19).
The Center for Health Leadership and Practice (12) speaks about best practices in mentoring to emphasize that it is in the practice or exercise of mentoring that effectiveness will be achieved. A mentor does not have all the answers and does not give advice. Mentors assist the mentee to think more broadly and deeply; share experiences and lessons learned; assist the mentee to see alternative interpretations; provide supportive feedback and encouragement; and help the mentee to apply their own successful strategies from their past work to the new challenges.
Similarly, Sherk (Reference Sherk27) of the EMT Group of Companies (USA) consulted both administrators and mentors of mentor programs in community groups to identify best practices. Best practice in community groups develops networking to ensure quality improvement and recruiting mentors to sustain the program. The supervision of mentors ensures that mentor/mentee relationships prosper (Reference Sherk27). Sherk's work also identified pitfalls of mentoring: inadequate personnel and/or resources to run the mentoring program; lack of commitment by a program manager; lack of integration of mission and goals, or poor integration of the program within the community; superficial support of mentors; and finally, a program that offers little that is unique. Sherk suggests starting out small, with passionate leadership, learning from mistakes, and slowly building up a program. Good resourcing, especially with regard to personnel, is important.
Applying standards for best practice can assist effectiveness (28). Elements of Effective Practice for Mentoring bases effectiveness on a set of standards the organization has researched and practitioners of mentoring have approved in mentoring youth (28). Standard One is recruitment based on realistic aims, objectives, and outcomes. Standard Two concerns screening of mentors to assess their suitability as mentors and have time to commit to mentoring. Standard Three argues for training of both mentors and mentees to ensure a culturally safe mentoring relationship. Matching mentors with mentees to produce sustainable relationships is Standard Four. Monitoring and support by the parent organization should be provided throughout and constitutes Standard Five. Finally, Standard Six relates to a fulfilling closure to mentoring.
Limitations
We acknowledge the limitations of “rapid reviews”; in this case, a level of bias in terms of selectivity as the project team explored articles that would inform the design of a mentoring program for consumers in the health technology areas (Reference Tricco, Antony, Wasifa, Strifler, Ghassemi and Ivory8). Much of the literature cited dealt with only one aspect of a mentoring program and related successful outcomes.
The literature was generally positive in its assessment of mentoring, with no reporting of any adverse events, though some programs provided opportunities for both mentors and mentees to do so, suggesting that such events occur.
The greatest limitation of this review relates to the lack of published peer-review evaluations, making it difficult to assess the robustness of any data. Acknowledging the lack provides the opportunity to address it through data-driven evaluations in future mentoring work in Australia.
Discussion
Applicability of Findings
This review was conducted to assist consumer representatives in the Australian Government Department of Health HTA committees decide on components of mentoring applicable to assist them to perform effectively, either as mentors or mentees. While the literature search revealed few articles that had an empirical evidence-base, it demonstrated that mentoring as a form of adult learning is widely applied. There was evidence that some mentoring programs had improved the quality of a program by training mentors or providing more resources, but little data to support these claims. Largely the project team “gleaned” the essential components to include in a mentoring program and concluded that evaluation should also be undertaken to demonstrate effectiveness.
Given that current HTA consumer representatives are established in their roles, mentoring new consumer representatives in HTA processes offers a more flexible approach than training. Training is largely didactic and would require multiple curricula to work across committee structures, where needs and roles differ. As mentoring is built on mutual respect, sharing, and listening, it should also serve to attract and sustain future consumer representatives in the HTA space. Mentees are also potential future mentors.
Applications to Assist Consumer Representatives Contribute Effectively to HTA Committees
The review serves to provide knowledge and understanding to construct a model suitable to the Australian HTA context. Currently, project consultants are using the review to develop a pilot model, the design of which needs to consider that mentoring will take place in participants’ own time before or after formal meetings. Unlike mentoring that takes place within employment, Australian CCC members reside in different time zones, across the continent. Online resources may help address potential issues related to undertaking mentoring in one's own time (17;Reference Meagher, Taylor, Probsfield and Fleming18;Reference Nick, Delahoyde, Del Prato, Mitchell, Ortiz and Ottley26). A handbook and resources could be tailored for CCC members as an adjunct to an online learning platform where people join one another online at convenient times. A handbook has the benefit of contributing to consistency across the mentoring process as does an online program; it should provide essential skills and direction on the HTA procedures to mentors and mentees in Australia.
Another possibility is that a group program could be developed, trialled, and made available online so that mentors and mentees can meet up at times convenient to themselves. Online handbooks and other resources would also be made available.
Other considerations are mentor burnout and difficulties in pairing. Over-burdening mentors will be monitored and evaluated in the initial pilot program.
Evaluation
Evaluating the effectiveness of mentoring may benefit from basic organizational structures and a set of evaluation tools. The organizational structure for a mentoring program and its evaluation could include a project officer, budget, resources for both mentors and mentees, and time allocation, providing essential skills and direction to participate in HTA procedures in Australia. Best practice in the literature suggests mentoring should include clearly stated goals (usually set by the mentee), that there be a timeframe established which includes preparation, enabling, and closure. Mentors/mentees recording these steps should show whether goals were achieved or if goals have changed. The literature review provided access to a number of validated tools that may prove extremely useful in evaluating the effectiveness of mentoring to consumer representatives in their respective committees. As best practice (Reference Meagher, Taylor, Probsfield and Fleming18;Reference McConney and Maor19), evaluation has an additional advantage of contributing to accreditation of a mentoring program.
Challenges
In exploring mentoring and its applicability to the HTA area, the project team found useful indicators on what constitutes best practice as well as measuring effectiveness, but little evidence of their widespread use. Additionally, both peer-reviewed and “gray” literature, provided little detail of any adverse events arising from mentoring programs. One reason given was that both mentors and mentees do not like reporting negative experiences in their relationships (17). The development of resources and handbooks to mitigate against adverse events suggests it is a problem requiring some attention (17).
A related issue for the HTA CCC is the need to recruit more consumer representatives to HTA committees as current ones retire. A mentoring program could assist recruitment by providing introductory resources to applicants considering applying, detailing HTA work, and available help, including mentoring through the CCC.
Conclusion
This review provides descriptions of a range of mentoring programs that informs development of a pilot mentoring program for consumer representatives engaged in committees in the complex areas of Australian HTA. The aim is to sustain a consistent level of capabilities in health consumer representation on HTA committees.
Mentoring was chosen as the most appropriate vehicle to create a consistently supportive program as those who would be either mentors or mentees are usually experienced consumer representatives already participating in other committees in the health system.
The review explored definitions, effectiveness of mentoring, best practice, evaluation including measures, program design, and both group mentorship and one-on-one mentorship.
While training programs are used to upskill patient input in some health technology assessments such as in Canada and the UK, there were no examples of mentoring in HTA in the identified literature. However, useful examples emerged from academic programs where mentoring is used to retain young researchers, first-year teachers, and assist newly qualified clinicians and nursing staff.
This review provides background material for the development of a pilot program specifically for new CCC members which includes evaluating the program in terms of its effectiveness of consumer representatives' contributions to HTA committees. Currently, there is an ongoing pilot with one mentor and one mentee. As consumer representatives complete their terms, they can offer to mentor their replacements. Greater choice for mentees may evolve over time.
Acknowledgments
The authors acknowledge the contributions through discussions made by members of the Consumer Consultative Committee. The authors thank the reviewers of this article for their helpful comments in the preparation of this article.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflict of Interest
CW and JD are members of the HTA Consumer Consultative Committee, Department of Health, Australian Government and receive sitting fees.
JW is a Chair of the HTA Consumer Consultative Committee, Department of Health, Australian Government and is a full time Office Holder on the Pharmaceutical Benefits Advisory Committee (PBAC) in Department of Health, Australian Government.