Stakeholder involvement

It is pleasing that stakeholder involvement in the development of outcomes and measurement instruments has increased in recent years, and there are now many examples of key stakeholders being involved in the development of measurement instruments. However, one critique is that often stakeholder involvement is limited to providing views on predetermined outcomes, constructs, or domains that researchers and professionals have already decided are of interest. Of course, it would be foolish to completely dismiss professional expertise, which is often accrued over many years of research and or professional practice. Yet, researchers could and should give greater value to the expertise of lived experience – including in deciding what outcomes to measure (Bagley et al., Reference Bagley2016). It is no longer satisfactory to limit stakeholder involvement to areas already decided upon by professional groups. The research community is at a relatively early juncture when involving stakeholders in the development of outcomes and measurement instruments. In particular, it is important to facilitate meaningful involvement and avoid tokenistic consultation at all stages of the research process (Swarbrick et al., Reference Swarbrick, Doors, Davis and Keady2016), and this should extend to stakeholder involvement on outcome measurement development. It is the only remedy to a rather uncomfortable question; if outcomes and measurement instruments are not reflective of what key stakeholders value, can research really claim to create high quality or even adequate evidence to inform policies and practices?

Another pleasing development is the place that stakeholder involvement now has in recent appraisal checklists, such as the COSMIN Risk of Bias checklist (Mokkink et al., Reference Mokkink2018). These are tools with which reviewers appraise the quality of outcome measurement instruments. Stakeholder involvement is now part and parcel of quality assessments, and this has not always been the case. Progress indeed, but do these appraisals capture meaningful stakeholder involvement? Such tools do not really accommodate anything beyond reporting that there has been some involvement. Perhaps, we should be grateful for this, and it does mark progress. But surely we can conceptualize this better. There is an obvious dichotomy in the ontological–epistemological perspective between rich and deep stakeholder involvement and the philosophy central to quantitative measurement. It is worth questioning whether meaningful stakeholder involvement can be adequately summarized on the same scale as other quantitative psychometric properties.

What, why, and how are outcomes and measures chosen at the macro and micro level

There is a heterogeneity of outcomes and measurement instruments. There have been a number of attempts to attain consensus on outcomes and or measurement instruments that are recommended for use in the psychosocial and non-pharmacological fields of dementia research. These consensus exercises are useful, but some have not involved people living with dementia (Moniz-Cook et al., Reference Moniz-Cook2008), and interestingly, the starting point tends to be choosing from existing measurement instruments (Webster et al., Reference Webster2017). Ironically, overall, there is also inconsistency in the recommendations from these exercises that have sought to provide clarity because different measurement instruments are recommended for measuring the same outcome (Harding et al., Reference Harding2020).

At the micro level of decision-making relating to specific trials, it is the view among some in the research community that the inclusion of measurement instruments is often based on that inclusion will be expected, as opposed to relating to a theory of change or a sound intervention logic model (Couch et al., Reference Couch, Lawrence, Co and Prina2020). Whether this suspicion has merit or not, there is emerging evidence that the use of outcomes and measurement instruments may well not be measuring what key stakeholders value (Tochel et al., Reference Tochel2019; Harding et al., Reference Harding2020). For example, cognition and memory performance outcome measures, such as the Mini-Mental State Exam (MMSE), dominate dementia trials (Couch et al., Reference Couch, Lawrence, Co and Prina2020). This will not come as a surprise as the MMSE has become synonymous with dementia trials, but what is surprising is that the MMSE was not designed as an outcome measurement instrument and, as Harrison et al. (Reference Harrison, Noel-Storr, Demeyere, Reynish and Quinn2016) state in their review of outcome measurement instruments, its psychometric properties are poorly or often not described. On the other hand, concepts around the quality of life and social health are greatly valued by people living with dementia and other key stakeholders (Reilly et al., Reference Reilly2020), and yet these outcomes are used far less frequently (Harrison et al., Reference Harrison, Noel-Storr, Demeyere, Reynish and Quinn2016; Couch et al., Reference Couch, Lawrence, Co and Prina2020). For example, quality of life has only been included in 13% of trials (Harrison et al., Reference Harrison, Noel-Storr, Demeyere, Reynish and Quinn2016). Social health is still an emerging concept and, while recently developed outcome measurement instruments such as the Engagement and Independence in Dementia Questionnaire (EID-Q, see Stoner et al., Reference Stoner, Orrell and Spector2018), we think have some overlap, ultimately, we are aware of no relevant outcome measurement instrument for social health in this field (Harding et al., Reference Harding2020). In respect of carers, it is the presence of negative aspects such as depression and burden that trialists tend to measure (Couch et al., Reference Couch, Lawrence, Co and Prina2020), rather than positive aspects such as positive psychology (Pione et al., Reference Pione, Spector, Cartwright and Stoner2021).

If outcomes and/or measurement instruments are being included because of a perceived expectation or because of out of date assumptions, then there is a danger that trialists are missing things that are important to key stakeholders or relate to interventions. Given that many trials often show minimal or no effect (Reilly et al., Reference Reilly2015; Woods et al., Reference Woods, O’Philbin, Farrell, Spector and Orrell2018), are trialists using the “right” measurement instruments that matter to key stakeholders, particularly key stakeholders with lived experience? If they are using the “right” measurement instruments, and they do in some cases reflect the theoretical focus of interventions, scrutiny should also be applied to the focus and characteristics of interventions. Are interventions meaningful and substantial enough to affect an outcome? What is clear, at least, is that neither proposition is desirable and both are uncomfortable.

The role of core outcome sets and using the best outcome measurement instruments is vital

If there are questions about the extent of stakeholder involvement in the development of outcomes/measures and what, why, and how measurement instruments are chosen, perhaps, there is a need to sweep aside all existing assumptions about outcomes and measurement instruments? At the very least researchers need to ascertain which are the best and most robust outcome measurement instruments in a given field, and there needs to be some consistency in their application.

Core outcome set research, if done well, is capable of addressing many of the issues highlighted so far. Core outcome set research first asks the question “what should be measured as a minimum?” across all trials to increase the ability to compare for effectiveness before asking “how to measure?”. The Core Outcome Measures in Effectiveness Trials (COMET) initiative (http://www.comet-initiative.org/) has spearheaded the development of the core outcome set approach (Williamson et al., Reference Williamson2017). The “what should be measured?” question tends to be addressed by a multiphase and mixed-methods study design including using literature reviews and qualitative research to elicit a “long-list” of existing and new outcomes of interest (Harding et al., Reference Harding2019). It is then recommended to use consensus methods such as Delphi surveys and consensus workshops to identify the most important – or core – outcomes (Reilly et al., Reference Reilly2020). The involvement of key stakeholders throughout is key because then the core outcomes will be aligned to the priorities of key stakeholders, including people living with dementia and family carers (Morbey et al., Reference Morbey2019). To our knowledge, there are only two published core outcome sets in the field of dementia, concerning people living with dementia in relation to physical activity interventions (Gonçalves et al., Reference Gonçalves, Samuel, Ramsay, Demain and Marques2019) and broader non-pharmacological interventions (Morbey et al., Reference Morbey2019; Reilly et al., Reference Reilly2020).

“How to measure?” is best answered using a systematic review of the psychometric properties of outcome measurement instruments, including ascertaining face validity assessments of whether measurement instruments adequately capture core outcomes (Harding et al., Reference Harding2020). However, systematic reviews of outcome measurement instruments are incredibly important in their own right. This type of research enables the selection of the most robust, validated, and appropriate outcome measurement instruments. The example by Pione et al. (Reference Pione, Spector, Cartwright and Stoner2021) in the context of positive psychology measures relating to family carers is a great example of how to conduct a review of the psychometric properties of outcome measurement instruments. Researchers have a responsibility to adhere to the recommendations made in these types of reviews, and more broadly to ensure that the best and most appropriate outcome measurement instruments are used. Arguably, funders and journal editors too have a responsibility to encourage researchers to use the best and most appropriate outcome measurement instruments. Lest we forget, it is only the application of the “best” outcome measurement instruments that can support trialists and intervention research to create the high-quality and meaningful evidence that people living with dementia, family carers, policymakers, and practitioners yearn for and deserve.