Introduction
Adjudicative tribunals are an integral component of health system governance. Created by legislation, these quasi-judicial bodies are granted regulatory, oversight and dispute resolution powers to promote fairness in health decision-making, build confidence in health system management, strengthen governance structures, and improve health services and patient safety.Footnote 1 Their actual impact on the health system, however, remains largely unknown. Most evaluations of their work have focused on internal measures of accountability and independence rather than external indicators of societal influence. When their effectiveness is examined, assessors tend to utilize anecdotes from various experts and stakeholders rather than the rigorous empirical data that are almost certainly better suited for the purpose. As efforts to reform health systems continue internationally, it will be increasingly important to truly understand the benefits, costs and implications of adjudicative tribunals for providers and consumers of health-care services as well as the institutional structures on which they rely. The dynamic, independent and powerful oversight mechanism of administrative bodies, and their dispute resolution potential, may only be realized with further information on the ways in which they interact with the rest of the complex health system and the impact they have within it. A strong and accountable health system may depend upon it.
In this context, empirical evaluations are an opportunity to inform health policymaking through the collection of objective data regarding the impact of adjudicative tribunals on the health system. Such research includes both quantitative and qualitative investigations on the effects of enacted or proposed interventions – including laws, regulations and policies – on economic, social or health outcomes. Empirical evaluations may be distinguished from other types of research by their reliance on data and use of the scientific method of inquiry (Mello and Zeiler, Reference Mello and Zeiler2008). Empirical study designs range from experimental (e.g. randomized controlled trials) to quasi-experimental (e.g. interrupted time-series studies) to observational (e.g. cohort, case–control and cross-sectional studies), with data often gathered from surveys, interviews, focus groups, statistical inventories, performance data or documentary analyses.
Empirical research, however, is not new to the health or legal spheres. For health, experimentation, observation and the scientific method have at least notionally been at its core for over a hundred years, with modern ‘evidence-based medicine’ (Evidence-Based Medicine Working Group, 1992) even going as far as prioritizing the ‘conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients’ over all other possible inputs (Sackett et al., Reference Sackett, Rosenberg, Gray, Haynes and Richardson1996). In the legal arena, empirical research has also started to expand both in general (George, Reference George2006) and in health-related studies specifically (Mello and Zeiler, Reference Mello and Zeiler2008). However, when compared with the health sphere, it is clear that empirical methodologies in studies of legal institutions have been relatively underutilized. Existing literature explains this observation as a consequence of the complexity of legal interventions, the dearth of large-scale accessible data sets upon which to rely and the heterogeneity of legal interventions, which prevent natural experiments of cross-jurisdictional comparisons (Mello and Zeiler, Reference Mello and Zeiler2008).
In addition to these general challenges faced by all empirical legal researchers, any attempt to evaluate the impact of a health-related adjudicative tribunal faces additional hurdles. Not only has such an assessment never before been comprehensively undertaken but the most suitable research methodology to do so also remains elusive. Much of empirical health research, for example, relates to patient outcomes and the costs associated with achieving these outcomes. In the setting of adjudicative tribunals, these metrics may not apply. A proceeding before a health tribunal may take place after the outcome for the patient already has occurred, and for this reason the tribunals in fact may impose additional costs on the health system without directly yielding improved health outcomes. Although these additional costs may well lead to better practices and procedures on the part of other actors in the health system (e.g. regulatory colleges, insurance plans, hospitals), this type of benefit is indirect, may only become apparent over time and is inherently difficult to measure.
Distinctions in statutory mandate and the absence of clear statutory language setting out the purposes of adjudicative tribunals often leave no final target outcomes against which services can be evaluated. Further, as creatures of statute that serve quasi-judicial functions, adjudicative tribunals sit at the intersection of the legal and health spheres. These tribunals operate within these two paradigms – a dichotomy of process and outcomes – whose goals may sometimes diverge. Indeed, these administrative bodies are expected to preserve the legal focus on process, fairness and individual-level dispute resolution while at the same time working to improve health-related outcomes by enhancing the overall effectiveness of the health system.Footnote 2 The tension between a process and a substance-based mandate presents distinct challenges for empirical evaluation, especially as evaluations of either dimension would be difficult. The complex co-dependence and interconnectedness of these tribunals with the health system's constituent elements ensure that simple appraisal techniques cannot be effectively utilized. To the extent that adjudicative tribunals have an impact on the health system – which has yet to be empirically proven – this impact is likely linked to a host of other variables. The fact that evaluation is not easy, however, does not detract from its importance.
The absence of comprehensive empirical evaluations on the impact of health-related adjudicative tribunals, and the potentially significant benefits of doing so, certainly provides sufficient justification for further exploring this possibility. Empirical research, for example, is one of the most accurate ways to systematically assess the population's needs, capture stakeholders’ perceptions, test the effectiveness of new initiatives and verify improvements over time. It can help to identify areas of strength and weakness, point to opportunities for growth or improvement, and facilitate a continual process of enhancements so as to better serve the tribunals’ constituents and strengthen the health system. Empirical evaluations may also allow for greater accountability for the investment of public resources in health-related adjudicative tribunals. They can help to assess value for money and to determine whether and to what extent such tribunals have any impact at all, which may be particularly important in light of the recent global economic crisis and the austerity measures that have followed.
Despite these benefits and recognized importance, the evaluation and accountability of adjudicative tribunals is also one of the least scrutinized areas of administrative and health law. The topic necessarily engages the issue of administrative independence, the statutory environment within which all adjudicative tribunals operate, the policy priorities of government that funds tribunals, the complexity of the health system and the role of the court in supervising health-related adjudicative tribunals through the mechanisms of judicial review. Evaluating impact in the health sector is also necessarily a contextual exercise. As Peter Cane observed in the administrative law context, “the impact of judicial review needs to be studied in a contextualised way by reference to judicial review's objectives and functions. Also, it should not be assumed that, when we discuss the impact of judicial review, we are all talking about impact of the same thing or, at least, of a single institution with a single set of objectives and functions” (Cane, Reference Cane2004; see also Sunkin, Reference Sunkin2004; Pearson, Reference Pearson2007). A similar approach is necessary for health-related adjudicative tribunals but has never been systematically followed.
This paper aims to explore the context, challenges and opportunities for empirically evaluating the impact of adjudicative tribunals in the health sector. First, we will discuss the purpose, function and importance of these bodies within the health system, including their statutory mandates and policy goals. Second, we will examine the various ways in which their performance could potentially be assessed and will justify why there is a need to develop empirical approaches for the assessment of adjudicative decision-making. This part of the study will be illuminated with examples from a focused scoping review of past evaluations that we identified through searches in legal, medical and general databases (i.e. Westlaw, Quicklaw, MEDLINE, Cochrane Library, Campbell Library, Google Scholar), an existing review of empirical evaluations relating to administrative tribunals (Partington et al., Reference Partington, Kirton-Darling and McClenaghan2007) and other studies that were already known to us and the various experts and colleagues we consulted. Special effort was made to find empirical studies, with non-empirical works collected and reviewed as found. Third, we will identify the extensive barriers to empirically evaluating the societal impact of adjudicative tribunals, which we situate into three distinct categories: (1) complexity of the health system, (2) methodological difficulties and (3) realities of the legal profession and the environment in which it currently operates. Finally, based on this analysis, we will advance what we believe to be the most constructive path forward for the empirical assessment of adjudicative decision-making. We hope that this work will encourage and inform future empirical evaluations of adjudicative tribunals in the health sector that will help to assess their worth, improve their performance, enhance health decision-making, advance patient safety and facilitate the achievement of population health goals. Such evaluations would also help to confirm or dispel current beliefs and opinions on adjudicative tribunals, which in most instances, including our own, are based predominantly on personal experience rather than more objective and systematic assessments.
To enhance the utility of this paper for health policymakers in any context, each of these sections will be brought to life by focusing on their application to two prominent health tribunals in Canada, namely, the Ontario Health Professions Appeals and Review Board (HPARB) and the Health Services Appeals and Review Board (HSARB). Both HPARB and HSARB have statutory mandates to review important health decisions that intimately affect the lives of their constituents. Using these two bodies as case studies for exploring the context, challenges and opportunities for evaluating adjudicative tribunals may enrich our understanding of administrative tribunals throughout other sectors as well. More broadly, we believe that this analysis may resonate with the experiences of health-related adjudicative tribunals in other common law jurisdictions including the United States (e.g. U.S. Department of Health & Human Services Appeals Board, Medicare Appeals Council, California Department of Managed Health Care's Independent Medical Review Program), United Kingdom (e.g. Health Professions Council, Family Health Services Appeal Authority, Mental Health Review Tribunal), Australia (e.g. Health Professions Licensing Authority, Occupational Safety & Health Tribunal), New Zealand (e.g. Health Practitioners Disciplinary Tribunal, Pharmacy Board of Appeal) and South Africa (e.g. Health Professions Council, Mental Health Review Board). The various shared features among all adjudicative tribunals – including their creation by legislation, use of delegated powers, dispute resolution services, legal status and oversight function – ensures cross-border utility of this analysis and the relevance of case studies from one jurisdiction to another.
The purpose, function and importance of adjudicative tribunals in the health sector
Adjudicative tribunals are administrative bodies that are created by statutes and that exercise delegated decision-making powers of the executive branch for the purposes of achieving certain policy goals. They serve as an oversight mechanism for lower-level decision-makers and apply legal and normative principles to resolve disputes between conflicting parties. They are independent – operating at arm's length from the government – and serve quasi-judicial functions otherwise fulfilled by the formal judicial system. This independence, however, also has limits; their members are appointed by the executive branch of government (in the case of HPARB and HSARB, the power of appointment is effectively in the hands of the Minister of Health), which also sets their staffing allowances and budgets. Their decisions, although often final, must be authorized by their enabling statute and are subject to judicial review by the courts. Governments pursue policies in relation to these bodies for a host of reasons. The government may, for example, wish to remove the need for court intervention, facilitate opportunities for settlement, enhance access to efficient dispute resolution mechanisms or promote fairness.
In the health sector, adjudicative tribunals may be involved with resolving disputes regarding medical malpractice claims, insurance coverage for health-care services, determination of mental capacity, licensing decisions for health-care facilities and patient safety procedures. They serve as an accountability mechanism that ensures health decision-makers follow appropriate processes and act according to their respective statutory mandates. Adjudicative tribunals aim to boost public confidence in the credibility of decision-making within the health system, facilitate better and more consistent decisions, and reduce the risk of errors that in this context can have deadly consequences. Finally, they can help to promote independence, fairness and justice within health care, militate against self-interest and corruption, and provide opportunities to address wrongs through redress.
The HPARB, for example, is an integral part of Ontario's self-regulating health professional system. It helps to ensure that the health professions are regulated in the public interest, that appropriate standards of practice are created and maintained, that patients have access to the health professional of their choice and that they are treated with respect and sensitivity by health professionals. HPARB was established as a response to two related phenomena in the late 1960s and early 1970s: first, the recommendation arising out of the Report by the Honourable James Chalmers McRuer's Royal Commission Inquiry into Civil Rights, which emphasized the need for public interest oversight over self-regulating professional bodies (McRuer, Reference McRuer1968); and second, the Committee on the Healing Arts tabled by the government on 28 April 1970, which also emphasized the primacy of public interest regulation of health professionals (Ontario Committee on the Healing Arts, 1970). Under the Province of Ontario's Regulated Health Professions Act (RHPA), people may appeal the decision of a self-regulated health professional college to not pursue a disciplinary proceeding before HPARB.Footnote 3 If the appropriate statutory processes were not followed by the relevant college, then the board is empowered to send the matter back to the college for reconsideration. HPARB also hears appeals from adverse decisions by the colleges in relation to registration requests.
The HSARB, similarly, is a part-time board providing oversight for the decisions of various actors within the health system. Its broad jurisdiction arises from 14 different statutes and includes reviewing decisions concerning payment for health-care services under the Ontario Health Insurance Plan (OHIP), eligibility for housing in long-term care facilities, licensing of nursing homes and other independent health facilities and the decisions of public health officials (Pitfield and Flood, Reference Pitfield and Flood2005).
Both HPARB and HSARB have a full-time chair,Footnote 4 and a roster of part-time members, some of whom have legal training (and in the case of HSARB, medical training) and some who do not. Each board is supported by a shared secretariat, which forms part of the Ontario Ministry of Health. Both Boards have been held to be expert bodies by reviewing courts, which warrant deference. Their substantive decisions may only be overturned if found to be ‘unreasonable.’Footnote 5
As indicated above, a key aspect of evaluating tribunals created by statute is to assess whether a tribunal is fulfilling its statutory objective(s). This may be especially challenging, for example, if the specific goals of the relevant tribunal are diffuse and ambiguous in their enabling legislation. Ontario's RHPA, for example, does not detail the purposes of the boards, so this must be inferred from the powers and authority they have been provided. For example, HPARB has the power to review decisions of regulated health colleges not to refer complaints to discipline on grounds of the reasonableness of the college's decision and the adequacy of the college's investigation.Footnote 6 HPARB has broader jurisdiction to review decisions by colleges to deny registration to applicants.Footnote 7 Thus, although HPARB's role is generally to ensure public interest accountability over decision-making by regulated health colleges, HPARB's role in reviewing complaints suggests a different purpose, and a more deferential standard of review, than its role in reviewing denials of registration. Evaluation needs to be responsive to these differences of statutory mandate.
Approaches to evaluating health-related adjudicative tribunals
Assessing the work of these adjudicative tribunals and others in the health sector is an inherently complex enterprise. However, evaluations can be thought of and categorized according to their orientation and methodology (see Table 1).
Categorizing questions about health-related adjudicative tribunals according to their orientation and the possible methodologies that can be used to answer them
Examples of research questions that can be answered by empirical impact evaluations, the focus of this paper, are located in the bottom right-hand cell of this table.
In terms of orientation, evaluations of tribunals can be focused on how they function or what impact they have. The former would analyze the internal operations of a tribunal while the latter would assess the body's external effects on a specified population. Procedural analyses are important to promote coherent internal management structures, good governance, accountability, efficiency and efficacy. External impact evaluations, on the other hand, represent a way to assess the real-world effectiveness of the adjudicative tribunal, its impact on others within the health-care system and the benefits (or consequences) that this impact yields. Such studies can determine whether or not these bodies support and/or enhance the functioning of various health system institutions and decision-makers and whether or not they ultimately influence service provision, access to justice in the health sector and health outcomes. External impact evaluations require expertise and independence – they are not traditionally conducted by auditors (e.g. McCarter, Reference McCarter2008), ombudsmen (e.g. Marin, Reference Marin2008) or internal staff (e.g. Health Professions Regulatory Advisory Council, 2008).
Government evaluations of both health and non-health administrative bodies tend to focus on issues related to internal operations rather than external impact. The recent report of the Ontario Security Commission's Fairness Committee, for example, examined whether the agency's internal governance structure created a perception or reality of bias in its adjudicative responsibilities (Osborne et al., Reference Osborne, Mullan and Finlay2004). The United Kingdom's National Audit Office similarly reviewed the procedures used by its Department of Work and Pensions to medically assess incapacity and disability (National Audit Office, 2003a) and to hear appeals of social security benefit decisions (National Audit Office, 2003b). Some reviews examine particular problems that had previously been identified (e.g. Blumenthal and Wessely, Reference Blumenthal and Wessely1993; Blumenthal and Wessely, Reference Blumenthal and Wessely1994; Society of Ontario Adjudicators and Regulators, 1995), while others focus on users’ satisfaction with a tribunal's provision of services (e.g. Carscallen et al., Reference Carscallen, Gray and Pink2003; Confederation of British Industry, 2005; Employment Tribunals Service, 2005; Aston et al., Reference Aston, Hill and Tackey2006). Several assessment efforts have even focused on the internal operations of multiple tribunals or a jurisdiction's entire tribunal system, including the report of Ontario's Agency Reform Commission (Guzzo et al., Reference Guzzo, Baird, Grimmett, Martiniuk and Flaherty1998), the UK's Leggatt Review of Tribunals (Leggatt, Reference Leggatt2001) and the report of the UK's former Council on Tribunals (Adler and Gulland, Reference Adler and Gulland2003). Academics also tend to focus on the internal operations of tribunals across various topics – whether they regulate securities (Moyer, Reference Moyer1997; Rousseau, Reference Rousseau2008), medical malpractice claims (Siegal et al., Reference Siegal, Mello and Studdert2008), denials of health insurance coverage (Gresenz and Studdert, Reference Gresenz and Studdert2005), privacy (Jacobs, Reference Jacobs2008), pensions (Sossin, Reference Sossin2007) or determinations of medical incapacity (Peay, Reference Peay1989; Bradley et al., Reference Bradley, Marshall and Gath1995; Dolan et al., Reference Dolan, Gibb and Coorey1999) – and often examine users’ experience (Genn et al., Reference Genn, Partington and Wheeler2006a, Reference Genn, Lever and Gray2006b). While not a single governmental evaluation could be found that focused on the external impact of adjudicative tribunals, at least one academic publication discusses the potential benefits that administrative ‘health courts’ (which resolve malpractice claims) can have on patient safety (Mello et al., Reference Mello, Studdert, Kachalia and Brennan2006).
In terms of methodology, assessments of tribunals can either be conducted through expert reviews or empirical evaluations. The first approach would take advantage of the personal experiences and perspectives of an investigator while the second approach harnesses the objectivity of the scientific method and the generalizability of data that were collected from many people. Expert reviews rely upon the contextual and reflective expertise of the authors and are important for probing the etiology of complex challenges within the tribunal system, raising questions of possible concern or future inquiry, identifying structural problems and possible ways to overcome them, justifying political decisions (either from the past or those planned for the future) and suggesting recommendations for reform. This approach is also more likely to have fewer costs and a faster completion timeline. Empirical evaluations of tribunals, by contrast, seek scientific, evidence-based methods and can be used to, inter alia, quantitatively or qualitatively assess their impact on the health system, identify the factors that determine their successful operations, and track perceptions of them over time. An empirical evaluation may or may not lead to recommendations for reform. That said, these two methodological approaches cannot in reality be so strictly distinguished. Experts often utilize empirical methods and even the most scientifically rigorous and objective evaluations must be interpreted by individuals (often experts in the field) who may have stated or unstated agendas for reform.
Reviews of adjudicative tribunals have been conducted using both expert and empirical methodologies. Prominent observers, academics and practitioners, for example, have assessed various tribunals’ organizational structures (Moyer, Reference Moyer1997; Osborne et al., Reference Osborne, Mullan and Finlay2004), efficiency (Carscallen et al., Reference Carscallen, Gray and Pink2003), accessibility (Adler and Gulland, Reference Adler and Gulland2003), independence (Rousseau, Reference Rousseau2008), performance standards (Society of Ontario Adjudicators and Regulators, 1995), accountability (Allsop and Jones, Reference Allsop and Jones2008) and overall effectiveness (Guzzo et al., Reference Guzzo, Baird, Grimmett, Martiniuk and Flaherty1998; Leggatt, Reference Leggatt2001; Thomas, Reference Thomas2005; Mello et al., Reference Mello, Studdert, Kachalia and Brennan2006; Sossin, Reference Sossin2007). Other reviews feature empirical elements such as (1) surveys that capture the perceived quality of services offered (Employment Tribunals Service, 2005), stakeholder attitudes toward the tribunal (Confederation of British Industry, 2005) and the functioning of a certain process (Corsi and Hurley, Reference Corsi and Hurley1979; Toubman et al., Reference Toubman, McArdle and Rogers-Tomer1995; Latreille et al., Reference Latreille, Latreille and Knight2004, Reference Latreille, Latreille and Knight2005); (2) interviews that probe users’ experiences with the tribunal (Aston et al., Reference Aston, Hill and Tackey2006; Genn et al., Reference Genn, Lever and Gray2006b), its perceived impartiality (Jacobs, Reference Jacobs2008) and the effectiveness of a particular procedure (Bradley et al., Reference Bradley, Marshall and Gath1995; National Audit Office, 2003a; Siegal et al., Reference Siegal, Mello and Studdert2008; Urwin et al., Reference Urwin, Karuk, Latreille, Michielsens, Page, Siara, Speckesser, Boon and Chevalier2010); and (3) performance data and documentary analyses for examining key features of a tribunal's caseload (Hayward et al., Reference Hayward, Peters, Rousseau and Seeds2004) and arrangements for how, when and why it makes appeal decisions (Peay, Reference Peay1989; National Audit Office, 2003b; Gresenz and Studdert, Reference Gresenz and Studdert2005).
The challenge in evaluating health-related adjudicative tribunals, therefore, seems to lie at the intersection of orientation and methodology. Assessments of adjudicative tribunals have focused on both process and impact, and have been conducted using both expert reviews and empirical methods; yet, not a single review could be found that empirically evaluated the external impact of an adjudicative tribunal, despite extensive searching (see Online Appendix A for 27 examples of adjudicative tribunal evaluations listed according to these categories). While this lack of research may indicate that such undertakings are not important, interesting or possible, the evidence suggests otherwise: the need for external impact evaluations is evident (OECD Development Assistance Committee, 1998; Hertogh and Halliday, Reference Hertogh and Halliday2004; Center for Global Development's Evaluation Gap Working Group, 2006; World Bank, 2006) and such evaluations have been conducted with success in related settings, which also involve the nexus of the health and law sectors and beyond (Jones et al., Reference Jones, Jones, Steer and Datta2009).
The dearth of externally focused empirical evaluations is not only a missed opportunity but it may also pose a significant risk. On the one hand, the lack of an empirical rationale for the benefits of adjudicative tribunals may render them vulnerable to opposition or simply to general cost-cutting initiatives. On the other hand, ineffective tribunals may unjustifiably be receiving government funding at the expense of other programs, which cannot be supported as a result of limited resources. Without these data, adjudicative tribunals may also lack the baseline measures needed to track changes over time, evaluate the performance of decision-makers and staff, and engage in longer-term strategic planning.
Indeed, it is widely accepted that data-driven strategies are more likely to help decision-makers achieve their goals in a cost-effective way than policies pursued in the absence of evidence (World Health Organization, 2004; Maynard, Reference Maynard2006; Chalkidou et al., Reference Chalkidou, Tunis, Lopert, Rochaix, Sawicki, Nasser and Xerri2009; Hoffman et al., Reference Hoffman, Lavis and Bennett2009). Information gathered by health-related adjudicative tribunals like HPARB and HSARB through empirical methods may be of particular interest to government officials as it can demonstrate performance benchmarks and ensure that public funds are being invested and spent effectively. If reform is called for, empirical data will be essential in identifying what needs to change. For academics, it is an under-scrutinized sphere of administrative law and health systems functioning that is both ripe for research and, potentially, reform.
Challenges for empirically evaluating adjudicative tribunals in the health sector
Yet despite the tremendous benefits, empirical impact evaluations of health-related adjudicative tribunals are not being conducted. This absence of assessment efforts is most likely attributable to the various challenges facing anyone who embarks on undertaking such a project. These obstacles can be divided into three categories: (1) complexity in the health system; (2) methodological complications; and (3) environmental realities in which the legal profession and tribunals operate (Table 2).
Summary of the complexity, methodological and environmental challenges associated with empirically evaluating the impact of adjudicative tribunals in the health sector
Challenges with complexity in the health system
Empirically evaluating the impact of any adjudicative tribunal is a naturally difficult enterprise as it requires the body's various effects to be isolated from the larger social context within which it operates. This is no doubt complicated for tribunals in every sector because their activities are usually only indirectly related to their existential goals. This challenge, however, may be further exacerbated in the health context due to its overwhelming complexity.
Indeed, health systems are increasingly being recognized as complex adaptive systems that are multi-layered, non-linear and highly sophisticated. They consist of countless sub-systems with immeasurable independent actors, established policies, zealously guarded interests, entrenched professional ‘silos’ and divergent cultures that can all influence each other and even alter their external environments. This web of elements, and the unpredictable interactions among them, ensures that conventional mechanistic or ‘cause–effect’ conceptualizations of the health system are inaccurate and oversimplifications of its complex dynamics (Zimmerman et al., Reference Zimmerman, Plsek and Lindberg1998).
While scientific knowledge has been greatly advanced by breaking big questions into smaller ones that can be observed, analyzed and understood through rational deduction, this process is severely limited when the studied phenomenon or intervention is located within a system whose constitutive parts are not independent, constant or predictable. The fact that the health system exhibits characteristics of distributed control, co-dependence and nesting of smaller systems within other larger systems further aggravates this challenge and makes it difficult to fully examine adjudicative tribunals without reference to other actors and institutions (such as its adjudicators, staff, government policymakers, regulatory colleges, relevant expert panels, the traditional court system and the public). Isolating and attributing impact are further problematized by the fact that health-related adjudicative tribunals serve diverse functions according to various players within completely different contexts (Plsek and Greenhalgh, Reference Plsek and Greenhalgh2001).
Challenges with research methodology
Yet in addition to the daunting barriers of evaluating adjudicative tribunals caused by health system complexity, there are further methodological barriers associated with such an undertaking. The primary challenge, as highlighted above, is that simple research designs cannot be used to isolate adjudicative tribunals and to elegantly locate causal inferences between them and their goals. But above and beyond the various explanations illuminated by the complexity perspective is the fact that efforts of adjudicative tribunals are only indirectly related to their goals. Indeed, health services themselves only partially help meet their goal of improved health for people. Any legal, regulatory or oversight ‘intervention’ that serves to better structure these services would be even further removed from their ultimate goals. Empirical impact studies of such interventions must be expertly designed to account for this complexity.
However, even if simple methods did exist to observe the relationship between adjudicative tribunals and their goals, there is currently a lack of clear evaluative criteria against which particular adjudicative tribunals can be measured. This is because their goals are not easily articulated and have thus not been defined with adequate precision – if defined at all. Desired outcome measures are consequently absent, which ensures that suitable quantitative and/or qualitative research methodologies cannot be matched to them. This problem, however, cannot simply be overcome by brainstorming possible goals of adjudicative tribunals. Indeed, the existential purpose of these bodies may change and evolve over time with new legislators, government policymakers, adjudicators and tribunal staff who can each contribute toward a shift in the focus and priority of their operations over time.Footnote 8 Various community stakeholders may also perceive the role of a particular adjudicative tribunal in their sector very differently depending upon their own mandate, ideological perspective and unique vantage point. While reference to a tribunal's enabling statute may be informative in crafting an outcome measure, it is not always decisive. In the case of HPARB, legislative provisions suggest that this body was created to ensure effective regulation of the health professions in the public interest;Footnote 9 yet, this goal is not easily quantifiable. Indeed, the ability to empirically evaluate a complex intervention like a health-related adjudicative tribunal depends upon having a desired outcome that is observable, measurable and testable against a null hypothesis. Some intended outcomes may be impossible to measure or may depend on factors outside a tribunal's control, such as how others respond to their decisions and recommendations.
A desire to empirically ‘prove’ causal inferences between adjudicative tribunals and a particular outcome is also complicated by the impossibility of randomly allocating potential users of existing tribunals into groups that either receive or do not receive their services. Randomized controlled trials – the most rigorous of discrete empirical evaluations (GRADE Working Group, 2004) – assess the effect of an intervention on a test population in comparison with a theoretically identical population. This method, however, requires properly constituted (i.e. randomized) and adequately sized (i.e. large) treatment and control groups with both known and unknown confounding factors evenly distributed between them in order to isolate the impact of tribunal services and measure it against a benchmark. Non-randomized retrospective evaluations comparing users of tribunals to non-users (or the situation of the general public in jurisdictions with and without comparable tribunals) may not be an ideal solution to this challenge as this creates a situation where user-status and outcomes are measured at the same time. This makes causal determinations infinitely more challenging and requires evaluators to systematically identify and control for confounding factors in a separate process (Mello and Zeiler, Reference Mello and Zeiler2008).
A penultimate methodological challenge for conducting external impact evaluations of health-related adjudicative tribunals is that there are few examples of past efforts upon which to emulate. As previously mentioned, many empirical studies have examined the internal processes of tribunals, but none could be found that focused on their societal impact. This is exacerbated by the dearth of obvious empirical data sets that can be analyzed and from which potential evaluators can draw (Mello and Zeiler, Reference Mello and Zeiler2008). Although hospitals may be able to compare their patient population and its outcomes to those from neighboring hospitals, adjudicative tribunals may not be in a position to continually collect data about their past users or to compare this information with existing data sets from the same region or others.
Finally, the identity and background of the researchers evaluating the impact of an adjudicative tribunal must also be considered. While the goal of empirical study is to avoid bias and ideological assumptions, every researcher brings a particular matrix of perspective, orientation, experience and values to their work. Insiders, for example, may bring intuition and experiential judgment, while outsiders may bring independence, fresh eyes and objectivity.
Challenges with the legal environment
As institutions that function within both the health and legal systems, health-related adjudicative tribunals must also overcome the realities of the legal sector that may not be particularly nurturing for empirical impact evaluations. For example, legal actors are often focused more on achieving due process, transparency and good governance than specific societal outcomes (like improved health status, which is the goal of direct clinical health care). Excellent process in the legal world is often thought to be the most likely way to achieve the best outcome and may indeed be constitutive of it in the absence of gold standards or clear benchmarks.
There is also greater concern for maintaining independence and avoiding any apprehension of bias within the legal sector than in other areas. Like impartiality, independence is a common law right of procedural fairness enjoyed by parties who come before administrative bodies in common law jurisdictions (including Canada, United States, United Kingdom, Australia and New Zealand). In Canada, independence for adjudicative tribunals is based on the categories of judicial independence identified by the Supreme Court of Canada in Valente v. The Queen (i.e. security of tenure, financial independence and administrative independence over adjudicative matters)Footnote 10 and applied to administrative bodies in Canadian Pacific Ltd. v. Matsqui Indian Band – albeit in a more flexible and contextually sensitive manner.Footnote 11
Respecting this independence of adjudicative tribunals will naturally influence the process and content of any evaluation in multiple ways. For example, independence suggests that governments should refrain from evaluating tribunals’ substantive decisions, lest reasonable observers reach the legally problematic conclusion that tribunals may adjust their decision-making to align with what the government of the day perceives as ‘successful’. Similarly, it may also be difficult for a tribunal to establish evaluative criteria or outcome measures for itself as this might lead a reasonable observer to conclude that the tribunal may pursue these goals at the expense of fairness to the parties. This concern for independence even questions the extent to which tribunals’ staff and members can be directly involved in any evaluation for fear of influencing or interfering with their services that must remain neutral at all times. In contrast to encouraging self-evaluation as is common within the health sphere, the legal environment may actually discourage adjudicative tribunals from assessing their own external impact, especially as such undertakings are not explicitly part of their statutory mandates.Footnote 12
Finally, as recently highlighted by the Nuffield Inquiry on Empirical Legal Research, the legal academy also suffers from a dearth of empirical competence and capacity to conduct such studies (Genn et al., Reference Genn, Partington and Wheeler2006a). While the field of empirical health law scholarship has recently grown exponentially (Mello and Zeiler, Reference Mello and Zeiler2008), it is generally accepted that the current capacity is inadequate and that it may further diminish over time. Empirical legal methodologies are also not generally recognized to be as prestigious within the academic community as traditional doctrinal investigations (Genn et al., Reference Genn, Partington and Wheeler2006a). The pervasive culture of deference to experts and authority must further diminish the perceived value of objective empirical work and weaken any apparent need for more rigorous research that is higher on the hierarchy of evidence (GRADE Working Group, 2004). Again, the focus on elements of process (e.g. bias and independence) rather than impact (e.g. judicial decisions) as indicators of quality and performance must also deter legal scholars from conducting work in this area such that target outcomes are less likely to be assessed.
Reasons for optimism
However, despite the challenges faced by potential evaluators of adjudicative tribunals, there is reason for optimism: each of the various identified barriers can be overcome and have indeed been circumvented in similar evaluations. For example, as previously mentioned, many empirical evaluations have been conducted that focus on the internal operations of these bodies. A major literature review in 2007 highlighted much of the work that has been conducted and published in this area (Partington et al., Reference Partington, Kirton-Darling and McClenaghan2007). Yet in addition to these studies, empirical evaluations have also been undertaken to assess the external impact of similarly functioning specialty courts that operate within the judicial system. A systematic review of the research evidence has even been conducted on the societal impact of at least one type of these judicial organs (Wilson et al., Reference Wilson, Mitchell and Mackenzie2006).
Indeed, methodologically, there may be much to learn from external impact evaluations of specialist courts in the judicial sector (Plotnikoff and Woolfson, Reference Plotnikoff and Woolfson2005). For example, ‘drug courts’ have been extensively evaluated in the United States and in other jurisdictions regarding their ability to increase treatment rates, lower criminal recidivism, and enhance cost-effectiveness of prosecution (United States Government Accountability Office, 2005; Wilson et al., Reference Wilson, Mitchell and Mackenzie2006; Green et al., Reference Green, Furrer, Worcel, Burrus and Finigan2009). Domestic violence courts and community courts have similarly been assessed for compliance, cost-effectiveness, conviction rates and public perception, and mental health courts have been comprehensively examined for reducing criminal violence, enhancing community safety, conserving fiscal resources and improving clinical outcomes (see Online Appendix B for a list of 91 external impact evaluations of specialist courts). However, it must be recognized that the context within which these judicial bodies operate is very different from that of health-related administrative tribunals. Not only are they part of the judiciary rather than the executive branch of government, but their existential goals are usually related to diverting complex or special cases from traditional courtrooms rather than supporting the infrastructure of a completely different system (like that of the health system). Empirically tracking desired outcomes like cost-savings and reduced reoffending rates will naturally be easier in this context when the intervention or service is more directly related to its goal. Yet, alternatively, it may actually be more difficult for these judicial organs to evaluate themselves due to their strict separation from the executive (which has the financial resources to fund such undertakings) and the likelihood of them to zealously guard their independence.
Researchers seeking to empirically evaluate health-related adjudicative tribunals can also benefit from earlier accounts of similar efforts. For example, Mello and Zeiler (Reference Mello and Zeiler2008) describe the diversity and comparative advantages of various empirical approaches that have been taken by scholars in the health law field to address issues as wide-ranging as medical malpractice reform and motor safety laws. Furthermore, on the use of randomized controlled trials, for which these two scholars are less optimistic, Pleasence (Reference Pleasence2008) provides an account of such an undertaking in the United Kingdom, which highlights the many technical, practical and ethical barriers that were faced, and suggests ways to overcome them in the future.
Opportunities for moving forward
Deliberate and concerted efforts, however, will be necessary to overcome the numerous barriers to empirical impact evaluations of health-related adjudicative tribunals among both individual evaluators and others that must support them. The analysis of challenges described above points to several strategies that can be pursued.
At the individual level, potential evaluators of adjudicative tribunals may need to assemble interdisciplinary teams to obtain the necessary methodological expertise, bring an aura of independence and credibility to their work, and save tribunal staff from the potentially uncomfortable situation of relinquishing their perceived independence by evaluating their own performance. Like the process for assessing the effectiveness of complex clinical interventions, evaluators of adjudicative tribunals may then be advised to conceptually map out the way in which their tribunal functions, its interactions and relationships with others in the health and legal systems, and its potential effects on each of them (Campbell et al., Reference Campbell, Murray, Darbyshire, Emery, Farmer, Griffiths, Guthrie, Lester, Wilson and Kinmonth2007). This will aid in focusing the inquiry, identifying areas in which little is known, generating suitable research questions and determining the appropriate methodology.
Potential evaluators must also thoughtfully consider both the target audience of their research and the overall goal that their particular health-related adjudicative tribunal is expected to help achieve, and then identify the most important targeted outcomes that are relevant to the audience and important for the goal's fulfillment. When such outcomes cannot directly be measured, as may often be the case, evaluators must identify strong surrogate endpoints, which are measurements that reflect important outcomes even if they are of indirect or diminished practical importance. Performance indicators can then be developed followed by the corresponding methodologies for tracking changes to them (see Panel 1).
Steps for moving empirical impact evaluations forward at the individual level
In the case of Ontario's health-related adjudicative tribunals, both HPARB and HSARB can describe their overall goal as contributing to the health of Ontarians by enhancing decision-making within the health system. If government officials are the evaluation's intended audience, targeted outcomes could include (1) confidence in the health system, (2) equity, justice and fairness in health decision-making, (3) strengthened health system institutions and (4) better health services and patient safety via enhanced regulation and oversight. As these outcomes would be nearly impossible to measure directly, surrogate endpoints can be developed and could possibly include (1a) access to adjudicative mechanisms for dispute resolution, (1b) perceived legitimacy of adjudicative decisions, (2a) satisfaction with adjudicative services, (2b) perceived fairness and legitimacy of adjudicative services, (3a) interaction among health system institutions and decision-makers, (3b) existence of support mechanisms for primary health decision-makers, (3c) effective oversight of primary health decision-makers, (4a) better decisions by primary health decision-makers and (4b) respect for the tribunal's oversight function. Performance indicators and their corresponding empirical methodologies could then range from the public's awareness for the tribunal's existence to the perceived concern among primary health decision-makers that their decisions will be reversed. Suggestions for inclusion as part of a government-focused, empirical evaluation matrix of the external impact of health-related adjudicative tribunals such as Ontario's have been presented in Table 3 to illuminate this process in the context of a real-world example. Evaluations of these bodies for other target audiences such as regulatory colleges, insurance plans, hospitals, individual health professionals or patients would likely lead to different measurable outcomes.
Suggestions for an empirical impact evaluation matrix for Ontario's health-related adjudicative tribunals
Note: The intervention to be examined in this evaluation matrix is the operation and services of Ontario's health-related adjudicative tribunals as part of the health system. The population under study is assumed to be Ontario residents in general and the users of the health-related adjudicative tribunals specifically, and the system goal is to promote better health among Ontarians through a strong health system that provides excellent health services. The counterfactual in this scenario would be the situation if Ontario's health system did not have any adjudicative tribunals. The primary target audience for this evaluation is assumed to be government officials.
Once a system of empirical observation is in place, potential evaluators can establish benchmarks according to which they can track and assess performance. Such benchmarks can be drawn from thoughtful consideration, aspirational goals of leaders, expert judgments on what is possible, data from similar tribunals in other jurisdictions (i.e. comparative analysis) or previous empirical observations from the same tribunal (i.e. interrupted time-series analysis). For experimental methods like randomized controlled trials that are rarer in socio-legal studies, the control group would serve as the comparative benchmark rather than any observational data that are external to the evaluation. Such comparisons are naturally better because they more accurately represent the counterfactual of what the situation would be like without the tribunal and can help lead to causal inferences.
However, overcoming the identified challenges and systematizing empirical impact evaluations of health-related adjudicative tribunals across time and jurisdictions require action from stakeholders throughout the health and legal systems. For example, scholars in the health law field must intensify their efforts to build capacity for utilizing empirical methodologies, enhance the status of such work within legal circles and overcome any real or perceived problems with maintaining independence. Health system institutions should also start to build policy-relevant databases that are rigorously compiled, comprehensive and publicly accessible. Finally, health planners and research funders must facilitate (or even catalyze) the continuous improvement of adjudicative tribunals by supporting undertakings to empirically evaluate their impact on society. Initial funding for small-scale evaluations and/or pilot projects would be particularly helpful, as would support for disseminating any lessons learned as widely as possible.
Syntheses of research evidence may be helpful in encouraging stakeholders to support empirical impact evaluations of health-related adjudicative tribunals, especially because they are likely to highlight the current dearth of knowledge in this area. A systematic review on the effect of adjudicative tribunals in the health sector, for example, would be a disciplined and rigorous approach to assess the current state of research evidence in this area and tracking developments in it over time (Lavis et al., Reference Lavis, Becerra-Posada, Haines and Osei2004; Petticrew and Roberts, Reference Petticrew and Roberts2008). This tool applies the scientific method to gathering, appraising and synthesizing what is known (and what is not known) on a particular topic such that publication and selection bias are limited (Lavis et al., Reference Lavis, Davies, Oxman, Densi, Golden-Biddle and Ferlie2005; Rothstein et al., Reference Rothstein, Sutton and Borenstein2005). Such a review has already been conducted, for example, to assess the impact of drug courts on criminal recidivism (Wilson et al., Reference Wilson, Mitchell and Mackenzie2006), and a protocol has been developed to evaluate the influence of these specialist courts on narcotics use in particular and criminal activity more broadly (Wilson et al., Reference Wilson, Mitchell and MacKenzie2007). User-friendly scoping reviews may also be extremely helpful for pushing this agenda forward in various contexts.
Conclusion
In summary, adjudicative tribunals serve an essential function within the health sector, yet their contributions and impact on the delivery of health services and society in general are not usually evaluated empirically. The focus of past tribunal assessments on their internal operations to the near-total exclusion of their external impact limit these bodies’ ability to inform continuous quality improvement efforts, enhance the public's confidence in them and maximize their societal impact. Three challenges, however, serve to complicate empirical impact evaluations of health-related adjudicative tribunals. First, the complexity of the health system and its countless independent actors prevents their mechanistic isolation, which is necessary to elegantly find causal inferences between them and their goals. Second, the indirect relationship between tribunal services and their existential purposes – exacerbated by uncertain objectives, difficulties with randomization and a lack of examples to follow – presents methodological barriers that cannot be easily overcome. Third, several realities of the legal profession and the environment in which tribunals currently operate further hinder evaluation efforts, including the dearth of empirical capacity, culture of deference to authority and focus on process and independence.
There are, however, two main reasons for optimism. Empirical evaluations of similar judicial bodies have been previously conducted and there is currently a rapid expansion of interest in empirical health law scholarship. This analysis of challenges to empirically evaluate the impact of adjudicative tribunals in the health sector highlights several potential ways to help move this agenda forward. Individual evaluators, for example, can assemble interdisciplinary teams, identify their tribunal's overall goal, develop surrogate endpoints and conduct a realistic evaluation that tracks each of them. Stakeholders within the health and legal systems, on the other hand, can support individual efforts by earmarking funds for such empirical impact evaluations, by building policy-relevant databases and by assisting with cross-jurisdictional learning and dissemination efforts. Syntheses of the research evidence on this topic, and systematic reviews in particular, may be helpful for highlighting the absence of knowledge in this area and for building support to capitalize on this otherwise missed opportunity.
Nevertheless, a preliminary question remains as to whether it is even the responsibility of adjudicative tribunals like Ontario's HPARB and HSARB to be empirically evaluating their own impact or to help others in doing so. Besides the limitations imposed upon them by their respective statutory mandates, these bodies and others may not necessarily be concerned about the impact of their decisions and could in fact be preoccupied with maintaining their independence. Further consideration must be given to these issues, and others, so that continuous quality improvement and self-evaluation can become part of tribunals’ core mandates as has been widely promoted for all government undertakings (Mayne, Reference Mayne2001); otherwise, it may be impossible for these bodies to provide the best services possible to their users, stakeholders and larger constituencies. Any ambitions for self-improvement among health-related adjudicative tribunals, however, must obviously be balanced with the legislative, political and social realities within which they operate.
Acknowledgements
The ideas shared in this paper were first presented at the Workshop on the Impact of Health Tribunals that was held at the University of Toronto's Faculty of Law on 28 May 2009 and funded by the Canadian Institutes of Health Research. We are grateful to the attendees of this workshop for their insights and feedback, especially Louise Bélanger-Hardy, Ron Ellis, Irwin Fefergrad, Colleen Flood, Chris Gill, Joan Gilmour, Grant Huscroft, Laverne Jacobs, Linda Pearson, Gabrielle St. Hilaire, Mariana Valverde, Kevin Whitaker and Albert Yoon. Additional helpful suggestions were offered by Michelle Mello and Martin Partington. An earlier version of this paper was also presented at the Canadian Institute for the Administration of Justice's Annual Conference held in Ottawa from 3 September to 2 October 2009 and published in a monograph summarizing that event. At the time that this study was conducted, Steven J. Hoffman was employed by the World Health Organization and Lorne Sossin was (and remains) a Vice-Chair of the Health Professions Appeals and Review Board and the Health Services Appeal and Review Board. The opinions expressed in this paper are those of the individual authors writing in their academic capacities and do not necessarily represent the views of their affiliated institutions.
Online Appendices
Appendix A: Twenty-Seven Examples of Evaluations of Adjudicative Tribunals.Appendix B: Ninety-One Examples of External Impact Evaluations of Specialist Courts.
Available at http://www.journals.cambridge.org/hep.
