Australia has been part of the global movement towards the inclusion of students with disabilities into regular schools and classes, having signed the Salamanca Agreement in 1994 (United Nations Educational, Scientific and Cultural Organization) and the United Nations Convention on the Rights of Persons With Disabilities in 2008 (United Nations Committee on the Rights of Persons with Disabilities, 2006). The Disability Standards for Education were introduced in 2005 (Australian Government, 2005), requiring schools to make appropriate adjustments to curriculum and pedagogy for students with disabilities to enable them to access education on the same basis as other students. Despite the national commitment, the states differ in approaches to inclusion, in definitions of disability, and in resourcing, and all states retain specialised settings for some students (Anderson & Boyle, Reference Anderson and Boyle2019).
Debate continues, sometimes passionately, about the merits of full inclusion in mainstream classes for all students, including those with the most severe and complex disabilities, compared to specialised or a mix of specialised and mainstream provisions designed to meet the needs of specific individual students (Carter & Abawi, Reference Carter and Abawi2018; Kauffman et al., Reference Kauffman, Travers and Badar2020). In Australia, there are several groups advocating for full inclusion and the closure of all specialised settings, including groupings allowing for targeted instruction within classrooms (Australian Coalition for Inclusive Education, n.d).
Arguments for full inclusion from advocacy groups and others generally derive from a social justice and rights-based approach, often appealing to Article 24 of the United Nations Convention on the Rights of Persons With Disabilities. Inclusion is seen as necessary for people to participate fully and actively in their communities and to be respected and valued as members of that community (Kliewer, Reference Kliewer1998). Those opposed to full inclusion note the right of all students to appropriate and effective education (Kauffman et al., Reference Kauffman, Felder, Ahrbeck, Badar and Schneiders2018; McLesky & Waldron, Reference McLesky and Waldron2011), and the rights of parents to choose an appropriate education setting, and the rights of other students (Farrell, Reference Farrell2000). They also note that education is a means to an end, and full inclusion in a community after school may be possible after a specialised education (Farrell, Reference Farrell2000).
Proponents of inclusion use the research literature to make claims about the effectiveness of inclusion in the education of students with disabilities. Unfortunately, the interpretation of the inclusion research literature that reports student outcomes is recognised as being problematic in that theoretical positions, definitions of inclusion, outcome measures, participants, and circumstances differ across papers. In comparative research, where outcomes from settings with different degrees of inclusion are compared, the comparability of samples and settings may be questionable and the research may not be high quality (Dyson, Reference Dyson2014; Farrell, Reference Farrell2000; Florian, Reference Florian2014; Foreman, Reference Foreman2015; Lindsay, Reference Lindsay2007; Nilholm & Göransson, Reference Nilholm and Göransson2017). In addition, although proponents of inclusion claim research reviews generally support inclusion (e.g., Freeman & Alkin, Reference Freeman and Alkin2000; Ruijs & Peetsma, Reference Ruijs and Peetsma2009), other reviewers find the evidence is inconclusive (e.g., Göransson & Nilholm, Reference Göransson and Nilholm2014; Limbach-Reich, Reference Limbach-Reich2015).
In Australia, there are ongoing advocacy efforts to increase inclusion rates and to abolish specialist settings altogether. Given the ongoing debate about rights and evidence in relation to inclusion, and that policy development and implementation is influenced by both values and evidence (Lindsay, Reference Lindsay2007), it is of interest to consider how research evidence is used in advocacy for full inclusion in Australia. One influential advocacy paper (Cologon, Reference Cologon2019) was produced by the organisation Children and Young People with Disability Australia (CYDA). CYDA is a not-for-profit organisation advocating for the rights of children and young people with disability, including the right to a fully inclusive education for all (CYDA, 2020a). It is a member of the Australian Federation of Disability Organisations and receives funding from the Australian Government. During 2019 and 2020, CYDA lobbied government departments and relevant ministers, participated in advisory committees, made submissions to inquiries and forums, and worked collaboratively with a large number of other organisations (CYDA, 2019, 2020b). CYDA also distributes fact sheets (https://www.cyda.org.au/resource/fact-sheets?start=15) based on Cologon (Reference Cologon2019).
Cologon (Reference Cologon2019) is an updated version of a similar paper by Cologon (Reference Cologon2013). Both papers were produced under the auspices of CYDA, and funding was received from the Australian Government to support their production. The position taken in these papers is strongly rights-based, citing the Convention on the Rights of Persons With Disabilities (United Nations Committee on the Rights of Persons with Disabilities, 2006). Both papers provide a consideration of the outcomes of inclusive education as well as discussion of issues around defining inclusion, attitudes to inclusion, means to increase inclusion, and the implications of research for policy development related to inclusion. Cologon advocates strongly for full inclusion and draws on the document produced by the United Nations Committee on the Rights of Persons with Disabilities, General Comment No. 4 (GC4), which provides further detail and explanation of Article 24 of the Convention on the Rights of Persons With Disabilities (United Nations Committee on the Rights of Persons with Disabilities, 2016).
The position presented is that research demonstrates the superiority of inclusive settings for all students, and services such as intensive specialised instruction based on need for individual or small group lessons whether in specialised settings or mainstream schools are opposed because they represent forms of exclusion or micro-exclusion. All necessary supports should be ‘embedded inclusively within everyday practices’ (Cologon, Reference Cologon2019, p. 23).
The report has been widely disseminated and links to it are provided on the websites of other disability organisations lobbying for inclusion (e.g., Australian Alliance for Inclusive Education; https://allmeansall.org.au/research/). It is referred to as an ‘evidence review’ by the Australian Coalition for Inclusive Education, a coalition co-convened and chaired by CYDA (https://acie.org.au/2019/11/09/disability-royal-commission-education-and-learning-frequently-asked-questions-faqs/). CYDA claims its work is ‘extensively cited’ (2020b, p. 14) in government and parliamentary reports.
Because this CYDA paper makes a clear claim that full inclusion for all is supported by research, we provide a detailed and specific analysis and critique of this aspect of the advocacy paper. The particular focus of our analysis is the use of inclusion research in advocacy in relation to the outcomes of inclusion, which is addressed in the second chapter of Cologon (Reference Cologon2019). In setting out to critique the use of research evidence in Cologon (Reference Cologon2019), and more specifically in the chapter addressing the outcomes of inclusion, we asked the following research questions:
-
1. What are the characteristics of the sources relating to inclusion that are used in this paper?
-
2. To what extent do sources cited as supporting inclusion provide conclusive evidence for the claims made about benefits and drawbacks of full inclusion and of specialised settings?
Method
The first stage of the analysis was to examine the reference list for the whole document to establish the nature of all the sources cited. Each item in the reference list was classified as being an original research paper in a refereed journal or not. Research sources included research articles, research reviews, or meta-analyses. Furthermore, the type of research reported in refereed journals was classified as original quantitative research (group or single case studies reporting results as descriptive or analytic statistics, survey or questionnaire, or analysis of data not collected by the authors), original qualitative research (reporting on interviews, observations, documents, etc., and analysed by qualitative methods), or mixed methods research (using both quantitative and qualitative methods). Review articles were those that contained a method that clearly described the search process and selection criteria for articles included and were classified as to whether they reviewed quantitative research, qualitative research, or both. Meta-analyses were identified separately. Articles that were published in refereed journals but that did not report original research were placed in the discussion category and included discussion articles, opinion pieces, or reviews that did not include a formal search or criteria for article selection, and position papers. Other references grouped together as agency reports included reports produced by government and non-government agencies, policy documents, articles in non-refereed journals, and curriculum documents. Books and book chapters were grouped together and no further analysis of the use of books as sources was carried out. Conference papers and theses were also identified.
Reliability of article classification by the first author was established by the second author independently classifying a random sample (chosen using random.org) of 20% (81) of the references. The classifications of the first and second authors for this sample were compared and percentage reliability calculated according to the following formula: agreements divided by agreements plus disagreements multiplied by 100. Reliability for classification as original research or not was 91%. Reliability for the more detailed classification was 81.5%.
The next stage was to analyse the references used in Chapter 2 addressing the outcomes of inclusive education to determine the number of references used, the frequency of their use, and the characteristics of sources used to support the claims contained in this chapter. Each statement that was footnoted with a citation or citations from the reference list was extracted. Each source cited for each statement was listed and the number of different sources used in the chapter and the number of times each source was cited were calculated.
As there were 93 footnotes, many with more than one citation, we then used random.com to select a sample of 20% (19) of the footnotes for further analysis. The first determination was to ascertain the standard of evidence provided by the source(s) for each claim made. The standard of evidence was established according to the criteria in Table 1.
Criteria Used to Determine the Standard of Evidence
We also extracted relevant data from research studies, including the definition of inclusion used by the authors. For research studies other than reviews and meta-analyses, we also extracted information on research design (group studies with comparison groups, single case designs, survey or questionnaire, or other), participants (students with disabilities, typically developing students, regular classroom teachers, special educators, paraprofessionals, student teachers, parents/carers/family, other), country of study, and setting of study (early childhood, elementary/primary/middle school, secondary school). For research studies (quantitative and qualitative) that included participants in settings described as inclusive, we extracted the nature of any supports provided, and the nature and number of participants with disability. For discussion papers and agency reports, we extracted the definition of inclusion used in the source. It was impractical to analyse books.
We also analysed the sources cited five times or more in Chapter 2 by extracting data as described in the above paragraph. We made a determination of the level of evidence that could be provided generally by each of these sources, according to the criteria in Table 1, but taking into consideration the paper as a whole.
Reliability of coding and data extraction for the random sample, for the most frequently cited papers, and for additional coding for research papers including students with disabilities was established by independent coding by the authors. Reliability of coding and data extraction for the random sample was 84.7% and was 85.5% for the most cited papers. Reliability for additional coding of quantitative and qualitative papers that included students with disabilities as participants was 100% for the number and nature of students with disabilities. Reliability for the nature of supports was below 80%, primarily because information about supports was scattered through the articles and little detail was provided. Extraction of data on supports was completed by consensus coding between the authors.
Given the high number of sources cited that did not support the claims made, five sources judged not to support the claim they referred to were randomly selected using random.org. These are used as examples to illustrate the use of sources and are related to footnotes 30, 34, 42, 58, and 98.
Finally, we selected five broad, general claims that were made in Chapter 2 that we considered potentially misleading. The selected claims drawn from pages 9–11 are listed in Table 8.
Results
Complete Reference List
There were 404 items listed in the reference list for the whole paper. Of these, 208 (51.5%) were research papers published in refereed journals (93 original quantitative research, 76 qualitative research, 17 mixed methods, eight quantitative reviews, one qualitative review, four meta-analyses, and nine mixed reviews). The remaining items consisted of 109 discussion papers in refereed journals, 34 agency reports, 49 books, two conference papers, and two unpublished master’s theses.
Chapter 2: All Sources
For Chapter 2, there were 93 footnotes with a total of 320 citations using 149 different sources. One footnote was not a citation and two citations could not be identified, as there were two items in the reference list with the same author and date, leaving 147 that could be coded. There were 96 (65.3%) research papers (47 original quantitative research papers, 29 original qualitative research papers, eight mixed methods, four quantitative reviews, five mixed reviews, and three meta-analyses), 36 discussion papers, six agency reports, eight books, and one conference paper. The most frequently cited sources (five uses or more) are shown in Table 2 and comprise 32.5% (104 out of 320) of the citations. None of the most cited papers included students with disabilities as participants, and only two were judged able to provide definitive support for a claim.
Sources Cited Five Times or More in Chapter 2
Note. Items marked with an * were included in the random sample.
Random Sample From Chapter 2 Citations
In the sample randomly selected for more detailed analysis, 56 sources were cited with some sources used more than once. Sources used more than once supported different claims, so each use is counted separately. Only seven sources provided definitive evidence for the claim made, and 27 sources did not support the claim made. Table 3 provides summary statistics of the level of evidence provided by the 56 sources.
Level of Evidence Provided by Random Sample of Sources in Chapter 2
The majority of the 18 different quantitative studies in the random sample were surveys (n = 11), and two mixed research papers both included a survey and a qualitative component. Four studies compared groups of students in inclusive and specialised settings. There was one intervention study, and two papers analysed large datasets.
Table 4 provides information about the participants, country, and setting of quantitative (n = 18), qualitative (n = 7), and mixed (n = 2) research papers. Only half of these papers included students with disabilities as participants and most commonly studies were carried out in the United States in elementary/primary school settings.
Participants, Country, and Settings of Research Studies in Chapter 2 Sample
The definitions of inclusion, when provided, varied considerably across the 18 quantitative papers, and it appears few, if any, students were fully included. Table 5 summarises the definitions.
Description of Inclusion in Quantitative Studies in Chapter 2 (n = 18)
The information provided about supports and special education services provided in the studies in the Chapter 2 sample that included students with disabilities in mainstream settings was also very variable (Table 6).
Special Education Services Provided in Inclusive Settings for Studies Including Students With Disabilities in Chapter 2 Sample (n = 13)
Note. Some studies included students receiving more than one service.
Random Sample of Sources That Did Not Support the Claim Made
The sample of five sources that were judged not to support the claim made, along with the relevant claim and footnote number and reasons the claim was not supported, is provided in Table 7.
Claims, Sources, and Reasons Claim Is Not Supported
Generic Claims and Contradictory Evidence
Claims Made in Chapter 2 for Which There Is Contradictory Evidence
Table 8 lists each claim and provides examples of contradictory evidence that should be considered when evaluating these general claims.
Discussion
The aim of the analysis of Cologon (Reference Cologon2019) was to establish the characteristics of the sources used in advocating for full inclusion for students with disabilities and provide a basis for a critique of the evidence used to support the claims for full inclusion for all students. The implications of this analysis may go beyond this single advocacy paper. If other advocacy papers are similarly flawed, misrepresenting the research literature and making claims that appear to be unsupported, policymakers cannot use them for decision-making without further scrutiny of the relevant research. If policymakers and others use Cologon, and papers like it, as sources of evidence about the outcomes of inclusion, they need to consider how accurately they reflect the research evidence available.
In considering the certainty of evidence provided by this paper, we have analysed in some detail a sample of the sources used in the chapter relating to the outcomes of inclusion, as well as reviewing the sources used in the paper as a whole. Although it is claimed that research supports full inclusion, research articles (quantitative, qualitative, reviews, or meta-analyses) comprised little more than half of all sources cited for the whole paper. The percentage of research papers was higher for Chapter 2 (just over 65%), but just over 30% of these articles reported qualitative research, which cannot give evidence of cause-and-effect relationships. Similarly, there was only one quantitative research paper among the most cited sources, and the most frequently cited source (Hehir et al., Reference Hehir, Grindal, Freeman, Lamoreau, Borquaye and Burke2016) was an unrefereed agency report that specifically set out to summarise research that ‘demonstrates the benefits of inclusion’ (Hehir et al., Reference Hehir, Grindal, Freeman, Lamoreau, Borquaye and Burke2016, p. 2). The heavy reliance on opinion/discussion articles and unrefereed sources and the small number of research articles including students with disabilities in inclusive settings as participants is a concern when claims about outcomes are purported to be based on evidence.
Cologon (Reference Cologon2019) is described as ‘an extensive systematic literature review’ that ‘addresses the outcomes of inclusive education’ (p. 4). No information about the search procedure, nor about the criteria for including sources in the review, is provided, and no further information could be obtained from the author or CYDA. A systematic review is usually considered as one that is conducted in a methodical way to reduce bias in the selection of articles to appraise. As such, it will include a search strategy that is described in sufficient detail to be reproducible, a predefined set of eligibility criteria, and some assessment of the quality of research reviewed (Hanley & Cutts, Reference Hanley and Cutts2013). Tools to improve the quality of systematic reviews are available, such as the PRISMA checklists (Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2009) and the handbook for systematic reviews from the Cochrane Collaboration (Higgins & Green, 2008). One would expect that a methodical search would have turned up some of the studies illustrating positive outcomes in specialist settings, and some of the less positive findings on inclusion. It seems that little research reporting negative results, apart from Gottfried (Reference Gottfried2014) and Fletcher (Reference Fletcher2010) on the impact of students with emotional or behavioural disorders, was included. Discussion and opinion papers by vocal critics of full inclusion, such as James Kauffman, were absent, as were papers suggesting alternative views on rights-based arguments (e.g., Farrell, Reference Farrell2000).
When we looked more closely at a random sample of the sources used in Chapter 2, we found that only seven of the sources could provide definitive evidence for the claim made. There was a heavy reliance on discussion papers, which by their nature do not provide direct evidence, only the authors’ opinions, and on non-refereed sources, which may not be of a high standard, or, like Hehir et al. (Reference Hehir, Grindal, Freeman, Lamoreau, Borquaye and Burke2016), offer only supportive evidence. Of most concern, we judged that 27 sources did not support the claims made for various reasons, such as not providing any empirical comparison data when claims were made that inclusive settings were superior or the article simply not addressing the claim made.
The relevance of the sources used for providing evidence about the outcomes of inclusion is also problematic. Claims about the relative benefits of inclusive compared to specialised settings must be based on studies that have compared outcomes for similar students in both settings, but there were very few sources proving this kind of data. None of the most cited sources and only four of the sources in the Chapter 2 sample included comparisons. Indeed, only 11 of the 29 research papers used in the sample from Chapter 2 included students with disabilities as participants. Göransson and Nilholm (Reference Göransson and Nilholm2014) call this ‘tumble-weed referencing’ (p. 276), where research is used as though reliable evidence on the benefits of inclusion exists, but in actual fact the evidence is weak or non-existent.
When making blanket claims about the overall effectiveness of inclusion for all students, it is also important to look at the evidence in terms of the groups and contexts being examined. It may not be appropriate to generalise findings from country to country, from one disability group to another, or from setting to setting. The paper consistently cites sources describing specific students and contexts, with a preponderance of research from United States elementary schools, as providing generic evidence in favour of inclusion, but extreme caution must be used in using research findings in this way.
It is also important to consider the frequently noted problems with research on inclusion. Reviewers have found that definitions of inclusion vary considerably (Nilholm & Göransson, Reference Nilholm and Göransson2017), and the definitions of inclusion used in the 18 quantitative papers selected from Chapter 2 illustrate this variation. Ten papers (55%) gave no definition at all and others gave relatively vague definitions, such as more than 50% or more than 80% of the school day in mainstream classes. The Szumski et al. (Reference Szumski, Smogorzewska and Karwowski2017) meta-analysis cited in the paper does not provide a definition of inclusion in the criteria given for selecting papers. Another aspect of the definition of inclusion is the nature of additional services and supports provided, and although these were not well described in the sample sources that included students with disabilities as participants, there were few studies where students did not receive support from a special educator or a paraprofessional within the mainstream class.
Cologon (Reference Cologon2019) does address the limitations of the existing research and concludes that ‘even poorly-done [sic] inclusion in the form of integration is still better [than specialised settings]’ (p. 13). An alternative, and perhaps better explanation, given that few sources compared outcomes for students who were fully included at all times and who received no special education services with those in specialised settings, may be that students with disabilities do better when they are provided with access to both mainstream classes and to separate specialised settings and/or instruction. The support offered by several authors for inclusion within multi-tiered or response-to-intervention approaches to instruction is based on the evidence that students with disabilities may need intensive instruction delivered separately to regular class instruction (Fuchs & Fuchs, Reference Fuchs and Fuchs2015). Others are of the view that the intensive and explicit instruction needed by some students with disabilities cannot be provided within regular classrooms (Kauffman et al., Reference Kauffman, Travers and Badar2020), and certainly there are calls for more research to establish effective tertiary practices, particularly for students with more severe disabilities (Snell, Reference Snell2008). A huge research base on effective instruction for students with disabilities that has come from the special education research exists already (McLesky et al., Reference McLeskey, Barringer, Billingsley, Brownell, Jackson, Kennedy, Lewis, Maheady, Rodriguez, Scheeler, Winn and Ziegler2017), and this literature has been almost entirely ignored in the paper, as have multi-tiered approaches and, indeed, separate small group or individual instruction is classified as micro-exclusion.
Finally, perhaps the reported experiences of students with disabilities themselves do not always endorse full inclusion. Nugent (Reference Nugent2008) interviewed children with dyslexia who were in specialised and mainstream settings and their parents. All children were generally positive about their experiences, but those in specialised settings were happier and found the work easier. The author notes the importance of being with ‘like’ others, and specialised settings may provide benefits for friendships. Olsson et al. (Reference Olsson, Dag and Kullberg2018) surveyed over 7,000 adolescents with deafness or hearing impairment and found those in specialised settings were more satisfied and felt more included than those in mainstream schools. Rathman et al. (Reference Rathmann, Vockert, Bilz, Gebhardt and Hurrelmann2018) surveyed over 5,000 German students and found that students with disabilities attending mainstream settings were more likely to report low life satisfaction that those in specialised settings.
At a broader level, our analysis of Cologon (Reference Cologon2019) may provide some guidance for readers of advocacy documents and policymakers who need to evaluate information about the research evidence presented by advocates of full inclusion. Readers should look for evidence that the review was systematic and followed accepted guidelines for searching for and selecting sources (Higgins & Green, 2008; Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2009). A systematic review should include studies that reveal benefits from specialised settings and less favourable outcomes from inclusive settings. The nature of the sources used needs scrutiny; many authors express opinions in favour of full inclusion, but these opinions are not research evidence and should not be treated as such. Given the lack of quality comparative research, readers should be wary of claims that one setting produces better results, especially when details about the contexts and students compared are not presented. Readers should also be wary of sweeping claims about benefits for all based on studies of a particular age group, in a particular country, or in a particular context, especially when it seems very few studies involve students who are fully included without some kind of individual or small group support outside regular classes. Finally, readers should consider what cited papers mean by inclusion. We have noted the lack of consensus on definitions of inclusion, and not all students who are described as being included are fully included all the time without any individual or small group supports.
We believe the overriding right of children with disability is to receive an education that meets their needs and leads to an adult life that is as independent as possible. For some students, their needs are more likely to be met if all or part of their education is in specialist settings rather than restricted to what is possible in a mainstream classroom (Calhoon et al., Reference Calhoon, Berkeley and Scanlon2019). There is no doubt there is research that demonstrates that inclusion can produce benefits for many students with disabilities, but several reviewers are critical of the quality and generalisability of this research given the lack of agreed definitions of inclusion (Dyson, Reference Dyson2014; Lindsay, Reference Lindsay2007; Shaw, Reference Shaw2017). Research that demonstrates positive effects for specialist provisions (Fuchs & Fuchs, Reference Fuchs and Fuchs2015) also needs to be acknowledged. Any evaluation of the desirability of moving to a full inclusion model of school education must consider the whole spectrum of research. It is important to identify what works as well as what does not work if all students are to receive the best education. Those who make policy decisions relating to education and those, such as parents and teachers, who are the targets of advocacy papers such as Cologon (Reference Cologon2019) must be confident that such papers fully represent the research available.
