%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
The Venezuelan National System of Youth and Children's Orchestras, better known as El Sistema, needs little introduction. Founded in 1975, it has become one of the largest and best-known music education programmes in the world. Over the last decade, its growing fame in the global North and the proliferation of ‘Sistema-inspired’ programmes in dozens of countries have led to an array of publications, which may be divided into three broad categories: (1) advocacy literature (e.g. Borzacchini, Reference BORZACCHINI2010; Tunstall, Reference TUNSTALL2012; Tunstall and Booth, Reference TUNSTALL and BOOTH2016); (2) critical academic studies (e.g. Logan, Reference LOGAN, McNeish, Borchgrevink and Logan2015a; Pedroza, Reference PEDROZA2015; Baker, Reference BAKER2016a, Reference BAKER, Elliott, Bowman and Silverman2016b); and (3) commissioned evaluations (e.g. GEN, 2011; Glasgow Centre for Population Health, 2015).
The critical academic studies have tended to focus primarily on ideological questions such as class, neoliberalism, and (neo)colonialism, scrutinising the programme from political, ethical, and historical perspectives (e.g. Borchert, Reference BORCHERT2012; Bull, Reference BULL2016; Fink, Reference FINK2016; Logan, Reference LOGAN2016; Rosabal–Coto, Reference ROSABAL–COTO2016), though some studies have combined such critique with ethnographic research (e.g. Baker, Reference BAKER2014; Dobson, Reference DOBSON2016). Commissioned evaluations, in contrast, have concentrated on more pragmatic and limited questions, which might be summarised as: does the programme work for current participants, and if so, in what ways? Surveying these two bodies of literature reveals a striking polarisation between the largely negative responses of critical scholars and the almost entirely positive conclusions of published evaluations.
This polarised scenario is the starting-point for our study. We ask: how might we account for the gulf between the findings of commissioned evaluations and critical writings (including published scholarship and blogs)?Footnote 1 We explore this question via two principal case studies, which involve detailed examination of evaluations of El Sistema in Venezuela and Aotearoa New Zealand, preceded by a brief historical introduction to the topic of evaluating this programme.
The history of evaluations of El Sistema in Venezuela
Divided opinions among researchers of El Sistema date back to the first attempts to evaluate the programme in 1996 to 1997, which were catalysed by El Sistema's efforts to secure funding from the Inter-American Development Bank (IDB). Four evaluations were produced by external consultants in this two-year period. The programme's efforts were successful: the IDB provided a Phase I loan of $8 million in 1998, and a Phase II loan of $150 million in 2008 – two of the most decisive developments in the history of El Sistema.
The first two reports, from 1996, were marked not only by a reverent tone but also by a striking lack of critical scrutiny or robust evidence of the supposed social benefits (see Baker & Frega, Reference BAKER and FREGA2018). Rather than analysing the official narrative, they adopted it, emphasising the spiritual richness provided by music and its supposed capacity to overcome material poverty. Such was the advocacy tone that a segment of the report now forms part of El Sistema's official vision statement.Footnote 2 However, it appears that the IDB was not satisfied; it hired two more consultants and repeated the process the following year. The second pair of consultants discovered numerous problems with the programme, which they documented in detail, revealing that the earlier evaluations had either missed or omitted many important issues and drawn dubious conclusions. Nevertheless, the IDB granted the $8 million loan, and the critical reports from 1997 were never made public.
The question marks that hovered over the robustness of the first two evaluations were not dispelled by subsequent studies. The next evaluation was carried out by the Universidad de los Andes in Mérida between 1999 and 2003. This quantitative study, too, reveals numerous flaws (Baker, Reference BAKER2014). As Hollinger notes (Reference HOLLINGER2006, pp. 41–42), it has ‘a number of inherent design weaknesses’ and resembles ‘less a scholarly endeavor than necessary documentation to advocate for The System.’ As with the 1996 evaluations, the researchers adopted El Sistema's proselytising tone and in effect assumed an advocacy position.
A new evaluation by José Cuesta (Reference CUESTA2011) was used to justify the IDB's Phase II loan of $150 million. Yet it presented evidence of correlation rather than causation; the use of the terms ‘treatment’ and ‘control’ was misleading; it did not consider pre-existing cognitive or social differences between children; and El Sistema's leaders appeared to have played a part in creating the report (Baker, Reference BAKER2014). Furthermore, the financial calculations were questionable (Scruggs, Reference SCRUGGS2015).Footnote 3 There are thus numerous reasons to doubt the study's speculative yet much-cited conclusion, a cost-benefit ratio of 1:1.68.
In fact, the IDB distanced itself from Cuesta's report. In 2011, the bank published a proposal for a new impact evaluation that would supposedly provide ‘the first rigorous evidence of the results of the programme’ (Sistema Nacional, 2011, p. 3). It also admitted that Cuesta's cost-benefit analysis ‘was the result of various suppositions and not of a rigorous measurement of the impact of El Sistema’ (p. 2). (Nevertheless, it had already agreed the loan by this point.) This proposal gave rise to a large-scale experimental study, which was intended to finally settle the longstanding, unresolved question over the efficacy of El Sistema and put to rest the history of flawed evaluations, divided opinions, and changing views.
The 2016 IDB report: starting-points and conclusions
The new research was proposed in 2011, carried out in 2012–13, and first reported in 2016 (Alemán et al., Reference ALEMAN2016). The researchers created a ‘theory of change’ which hypothesised that ‘short-term participation in orchestras or choruses may foster positive change in four child functioning domains: self-regulatory skills, behaviour, prosocial skills and connections, and cognitive skills.’ To test their theory, they measured 26 primary outcome variables within these four domains. Only two significant outcomes (at the 90% level) were found: ‘the early-admission group had higher self-control and fewer behavioral difficulties, based on child reports.’ There were thus no significant outcomes in 24 out of 26 areas, and the researchers ‘did not find any full-sample effects on cognitive skills [. . .] or on prosocial skills and connections.’
Perhaps more strikingly, the estimated poverty rate among the El Sistema entrants was 16.7%, while the rate for the states in which they lived was 46.5%. In other words, the El Sistema participants in the experiment were three times less likely to be poor than all six- to 14-year-olds residing in the same states. Consequently, the study ‘highlights the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program.’ Furthermore, 44% of students who were offered a place failed to complete two semesters. The study thus found little evidence to support the theory of change, but did find two statistics that raised doubts about the official narrative of El Sistema as a programme aimed primarily at, and with transformative effects on, the poor.
Looking at the genesis of this study is also revealing. A news article on the IDB's website, announcing the decision to undertake the research, opens with the words: ‘When the first orchestra for young people from low-income families from the most deprived neighborhoods in Caracas was founded back in 1975 . . . ’Footnote 4 In fact, the social composition of the first orchestra was predominantly middle-class, and most participants were conservatoire students (Baker, Reference BAKER2014). Reproducing a myth is hardly a promising start. The article continues with another myth, describing the programme as ‘giving priority to those from the lowest socio-economic levels’; yet El Sistema has no systematic targeting mechanisms, affirmative action policies, or quotas – hence the low poverty rate discovered by the IDB's subsequent research. These are surprisingly misinformed statements about fundamental aspects of the programme, considering that they come from its major non-state funder, and they suggest that the IDB's starting-point was closer to the advocacy literature than to scholarly research.
The IDB's formal proposal document details the goals of the study: ‘The expected impacts of El Sistema include the development of social skills and self-esteem, a reduction in the school dropout rate, particularly in secondary school, a reduction in the incidence of risky behaviours, and reduced frequency of unplanned pregnancies’ (Sistema Nacional, 2011, p. 2). The second to fourth categories are emphasised repeatedly: on the following page, the objectives of the study are stated as ‘to seek to generate rigorous evidence of the social effects of [. . .] El Sistema, including the impacts on school dropout, illegal behaviour, and unplanned pregnancies’ (p. 3). Under ‘expected results,’ we read: ‘The data will be used to evaluate rigorously the impacts of El Sistema on school dropout, risky behaviours, incidence of crime, and prevalence of unplanned pregnancies’ (ibid.).
However, the study itself, published five years later, does not discuss rates of school dropout, crime, or unplanned pregnancies, nor does it mention this major shift in the evaluation's targets and aims. This omission raises significant questions. What happened to the issues identified as central in the proposal? When and why did they disappear?
The 2016 IDB report: methods
At this point, we move to a critical analysis of the 2016 IDB report itself. The study makes a number of claims about the effects of El Sistema using statistical language; however, these claims fall short of the standards that would ordinarily be used for the conclusions drawn. There are four key ways in which the report is deficient: preregistration, the use of the 90% significance level, p-hacking, and subgroup analysis. Each of these issues is sufficient to raise questions about the validity of the report; in combination, they undermine the analysis entirely.
The study was registered at clinicaltrials.gov. However, its registration falls short of best practice in two ways. The first is that the registration on clinicaltrials.gov was first received in 2015, and last verified in 2016. Given that the data was collected during 2012 and 2013, this is not preregistration, which is considered best practice in medical research (see e.g. de Angelis et al., Reference DE ANGELIS2004). The second issue is that the analysis is not specified: while outcome measures are listed, detail is not provided, and measures are specified in five groups rather than as a number of individual items, while the analytical technique is absent. Consequently, it is unclear whether the key issues in the proposal document were discarded before the data were collected or afterwards (perhaps because no significant effects were found). Also missing from the registration was any subgroup analysis: the only specified arms of the study were the treatment and control groups, with no suggestion that differences would be identified within any smaller groups (see below).
In the paper itself, instead of comparing five composite measures (as implied in the trial registration), the authors compare the treatment and control groups across 26 different outcome measures, using 90% as a threshold for statistical significance. Opting for 90% is highly unusual. The overwhelming majority of papers that use significance testing employ the 95% significance level, and this itself has been frequently criticised as over-generous, with 99% or 99.9% being more appropriate for robust results. A single study with results significant at the 95% level would not ordinarily be sufficient to justify a policy intervention or a change in prescribing policy; it is highly unusual that a large organisation such as the IDB would use the weaker 90% threshold without a clear justification.
Differences that are addressed in the conclusion are first seen in the ‘Impacts’ section. The authors find differences between the treatment and control group in two out of 26 measures. If the authors were using no adjustment, given the large number of outcomes measured, and there were no genuine underlying differences, the probability of at least one difference significant at the 95% level is 74%: while this may seem unusually high, it can be understood by comparing the cumulative probability of each and every difference being nonsignificant. The probability of at least one difference significant at the 90% level is 92%. Therefore, the authors’ discovery of differences between young people participating in El Sistema is almost trivial.
However, the authors acknowledge the large amount of hypothesis-testing, which would ordinarily yield significant differences through sheer luck, and report that they control the k-familywise error rate – that is, the probability of observing a given number (k) of false positives, in the event that any true differences observed in the data were due to random noise – with an adjusted version of the Romano–Wolf procedure, a technique to adjust thresholds for significance testing given the number of comparisons being drawn simultaneously. The adjustment is that instead of using k =1, which they find too conservative, they use k=h/2 (where h = the number of variables in a domain). While the authors they cite (Delattre & Roquain, Reference DELATTRE and ROQUAIN2015) do demonstrate that the Romano–Wolf procedure is conservative, they do not demonstrate that k=h/2 is an appropriate adjustment. In addition, the measures in which they investigate differences are likely to be correlated, so while the authors are to be commended on adjusting their tests to acknowledge the numbers of tests they have conducted, their adjustment is far too generous, implying levels of significance unlikely to be upheld by their results. That the results they do yield are only significant at the 90% level (and only two out of 26 tests at that) does not provide support for the alternative hypothesis that there are differences between the students participating in El Sistema for a year and those who waited a year.
These differences between those participating in El Sistema and the apparent control group are not the only differences addressed in the conclusion. The authors also compare these groups within subgroups. These are broken down as follows. First, they compare whether students’ mothers have any college education or not; second, they compare students aged between six and nine, and between ten and 14; third, they compare the interaction between gender and exposure to violence (that is, comparing boys who have been exposed to violence with those who have not, with girls who have, and with girls who have not). With 8 different groups compared across 26 different measures, this leads to a total of 208 comparisons between treatment and control groups. The paper only reports coefficients that are significant at the 90% level, of which there are 13.
According to the experimental literature, subgroup analysis ‘can lead to overstated and misleading results’ (Wang et al., Reference WANG2007, p. 2189), and this is particularly the case in the event of multiple subgroups being analysed, a problem known as multiplicity. Here, there are eight different subgroups being analysed. While the experimental literature strongly advises that any subgroup analysis be registered before data is collected and analysed, this is not sufficient to solve the multiplicity problem where – again – significant results are likely to appear through sheer luck. In this case, there is no reference to subgroup analysis in the trial registration. A critical reader might wonder how many other subgroups have been analysed and discarded before these eight were settled on. Why analyse together boys and girls whose mothers are more educated, but separately boys and girls who have been exposed to violence? At least, as the authors are really testing an interaction effect here, any differences identified should be net of the direct effect: instead of comparing boys and girls who have been exposed to violence, it should be clear what the effect of exposure to violence on each outcome measure is. However, the direct effect of exposure to violence is not reported. Given the absence of preregistration, it seems likely that all possible subgroup analyses were conducted; it is not computationally intensive and there is no clear reason for choosing these subgroups rather than others. This is a technique known as p-hacking (see e.g. Head et al., Reference HEAD, HOLMAN, LANFEAR and JENNIONS2015; Bruns & Ioannidis, Reference BRUNS and IOANNIDIS2016). It is therefore probable that the use of the k-familywise error is insufficiently rigorous, and the apparent significant differences represent random noise.
These issues, taken together, make the report almost impossible to take seriously. The trial was not preregistered; the threshold to which the authors ascribe significance is half as demanding as the academic mainstream, yet one that they mostly fail to reach; and the number of analyses run implies that apparently significant results are likely to be the outcome of dumb luck. At the very least, the study's conclusions that ‘exposure to El Sistema might serve an important role as a preventive strategy to promote positive outcomes among disadvantaged children’ and ‘El Sistema is particularly effective for vulnerable males’ are notably overstated. But we would go further and argue that the data display no convincing evidence of El Sistema’s efficacy, and that the two statistically significant results were generated only by using an unusually generous threshold and a combination of questionable methods. Producing these results allowed the IDB and El Sistema to save face – to have found nothing after 18 years of investment and five years of research would have been embarrassing for both – and to present both the study and the program itself as a success.
Finally, there is a potential generalisation problem given the application process. Even if one were to accept that El Sistema had been proven to have beneficial effects on participants, the conclusions do not take account of the fact that all participants were signed up for the programme by a parent or guardian, who thus showed a certain level of commitment to the child's education. It cannot be assumed that effects on children from more supportive families will be mirrored in all children; there is no evidence that the study has external validity. It may be that parental support is a key ingredient in generating positive effects – something suggested by studies of El Sistema and other after-school programmes (Pérez & Rojas, Reference PEREZ and ROJAS2013; Baker, Reference BAKER2014; Cid, Reference CID2014). This would limit the wider applicability of El Sistema as a social inclusion programme, and if we also take into account the IDB's findings about poverty and dropout rates, El Sistema may in fact be quite ineffective in promoting positive change in the most disadvantaged sectors of society.
Evaluating Sistema Aotearoa
Evaluations of non-Venezuelan Sistema programmes have generally been assumed to be robust, and having been cited frequently in the media, they play an important role in advocacy for the global Sistema movement. However, Owen Logan's (Reference LOGAN2015b) scathing assessment of two evaluations of Sistema Scotland, followed up by Baker (Reference BAKER2017), suggests that more critical scrutiny would be worthwhile. An evaluation of Sistema Aotearoa, a government-funded programme that began in April 2011 in Auckland, New Zealand, is the focus of our second case study. The Ministry for Culture and Heritage and Auckland Philharmonia Orchestra, who jointly run Sistema Aotearoa, commissioned the Kinnect Group, a private sector evaluation consultancy based in Aotearoa New Zealand, to evaluate this programme. They produced an initial report in 2012 and an ‘outcome evaluation’ three years later (McKegg et al., Reference McKEGG, CROCKET, GOODWIN and SAUNI2015).
The evaluation draws on two sources: quantitative educational achievement data and qualitative ‘success case studies’ from participants and their families. The quantitative aspect of the evaluation draws on data from ‘overall teacher judgements’ (OTJs), which are part of the national standards for statutory education that were introduced in 2012 (Thrupp, Reference THRUPP2014). This policy has provoked numerous concerns, for example that using teachers’ judgements of pupils may exacerbate existing inequalities (see Thrupp & Easter, Reference THRUPP and EASTER2012; Thrupp & White, Reference THRUPP and WHITE2013). Additionally, the report is frank about further limitations of this data. These include the lack of baseline data from when the programme began; lack of data for three out of the seven Sistema Aotearoa schools; lack of data about children who dropped out of the programme; and use of aggregate data at the level of the school rather than individual pupils’ data. These issues, as well as the small size of the data sets, mean that any conclusions drawn from this data are partial and very limited. For the two years of data that are available, the study finds a statistically significant improvement in reading and maths achievement, but notes that ‘it is possible that the difference we have identified is because the higher achieving students are more likely to stay engaged with the programme’ (McKegg et al., Reference McKEGG, CROCKET, GOODWIN and SAUNI2015, p. 17). Given a dropout rate of 47%, this is an important caveat.
Since only very tentative conclusions can be drawn from the quantitative data, the qualitative element of the study is very important. However, the ‘success case’ methodology that is used is problematic. This approach is not a well-known social science methodology, and is not mentioned in the most reputable book on case study methods, by Robert K. Yin (Reference YIN2013). Nor does it appear in other relevant textbooks (e.g. Gerring, Reference GERRING2006; Woodside, Reference WOODSIDE2010). Stufflebeam and Coryn (Reference STUFFLEBEAM and CORYN2014) note that it has certain strengths, such as identifying what works well, reassuring funders, and boosting morale, and it is also quick and cheap. However, it has obvious weaknesses: it is narrow, short-term in outlook, suffers seriously from selection bias, and does not present ‘a comprehensive assessment of a programme's merit and worth’ (ibid., p. 143) (see Baker, Reference BAKER2016c).
For the Sistema Aotearoa evaluation, this method involved recruiting five pupils who were identified by teachers as being particularly successful in the programme and carrying out interviews with them and their families, as well as with Sistema staff. The reason given for adopting the ‘success case’ methodology is cultural sensitivity. The programme involves a high number of Pacific Island families, and the report argues that ‘it is considered impolite in Pacific cultures to talk in negative ways about a service or programme that is being received,’ which therefore suggests that a ‘success case’ approach will ‘generate responses that are both culturally valid and more accurate’ (McKegg et al., Reference McKEGG, CROCKET, GOODWIN and SAUNI2015, pp. 9–10). This is indeed a methodological hurdle, but one that is insufficient to justify such a partial approach. More imaginative methodological choices, such as Rimmer, Street and Phillips’ (Reference RIMMER, STREET and PHILLIPS2014) creative methods with children, could have overcome this issue.
This ‘success case methodology’ means that evidence within the data of less positive outcomes of the programme is not discussed. These include the dropout rate of 47%; the major intervention in family life that Sistema Aotearoa requires; the experience of stigmatisation described by some parents in the programme when attending prestigious concert venues; and signs that the programme adopts a deficit model of culture. To briefly discuss the last of these, the report notes that ‘orchestral music typically falls outside of what these children [. . .] see as “their culture”’ but also that through participation in Sistema Aotearoa, children and their families ‘are all gaining an appreciation for Western orchestral musical culture that they otherwise would not have had the opportunity to do’ (ibid., pp. 31–32). However, the report recommends that the programme outcomes acknowledge that Māori and Pacific Islanders are not simply recipients of knowledge from the programme, but already have existing knowledge and practices. In fact, public and media discussions in New Zealand have been more critical than the evaluation, showing concern that Sistema Aotearoa may be devaluing Māori and Pacific Island culture (Trinik, Reference TRINIK2014, p. 14; McPhail et al., Reference McPHAIL, RATA, SITEINE, McPhail and Thorpe2018, p. 4). These discussions reflect an awareness that Aotearoa New Zealand is still deeply shaped by its history of colonisation. Stark divides continue to exist in health, education, and social outcomes between Pākehā (white New Zealanders) and Māori and Pacific Islanders (Statistics New Zealand, 2016). Therefore a cultural education programme in which colonised people learn to ‘gain an appreciation’ for European culture requires contextualisation within a wider critical discussion, which the ‘success case’ methodology adopted in this evaluation does not allow.
In sum, while all social data is necessarily partial, the data in this study is particularly limited. Most problematic is the presentation of the report as an ‘outcome evaluation’ when it should more accurately be described as a collection of accounts from a handful of participants who enjoyed the programme. Missing are critical discussions of negative and null outcomes that inevitably occur as well, and of the wider context of class and race inequality in which Sistema Aotearoa operates, in particular how an education programme that teaches European high culture to Māori and Pacific Islanders should be understood in the context of Aotearoa New Zealand's colonial legacy. This is where academic research diverges from commissioned evaluations. Indeed, Aotearoa New Zealand academics lead the world in ‘decolonising methodologies’, examining ways of re-thinking how to do research with colonised groups (Tuhiwai Smith, Reference TUHIWAI SMITH1999). Proper assessment of a programme that includes in its aims an explicit transfer of the knowledge and culture of white Europeans towards groups who have been colonised by them necessitates a critical examination of this context.
The uses and abuses of Sistema evaluations
The flaws and limitations that have been found in evaluations of El Sistema, both in Venezuela and elsewhere, give cause for concern. They raise doubts about both the efficacy of El Sistema and the processes of evaluating such programmes. These concerns and doubts are only amplified by considering the post-publication trajectory of two of these studies.
Cuesta's (Reference CUESTA2011) principal conclusion, a cost-benefit ratio of 1:1.68, quickly became a mainstay of advocacy arguments in favour of El Sistema and Sistema-inspired programmes. It exemplifies Belfiore's (Reference BELFIORE2016, p. 212) statement, drawing on Max Singer's article ‘The vitality of mythical numbers,’ that ‘once a statistic is produced (no matter whether rigorously or incorrectly) and starts being quoted, it takes on a life of its own. As a result, the imaginary statistics might enter the official debate on cultural policy, being quoted for years without its original source and its reliability ever being verified.’ Four years later, the IDB very quietly recognised the speculative nature of Cuesta's conclusion, and more thorough academic critiques followed.Footnote 5 However, by this point the figure's work was already done: it had underpinned both the IDB's decision to issue a $150 million loan and advocacy arguments for establishing Sistema-inspired programmes around the world. The story of Cuesta's report illustrates that headline numbers may garner far more attention than the detail of the studies that support them, even with such large sums at stake, and that questions may come too late to make any difference and/or be ignored by interested parties (for example, Tunstall and Booth [Reference TUNSTALL and BOOTH2016, p. 228] employ this figure despite knowing that it had been criticised).
Even more striking has been the official dissemination of the 2016 report. Although the conclusions of this study were overstated in relation to the findings, the authors did signal two important negative findings – the low poverty rate and high dropout rate – and were open about having found no significant outcomes in 24 out of 26 areas, concluding: ‘We did not find any full-sample effects on cognitive skills [. . .] or on prosocial skills and connections.’
The IDB's blog post on the report, though, gave it a much more positive spin.Footnote 6 It mentioned none of the negative or equivocal findings, only the positives. It even made positive claims about gender equality, whereas the study itself had found the opposite. The misleading impression given by the blog post was that the study provided an unequivocal stamp of approval for El Sistema.
A launch event for the study in Caracas in March 2017, at which the report's authors, El Sistema leaders, and government representatives were present, continued in this vein. The press release declared that the research team ‘expressed its satisfaction with the possibility of confirming the transformative work of the programme.’Footnote 7 The researchers had concluded, it claimed, that the children and young people in El Sistema showed improved connections with school and family, a higher degree of cooperation with their peers, and greater self-confidence. According to one, Marco Stampire, ‘we found a decrease in levels of aggression and risk-taking [. . .]; and a willingness to take part in collective activities. The positive effects were also manifested in childhood IQ.’ These claims contradicted the evidence and conclusions presented by the same researchers in their published article, in which they had stated that they had not found any full-sample effects on cognitive skills or prosocial skills and connections.
Ferdinando Regalía, head of the IDB's Social Protection and Health Division, stated that the results ‘tackle the criticisms of El Sistema's work and reaffirm the value of social inclusion via a programme of artistic and musical education.’ Yet the findings about the poverty and dropout rates did not tackle such criticisms but rather confirmed their validity.
On the basis of the press release, it is impossible to be sure whether the IDB research team overstated the positives and omitted the negatives from its public presentation, or whether El Sistema's press office was responsible. Either way, there are reasons to be concerned about the way this study is being used. The press release suggested that the official line from the institutions involved was that the study proved El Sistema to be a success, and this impression was confirmed by subsequent publications such as a graphic summarising the study, released by the IDB, and an interview with El Sistema's executive director, Eduardo Méndez, both of which gave the findings an entirely favourable spin.Footnote 8 If the positive conclusions of the report were overstated in the first place, they were subsequently exaggerated further in its public presentation and shorn of important caveats.
The institutions involved all have good reason to portray the findings in the best possible light. The IDB and the Venezuelan government have invested hundreds of millions of dollars in El Sistema over a period of many years, and the unvarnished findings of the report provide little justification for this expenditure. The pressure on the researchers must therefore have been intense, which might explain the generous statistical approach used. But the public presentation of the report raises serious questions about the value of investing significant resources over a period of years in evaluating a large programme with major, longstanding support from politicians and multinational institutions. Similarly, the Sistema Aotearoa evaluation is designed in such away as to avoid bringing to light any problems or criticisms associated with the programme. Furthermore, in a country where recognition of Māori culture is enshrined in law (New Zealand Law Commission, 2001), it is highly unusual that the evaluation failed to discuss the effects of the programme on the cultural values of its predominantly Māori and Pacific Island participants. Both the Sistema Aotearoa and the IDB reports, therefore, and above all the associated publicity for the latter, appear more like efforts to justify investment and continuity than serious attempts to identify strengths and weaknesses and to improve the programmes accordingly.
Conclusion
We began our study with the problem of the marked divide within the Sistema research literature. There are a number of possible explanations for this scenario, including the disciplinary backgrounds and ‘homes’ of the researchers, and the organisations for which they work (for example, universities or consultancies). Also, one obvious reason that the two sub-fields produce quite different answers is that they ask quite different questions. Whereas evaluators tend to examine whether programmes achieve their goals, independent researchers are much more likely to interrogate the validity of those goals and consider cultural, political, or philosophical questions that they raise, drawing on academic fields such as music studies, sociology, and critical theory.Footnote 9
While this distinction is important, and indeed worthy of a separate study, here we have sought to shed light on our central question by shifting the focus of critical enquiry away from El Sistema and its spinoffs and towards the evaluations of these programmes. We conclude that the gulf within the literature may relate, at least in part, to flaws in the processes of some evaluations, which lead them to present an overly optimistic picture. Through our critique of these two studies, we suggest that the research foundation for El Sistema's claims of social transformation, and hence for its fame and international proliferation, may be weaker than first appears and therefore require further examination. (Indeed, a striking aspect of El Sistema's history is that repeated investigations in Venezuela over a period of 20 years have failed to generate robust evidence of its efficacy.) We are not making an a priori claim that all evaluations are flawed or that independent academic research is necessarily superior, but rather suggesting that evaluations (like all research) deserve careful scrutiny, and that when looking for explanations for gaps between the conclusions of evaluations and academic studies, one route to explore is critical revision of evaluative methodologies.
As a further step, it is illuminating to consider Eleonora Belfiore's call for a ‘critical research ethos,’ which she defines as
research that is disinterested, that is, indifferent to the requirements of advocacy – advocacy being a fully legitimate enterprise, but one completely distinct and, ideally, separate from genuinely explorative research. By ‘explorative’ research, I refer to a type of research that aims to describe, explore and illuminate complex issues around the role and condition of culture, cultural production, consumption and administration in contemporary society. (Belfiore, Reference BELFIORE2009, p. 354)
We suggest that the flaws in some Sistema evaluations are linked to the fact that many such studies – the 1997 reports on the Venezuelan programme being an important exception – display an alignment with advocacy rather than explorative research, and are examples of what Logan (Reference LOGAN2015b) calls ‘Sistema-friendly research’. This alignment takes a variety of forms, which include overplaying positive findings, underplaying negative ones, side-lining problematic issues, omitting reference to the critical research literature on El Sistema specifically and music education more generally, and/or adopting rather than scrutinising the programme's official rhetoric and proselytising tone. These features are characteristic of the advocacy literature referenced at the start of the article. There is thus a reproduction within the research field of the division between independent academic studies and non-academic advocacy writing.
Our primary aim has been to shed critical light on the role of evaluations in reinforcing, rather than genuinely testing, the excessively optimistic dominant narrative about El Sistema. However, we also hope that our research will make a contribution to debates within the field of cultural policy studies, which has been raising questions about evaluations of the social impact of the arts for a number of years (e.g. Belfiore, Reference BELFIORE2002, Reference BELFIORE2009; Merli, Reference MERLI2002; Selwood, Reference SELWOOD2002; Belfiore & Bennett, Reference BELFIORE and BENNETT2010; Johanson & Glow, Reference JOHANSON and GLOW2015; Lees & Melhuish, Reference LEES and MELHUISH2015). Finally, we also suggest that our study should encourage critical debate on the topic of evaluations in music education, such as reflection on what constitutes robust forms of evidence, the relationship between evaluation, advocacy, and explorative research, and ways in which evaluation data can be misrepresented and misreported. We hope to have shown that programme evaluations are a valid and worthwhile object of critical research. Yet, ideally, this debate should include but also go beyond the sort of scrutiny of existing evaluations that we have undertaken here and penetrate the commissioning and evaluation processes themselves, since post-hoc scrutiny is unlikely to affect policy and funding decisions.
We would also suggest that evaluations should be seen as political tools, in the sense that they frequently operate within challenging funding contexts in which there are losers as well as winners. As Bull (Reference BULL2016, p. 140) notes, In Harmony El Sistema England received considerable investment at a time when music education funding generally was being cut by nearly a third in England, and Sistema Scotland has flourished against a similarly concerning backdrop (Baker, Reference BAKER2017). Excessively optimistic El Sistema evaluations such as those studied in this article may therefore have implications for the wider field of music education, potentially diverting resources and/or attention away from more effective or equitable programmes.
If critical findings are made clearly visible in reports’ conclusions and are acknowledged by programmes, then evaluations may be a valuable spur to positive change in music education. But if such findings are played down in reports and then airbrushed out of the picture in their public presentation – as occurred with the 2016 IDB study, but can also be seen in evaluations, publicity, and media stories about Sistema Scotland – then evaluations simply serve as a justification for the status quo, however problematic it may be.
