Introduction
This paper presents a cost analysis of a universal, school-based social and emotional learning (SEL) and literacy intervention that was implemented in nine public schools over three consecutive years. The rationale for this research is threefold: (1) an effective SEL intervention can foster students’ social and emotional development, as well as academic performance and can help students achieve greater success in adulthood (Durlak, Weissberg, Dymnicki, Taylor & Schellinger, Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Payton et al., Reference Payton, Weissberg, Durlak, Dymnicki, Taylor, Schellinger and Pachan2008); (2) cost evaluations of SEL interventions are not typically conducted (Crowley, Jones, Greenberg, Feinberg & Spoth, Reference Crowley, Jones, Greenberg, Feinberg and Spoth2012; Duncan & Magnuson, Reference Duncan and Magnuson2007); but can provide valuable information for program implementers, schools, and other relevant stakeholders, and (3) there are several types of methods for conducting a cost analysis that can answer different questions concerning the costs of an intervention, and it is important for the evaluator to know which method will be most appropriate for their evaluation.
Overview of social and emotional learning
SEL is the process of integrating thinking, feeling, and behaving in order to become aware of the self and of others, make responsible decisions, and manage one’s own behaviors and those of others (Elias et al., Reference Elias, Zins, Weissberg, Frey, Greenberg, Haynes, Kessler, Schwab-Stone and Shriver1997). The popularity of SEL can be linked to the research on emotional intelligence (EI) that was promoted by Goleman (Reference Goleman1995). EI refers to an individual’s ability to process and respond to emotions; this can include recognizing the expression of emotions in others, and using emotions to improve thinking and to regulate one’s own emotions (Mayer & Salovey, Reference Mayer, Salovey, Salovey and Sluyter1997; Salovey & Mayer, Reference Salovey and Mayer1990). These EI abilities are often associated with the SEL skills that lead to social competence, adaptation, and academic success.
Definition of SEL
According to the Collaborative for Academic, Social, and Emotional Learning (CASEL), SEL involves the processes of developing social and emotional competencies in children (CASEL, 2005). CASEL has identified five sets of cognitive, affective, and behavioral competencies that make up SEL:
-
∙ Self-awareness: The ability to accurately recognize one’s emotions and thoughts and their influence on behavior. This includes accurately assessing one’s strengths and limitations and possessing a well-grounded sense of confidence and optimism.
-
∙ Self-management: The ability to regulate one’s emotions, thoughts, and behaviors effectively in different situations. This includes managing stress, controlling impulses, motivating oneself, and setting and working toward achieving personal and academic goals.
-
∙ Social awareness: The ability to take the perspective of and empathize with others from diverse backgrounds and cultures, to understand social and ethical norms for behavior, and to recognize family, school, and community resources and supports.
-
∙ Relationship skills: The ability to establish and maintain healthy and rewarding relationships with diverse individuals and groups. This includes communicating clearly, listening actively, cooperating, resisting inappropriate social pressure, negotiating conflict constructively, and seeking and offering help when needed.
-
∙ Responsible decision-making: The ability to make constructive and respectful choices about personal behavior and social interactions based on consideration of ethical standards, safety concerns, social norms, the realistic evaluation of consequences of various actions, and the well-being of self and others (CASEL, 2005, para 3).
The benefits of SEL
Programs that aim to improve students’ SEL may be effective for a number of reasons. Effective SEL programs can improve students’ self-efficacy, self-awareness, and academic confidence, which may then encourage them to persist in the face of academic and social challenges (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011). Students who have strong social–emotional skills in middle childhood have been shown to perform better academically during their school years (Payton et al., Reference Payton, Weissberg, Durlak, Dymnicki, Taylor, Schellinger and Pachan2008). Aber, Grannis, Owen and Sawhill (Reference Aber, Grannis, Owen and Sawhill2012) define social–emotional success as the ability to pay attention in class, manage anger, stay organized, persist in completing tasks, and other skills that indicate self-control and ability to learn. Also relevant to social–emotional skills is the absence of problem behaviors such as arguing, fighting, and aggression.
In Durlak et al. (Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011) meta-analysis of 213 programs, which covered over 30 years of SEL research, they revealed that a successful SEL program increased students’ academic performance by 11 percentile points, as compared to students who did not receive any SEL programs. These programs may also increase students’ problem-solving abilities, which can help them face academic challenges and better navigate interpersonal conflicts at school (Zins & Elias, Reference Zins and Elias2007).
The SEL programs also reduced aggression and emotional distress among students, increased helping behaviors in school, and improved positive attitudes toward self and others (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011). By reducing misbehavior and the amount of time spent on classroom management, SEL programs create more time for teaching and learning. SEL also strengthens students’ relationships with their peers, families, and teachers. Researchers have demonstrated the importance of caring teacher–student and student–student relationships in fostering students’ commitment to school and in promoting academic success (Hawkins, Smith & Catalona, Reference Hawkins, Smith, Catalona, Zins, Weissberg, Wang and Walberg2004). Durlak et al. (Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011) also found that effective SEL programs were those that incorporated four elements represented by the acronym SAFE: (1) sequenced activities that led in a coordinated and connected way to skills, (2) active forms of learning, (3) focused on developing one or more social skills, and (4) explicit about targeting specific skills.” (Jones & Buffard, Reference Jones and Bouffard2012, p. 6).
The positive outcomes of a successful SEL prevention program demonstrate that a school and its students can benefit both academically and socially/emotionally from such a prevention program (Aos, Lieb, Mayfield, Miller & Pennucci, Reference Aos, Lieb, Mayfield, Miller and Pennucci2004). However, the decision on which program a school selects to implement often rests on costs. To further complicate the matter, even if schools recognize the costs of their programs, they may not have information on whether the benefits to students served are worth these costs. Likewise, even if information about the benefits of programs is available, information about benefits relative to costs may not be. The scientific community, policymakers, and school leaders should have information on cost–effectiveness of school-based preventative programs to successfully advise about potential programs to select for their schools.
The National Interest in SEL
The importance of SEL in schools is also being recognized by the general public. According to a survey by the research-based polling company Gallup on the American public’s attitudes toward public education, 78% people polled believed that schools should teach students communication skills, and over half (51%) of the people polled believed that schools should also build students’ character (Bushaw & Lopes, Reference Bushaw and Lopes2013). Although the awareness and support for SEL is definitely improving, often cognitive skills take precedence. This focus on the academic curricula is especially true following the passage of the No Child Left Behind Act in 2001, when schools were pressured to focus more on academic subjects and prepare students for standardized testing. However, despite these challenges, the adoption of SEL programs in schools and of SEL standards in states has grown since the No Child Left Behind Act (Dusenbury, Zadrazil, Mart & Weissberg, Reference Dusenbury, Zadrazil, Mart and Weissberrg2011). Educators are realizing that SEL programs not only help students with their social and emotional skills, but can improve their academic achievement as well (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011).
The SEL movement has been both successful and moving rapidly in the policy area in the past several years. For example, forty-nine states have social and emotional learning standards for early childhood and prekindergarten education (Dusenbury et al., Reference Dusenbury, Zadrazil, Mart and Weissberrg2011). And many states are following Illinois’s lead, in making sure school districts take the proper steps to address SEL in their curricula and hold schools accountable for student’s growth in SEL (Gordon, Ji, Mulhall, Shaw & Weissberg, Reference Gordon, Ji, Mulhall, Shaw and Weissberg2011).
The need for a standardized cost analyses in education, specifically for SEL interventions
As more SEL interventions are developed, schools and policy makers will face more choices about which programs to pursue. A cost analysis can provide important information in making these choices, and may increase the likelihood that cost-effective SEL programs be implemented and sustained in more schools, which, in turn, may support the healthy academic, social, and emotional development of more children (Aos et al., Reference Aos, Lieb, Mayfield, Miller and Pennucci2004).
The need for standardizing cost analysis should be a concern not only in the smaller field of SEL interventions, but also for other programs that influence social policy, such as early childhood interventions, and substance abuse programs. In an extensive review of benefit-cost analysis (BCA) in early childhood interventions, Karoly (Reference Karoly2012) points out that costs are not always measured with the same high standards as benefits. Karoly stresses the need for cost data to “allow for estimation of the full economic costs associated with program delivery, including both resources that require cash outlays, as well as the opportunity costs associated with resources that may be provided in kind (e.g., space provided without charge or at a subsidized rate or volunteer labor” (Karoly, Reference Karoly2012, p. 11). Unfortunately, what can happen in these cost analyses is that estimates of program costs are based solely on the budget. Vining and Weimer (Reference Vining and Weimer2010) also expressed this need to standardize cost analyses of social programs, especially BCA, as a high-quality BCA can help make more informed choices regarding social policies.
Although there are resources that SEL program evaluators can use to help guide them in the methods of a cost analysis, like Boardman, Greenberg, Vining and Weimers (Reference Boardman, Greenberg, Vining and Weimer2011) classic textbook Cost–Benefit Analysis: Concepts and Practice, performing a cost analysis of an SEL program is still a relatively new concept. Furthermore, it is not as standardized in its methods as it is in other fields, such as health care (Drummond, O’Brien, Stoddart & Torrance, Reference Drummond, O’Brien, Stoddart and Torrance1997; Gold, Russell, Siegel & Weinstein, Reference Gold, Russell, Siegel and Weinstein1996). In health care, cost analysis is used often to explore whether a new treatment proves better than the standard care both in cost and effectiveness of the treatment. This allows the decision maker to consider whether the new treatment is better than the status quo. When Dr. Henry Levin, a leading education-based economist, began applying cost analysis to education, there were almost no cost–effectiveness studies on education reported in the literature (Levin, Reference Levin2001), as education evaluation often relies on budgets, which are usually developed before expenditures are realized, and which do not necessarily account for the true economic costs and benefits of a program (Rapporteur, Reference Rapporteur2009). Furthermore, there is often confusion or controversy about which measures of effectiveness to use, making cost analyses comparatively less prevalent in the field of education.
As funding for education receives increasing scrutiny, measuring the efficiency of educational spending becomes even more important. A cost analysis provides some of the information needed to answer the question, “Is this particular education program ‘worth it’ in a financial sense?” This answer is critical when helping to determine which programs to fund with limited resources.
The estimated total cost of U.S. spending on public elementary and secondary schools is approximately $638 billion per year. A two percent improvement in efficiency could potentially save $12.7 billion (Levin & McEwan, Reference Levin and McEwan2002; National Center for Education Statistics, 2012). By analyzing educational decisions in terms of both costs and outcomes, policy makers can select those educational approaches that are most promising in their use of resources.
An effective program may not necessarily be cost-effective. Many times, educational evaluators are focused only on effectiveness, and pay little attention to the cost. For example, reducing class size has shown tremendous effectiveness in increasing standardized test scores. However, the costs of reducing class size are large (e.g., new teachers hired, more classrooms built), and perhaps funds could be spent on other educational interventions that would produce similar outcomes (Brewer, Krop, Gill & Reichardt, Reference Brewer, Krop, Gill and Reichardt1999; Levin & McEwan, Reference Levin and McEwan2001).
Types of cost analysis for education evaluation
There are various methods of cost analysis that can answer questions an education policy maker might have.Footnote 2 It is important that an evaluator know the main questions of the evaluation so he or she can determine which form of cost analysis, if any, should be conducted (White et al., Reference White, Albers, DiPerna, Elliott, Kratochwill and Roach2005). For example, if an evaluator only wants to know how much a particular program will cost and whether it can be implemented within an existing budget, a basic-cost or cost-feasibility analysis may be appropriate. However, if an evaluator wants to also know about costs relative to the effectiveness or utility of a range of programs or interventions, a BCA, cost–effectiveness analysis (CEA), or cost–utility analysis (CUA) may be desirable (Levin & McEwan, Reference Levin and McEwan2001).
Examples of cost analysis of educational interventions (with SEL components)
Although cost analysis in education is still not being utilized as some researchers and educational policy makers would like (Levin & McEwan, Reference Levin and McEwan2002), those cost analyses that have been performed have provided a greater understanding of whether a particular program is a good investment. One notable example is the HighScope Perry Preschool, which was a significant, longitudinal study that measured the effects of high-quality early childhood care and education.
A distinguishing feature of this study was the BCA of the program. The analysis measured the cost of the preschool program against the economic benefits resulting from the program. The results indicated that adults at age 40 who participated in the preschool program had higher earnings, were more likely to hold a job, had committed fewer crimes, and were more likely to have graduated from high school. Overall, the study documented a return to society of more than $16 for every tax dollar invested in the early care and education program (Belfield, Nores, Barnett & Schweinhart, Reference Belfield, Nores, Barnett and Schweinhart2006; Schweinhart, Reference Schweinhart2002, Reference Schweinhart2003).
Another example was the CEA performed on the Fast Track, a prevention program that aims to prevent chronic and severe conduct problems for high-risk children. The CEA demonstrated that the intervention lacked strong enough effects to justify the average cost per child of $58,283 (Foster, Reference Foster2010). The information from this CEA might guide policy makers and decision makers to either eliminate or reduce some of the intervention components, or pursue a different intervention program. (Foster & Jones, Reference Foster and Jones2005; Foster, Jones & Conduct Problems Prevention Research Group, Reference Foster and Jones2006).
Rationale for this SEL intervention’s cost analysis
This cost analysis is part of the ongoing impact study of The 4Rs Program (Reading, Writing, Respect and Resolution), a universal, school-based intervention program that integrates the teaching of SEL and literacy skills for children in grades K–5. The 4Rs is a program of Morningside Center for Teaching Social Responsibility, which is based in New York City. To date, a 3-yearFootnote 3 (6-wave) longitudinal, school-randomized-controlled trial involving 18 New York City public elementary schools (nine intervention schools and nine control schools) has demonstrated positive impacts on the quality of classroom interactions after 1 year of program implementation (Brown, Jones, LaRusso & Aber, Reference Brown, Jones, LaRusso and Aber2010), and on children’s social-cognitive processes associated with aggression (e.g., self-report of hostile-oriented attributions, tendency to select aggressive responses in conflict situations), depressive symptoms, and teacher reports of children’s attention skills (e.g., attention-deficit/hyperactivity disorder or ADHD), social competence, and aggressive behavior after 2 years of program implementation (Jones, Brown & Aber, Reference Jones, Brown and Aber2011).
Objectives
The present analysis estimates costs during the 3 years in which The 4Rs Program was implemented as part of this randomized-controlled study.Footnote 4 Specifically this paper will: (1) establish a standardized and transparent methodology for conducting a basic-cost analysis, (2) determine the initial retrospective, basic costs of The 4Rs Program, and (3) lay the groundwork for more extensive cost analyses, such as a CEA.
Method
This cost analysis bases its methodology primarily on a Cost–Procedure–Process–Outcome Analysis (CPPOA) model that has been well-researched and tested in programs in the social services and behavioral health fields (Yates, Reference Yates1996, Reference Yates2009, Reference Yates, Cooper, Camic, Long, Panter, Rindskopf and Sher2012). The methodology also incorporates the ingredients model used by Levin and McEwan (Reference Levin and McEwan2001), to increase the accuracy and validity of our estimates.
Ingredients model
The ingredients model is a systematic approach to estimate the costs of social interventions (Levin, Reference Levin1983). In this model, resources are referred to as the “ingredients” of the program. The ingredients approach to cost estimation has three phases: “(a) identification of ingredients; (b) determination of the value or cost of the ingredients; and (c) an analysis of the costs in an appropriate decision-oriented framework, e.g., based on cost burden across various agencies; at different levels of scale; adapted for different geographical locations or modes of implementation” (Hollands et al., Reference Hollands, Pan, Shand, Cheng, Levin, Belfield, Kieffer, Bowden and Hanisch-Cerda2013, p. 9). The ingredients model encourages the cost evaluator to collect resource information from a various sources. Reports, articles, websites, and interviews with key members of the program help the evaluator construct a detailed ingredients spreadsheet, listing type and quantity of every resource (Hollands et al., Reference Hollands, Pan, Shand, Cheng, Levin, Belfield, Kieffer, Bowden and Hanisch-Cerda2013; Levin & McEwan, Reference Levin and McEwan2001; Yates, Reference Yates1999).
Cost–procedure–process–outcome analysis (CPPOA) model
Using aspects of the ingredients model, the CPPOA model (Yates, Reference Yates1996, Reference Yates1999) provides a useful framework to measure the types and amounts of (a) resources, (b) procedures, (c) processes, and (d) outcomes for a program or intervention (Yates, Reference Yates1996, Reference Yates1999). The CPPOA model does not simply list, but attempts to quantify relationships between the resources, procedures, processes, and outcomes in direction and strength (see Figure 1).
Basic logic model of CPPOA. Adapted from “Measuring and Improving Cost, Cost-Effectiveness, And Cost–Benefit for Substance Abuse Treatment Programs” by Yates (Reference Yates1999), National Institute on Drug Abuse, p. 11. Copyright 1999 by National Institutes of Health. Adapted with permission of the author.
Resources
This includes all personnel skills and time, facilities, equipment and materials, transportation, and all the other services (e.g., financial, security, accounting) that make the program possible. The monetary value of resources used in a program is its cost. CPPOA measures and retains information on the amount of each type of resource used by each procedure in the program, for each participant (i.e., for each student) (Yates, Reference Yates1996).
Procedures
These are the activities in which the participants are engaged as part of the program or intervention. These activities transform program resources into, ultimately, program outcomes (Yates, Reference Yates1996, Reference Yates1999).
Processes
These are neurological, cognitive, and affective events occurring within participants that link program procedures to program outcomes (e.g., fewer acts of aggression, more positive and focused thoughts, reduced anxiety and depression) (Yates, Reference Yates1999).
Outcomes
These are the results many of the stakeholders are most interested in, as reaching the outcomes can support or justify the costs of the resources. Outcomes can be nonmonetary or monetary. They can include proximal and distal observable acts or states of the participant, and can include employment and income produced by employment or entrepreneurship, (reduced) substance abuse, changes in use if health services and economic assistance, acts toward others (e.g., aggression) (Yates, Reference Yates1999).
Whereas the ingredients model discovers and measures the types and mounts of resources used by a program, CPPOA begins by listing program resources, procedures, processes, and outcomes and then assesses how much of each type of resource is used in each of the procedures that, in sequence or concurrently, constitute the program. Allocating program resources to one or more specific procedures is one of the most important steps in a cost-inclusive analysis.
Procedures are what program managers can later decide to retain, enhance, diminish, or drop altogether depending on constraints on the resources it uses and how it might affect outcomes (Yates, Reference Yates1980, Reference Yates1996). In its fullest implementation, which can be challenging to conduct empirically, CPPOA also quantifies which procedures contribute how much to which biopsychosocial processes, and how much change in each of those processes contributes to change in outcomes both nonmonetary and monetary (cf. Yates, Reference Yates1996, Reference Yates1999).
Cost analysis model for The 4Rs program based on CPPOA and ingredients model
For The 4Rs Program cost analysis we start with the CPPOA model (excluding the processes) and use Levin’s ingredients model to find the program ingredients and calculate their costs (illustrated in the bottom row of Figure 2). A benefit of applying this second model approach is that extensive research into the budgets, meeting with the program’s director and accountant helped ensure that the ingredients/resources were accurately captured.
Cost analysis model for The 4Rs Program. Adapted from “Resource consumption of a diffusion model for prevention programs: the PROSPER delivery system” by Crowley et al. (Reference Crowley, Jones, Greenberg, Feinberg and Spoth2012),Journal of Adolescent Health, 50. Retrieved from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3660099/. Copyright 2012 by the Society for Adolescent Medicine.
Collecting and analyzing costs
Since this is a retrospective cost analysis of the implementation of The 4Rs Program from the 2004–2005 through the 2006–2007 school years, costs are already known.Footnote 5 The main goal is to collect all the costs of The 4Rs Program and to assign them to specific activities. Although obtaining accounting records and budgets is helpful in estimating costs, it is also important to capture more abstract or indirect costs, such as time or labor. Not all of these expenditures may be captured in a budget (e.g., teachers’ time implementing the program in the school). To collect and analyze the costs of The 4Rs Program, we relied on budget and grant reports, accounting records, and teachers’ time logs, as well as interviews with several of the program’s administrators.
Rationale of cost categoriesFootnote 6
To understand and capture the costs of The 4Rs Program, all the resources used for program implementation were organized into a list of basic-cost categories. This first step is critical to producing an accurate cost analysis of any program. Cost categories for most programs can be organized into the following: (a) personnel costs, (b) facility costs, (c) materials costs, (d) other program inputs. Within each category, specific resources or ‘ingredients’ can then be identified.
According to Yates (Reference Yates1999) and Levin and McEwan (Reference Levin and McEwan2001), the standards for a complete and accurate list of program resources should be proportionate with the part a given resource has in relation to the program’s total cost. For example, personnel costs typically represent a large proportion of program costs (Levin & McEwan, Reference Levin and McEwan2001), and therefore should be analyzed in greater detail than other program costs.
Understanding the personnel category
All types of personnel need to be accounted for (e.g., full-time, part-time, consultants, and volunteers). Capturing the opportunity cost of labor can be difficult and needs to go beyond what is provided in the budget (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011). According to Yates’s CPPOA model, personnel should be measured in some unit of time (e.g., hours, daily), and then using an established rate for that time, a cost can be assigned. Some personnel costs can be difficult to assess. The salary of a given employee can vary significantly across schools and school districts.
In addition to estimating the costs associated with paid personnel, it is important to estimate the costs of volunteers’ time, if relevant. Volunteer time is especially important in educational cost analyses because teachers and other staff may be expected to contribute preparation time beyond their paid workday (White et al., Reference White, Albers, DiPerna, Elliott, Kratochwill and Roach2005). Referring back to Karoly (Reference Karoly2012), it is important that cost evaluators understand that volunteer time will not be represented in a program’s budget.
Understanding the facility category
Facility use is one of the most difficult resources to value in applied educational settings (White et al., Reference White, Albers, DiPerna, Elliott, Kratochwill and Roach2005). Yates (Reference Yates1999) has suggested using a similar strategy for finding the value of facilities as the one used to value personnel. This includes the following steps:
- Step 1.
-
Determine the span of time in which the space was used as part of the program. For example, the amount of time the program took place in a school.
- Step 2.
-
Determine the total cost per time. For example, if the school is leased or rented, one can calculate the daily or hourly rate of the rental, and then adjust for the proportion of space actually used for the program.
Sometimes an evaluator can estimate the cost of renting or leasing a similar space. However, if a space is owned, not rented (like most school buildings), Levin and McEwan (Reference Levin and McEwan2001, p. 67) recommend the following method for determining the annual cost of an owned facility:
-
(1) Estimate the facility’s replacement value.
-
(2) Determine the facility’s expected life.
-
(3) Divide the facility’s replacement value by its expected life to obtain the cost of depreciation for each year of use.
-
(4) Multiply the value of the undepreciated portion of the facility by an appropriate interest rate to obtain the opportunity cost that investment in the facility represents.
-
(5) Add the annual cost of the depreciated portion to the annual interest lost on the remaining investment to obtain the exact annual cost.
Ultimately, the cost evaluators decide what method to use for facility costs, and include a description of their rationale (White et al., Reference White, Albers, DiPerna, Elliott, Kratochwill and Roach2005).
Understanding the materials category
Although materials are often the most uncomplicated part of educational program cost evaluation, their costs are frequently underestimated (White et al., Reference White, Albers, DiPerna, Elliott, Kratochwill and Roach2005). It is important for cost evaluators to include all materials that are necessary to the functioning of the program (e.g., specialized program materials, office supplies, and other miscellaneous materials).
Understanding the transportation and refreshment categories (or “other program inputs”)
Other program inputs might include resources that do not fit into other categories, or that will later be subcomponents of categories. In The 4Rs Program this includes transportation and refreshments. These cost categories typically represent the smallest percentage of a program’s costs, and as such do not require as detailed a valuation process as personnel costs.
The 4Rs Program basic resource categories
This is also referred to as Ingredients (Levin & McEwan, Reference Levin and McEwan2001).
PersonnelFootnote 7
This resource category includes Morningside Center’s administrators, Morningside Center’s 4Rs Staff Developers, and teachers.
Consultants
This category includes consultants who were hired to help with the program, such as a translator consultant who translated certain 4Rs materials from English to Spanish.
Facilities
This includes cost estimates for the workshop space, and Morningside Center’s office space.
MaterialsFootnote 8
This includes not only the costs of The 4Rs Kits (e.g., teaching guides, children’s books, worksheets, and toys), which are the primary program materials provided to teachers, but also basic materials (e.g., extra paper, clipboards) needed for program implementation throughout the 3 years.
TransportationFootnote 9
This includes travel costs for the Morningside Center’s administrators and The 4Rs Staff Developers related to travel to and from schools (e.g., parking, tolls).
Refreshments
This includes costs for food and drink provided at a variety of implementation-related meetings and workshops throughout the 3 years.
Activities/procedures
After creating a list of basic resources, we sought to capture the activities that were involved in the implementation of The 4Rs Program. This allowed us to understand how resources were distributed throughout different implementation-related activities. Relevant activity categories include:
Internal meetings
These were meetings typically conducted at Morningside Center with Morningside Center’s administrators and The 4Rs Staff Developers to review progress at each school and address any implementation challenges.
Initial trainingFootnote 10
This was the time teachers spent with The 4Rs Staff Developers or a Morningside administrator being introduced to the program and practice of implementing The 4Rs curriculum in their classroom. This initial training also provided an opportunity for teachers to practice conflict resolution skills at the adult level through role-play and experiential learning (Jones et al., Reference Jones, Brown and Aber2011). This initial training took approximately 25 hours.
Ongoing training
This was the time teachers spent with The 4Rs Staff Developers or Morningside administrators on-site at their school in individual or small group sessions (approximately 12 contacts throughout each school year) to receive continued training and coaching in effectively progressing through the units of the The 4Rs curriculum. Ongoing training included lesson modeling and planning, and lesson observations and feedback (Jones et al., Reference Jones, Brown and Aber2011).
WorkshopsFootnote 11
These were full or half-day 4Rs Staff Developer-led workshops that the teachers and/or school administrators participated in to learn more about The 4Rs Program and how to implement it effectively in the classrooms and schools.
Delivery of The 4Rs curriculum
This was the time teachers spent during the week teaching The 4Rs curriculum in each of the 3 years of program implementation. The 4Rs curriculum is grade-specific and includes about 35 lessons – one per week throughout the school year. The teachers use high-quality children’s literature (e.g., Crow Boy, We Are Best Friends) to help the students gain skills and understanding in the areas of handling anger, listening, assertiveness, cooperation, negotiation, mediation, building community, celebrating differences, and countering bias (Phillips & Roderick, Reference Phillips and Roderick2003).
Calculating the costs per resource by activity
After developing the list of resources and activities, we calculated the costs of resources by activity in three separate stages based on the CPPOA model (Yates, Reference Yates1999):
- Stage 1.
-
Finding the resource use
- Stage 2.
-
Finding the cost per unit
- Stage 3.
-
Finding the resource cost (resource use
$\times$
cost per unit).
Personnel
Teachers
- Stage 1.
-
Resource use. These data were collected from Morningside Center’s accounting logs and teacher time logs and measured in hours to understand teachers’ costs for initial training, ongoing training and workshops. The costs of their time teaching The 4Rs curriculum during school year were calculated based on the average salary of a New York City elementary public school teacher (New York City Department of Education, 2013b ) and the teacher logs.Footnote 12
- Stage 2.
-
Cost per unit. The teachers were paid an hourly rate of $16Footnote 13 for initial training, ongoing training, and workshops. For the teaching The 4Rs curriculum, it was an average hourly rate of $45.
Morningside administrators
- Stage 1.
-
Resource use. These data were collected from Morningside Center’s accounting logs and measured in days that the Morningside administrators worked on The 4Rs Program.
- Stage 2.
-
Cost per unit. The Morningside administrators’Footnote 14 (including the Executive Director, one full-time administrator, and one part-time administrator) daily rate was approximately $356, ranging from $258 to $401 (or approximately $45 per hour).
The 4Rs staff developers
- Stage 1.
-
Resource use. These data were collected from Morningside Center’s accounting logs and measured in days.
- Stage 2.
-
Cost per unit. The 4Rs Staff Developers’ daily rate was approximately $363 (or about $45 per hour).
Consultants
- Stage 1.
-
Resource use. These data were collected using Morningside Center’s accounting records, and was considered as a single unit.
- Stage 2.
-
Cost per unit. This was represented as a set fee that the consultant billed.
Facilities
Office Space
- Stage 1.
-
Resource use. For the office space, this was measured as a single unit. These data were collected using Morningside Center’s accounting records.
- Stage 2.
-
Cost per unit. This was represented as a set fee that the facilities charged for rent each year.
Workshop Space
- Stage 1.
-
Resource use. For the workshop space, this was measured as a single unit. These data were collected using Morningside Center’s accounting records.
- Stage 2.
-
Cost per unit. This was represented as a set fee that the facilities charged for rent.
MaterialsFootnote 15
- Stage 1.
-
Resource use. The 4Rs Kits and any other materials needed in an ongoing manner were measured as a single unit, except for books, which could be quantified. These data were collected using purchase invoices as well as using ‘The 4Rs Teaching Guide’ to ensure all items of The 4Rs Kit were captured.
- Stage 2.
-
Cost per unit. This was represented as a set fee that appeared in Morningside Center’s accounting records.
Results
A total cost for the implementation of The 4Rs Program across the 3 years was estimated at $1,831,296. Year 1 was the costliest ($683,106) followed by Year 2 ($581,764) and finally Year 3 ($566,426) (Table 1).
Total costs by resource and activities (cumulative over 3 years).
Overall costs of resources (cumulative over 3 years) in 2005 dollars.
Costs were then compared by school, with the average yearly cost per school being $67,825. The average yearly cost per student was $130, ranging from $108 to $167 depending on the school.Footnote 16
The average costs of program resources were then compared with teachers being the costliest resource ($927,097, or 51% of the resources).
Additionally, the average costs by activity were estimated (Figure 3), with delivery of The 4Rs curriculum being the highest cost ($877,158 or 48% of the activities). These cost estimates provide insight into the variation of resources and activities needed to implement and sustain The 4Rs Program.
Discussion
This cost analysis contributes to the small, but growing, literature of formal cost evaluations conducted for an SEL intervention program. Knowing that The 4Rs Program cost approximately $130 per student per year can help school administrators decide whether or not a program like The 4Rs Program should be implemented in their school.
In the first year of The 4Rs Program’s implementation, the total resource costs were the highest. One plausible explanation for the higher costs in the first year is that more resources are needed to implement a program than to sustain a program. For example, startup costs, such as The 4Rs Kits are needed in Year 1 and no longer needed in Years 2 and 3. Furthermore, activities such as initial training are no longer needed after Year 1.
One interesting relationship is between the delivery of The 4Rs curriculum and the ongoing coaching and support. Specifically, ongoing coaching and support costs decreased each year, the lowest in Year 3 ($208,753) compared to Year 2 ($212,082) and Year 1 ($240,715), while the delivery of The 4Rs curriculum was higher in Year 3 ($286,607) compared to Year 2 ($274,316). One plausible explanation is that as teachers became more experienced in delivering The 4Rs curriculum (Year 3), they spent more time teaching the curriculum and needed less support from The 4Rs Staff Developers. Specifically, the teachers spent 6,452, 6,517, and 6,809 hours in Years 1 through 3. This relationship is also supported by comparing teachers and The 4Rs Staff Developer costs by year. Specifically, The 4Rs Staff Developer costs decreased each year, the lowest in Year 3 ($144,622) compared to Year 2 ($167,691) and Year 1 ($181,507).
This reinforces the idea that as teachers become more experienced with implementing The 4Rs curriculum, they need less support from The 4Rs Staff Developers. Unlike the increasing hours that teachers spent delivering The 4Rs curriculum, The 4Rs Staff Developers’ hours decreased over the years in ongoing training and support (from 3,240 to 3,084 to 3,096 in Years 1 through 3). This demonstrates that while Year 2 and Year 3 cost less than Year 1, the relationships between the largest cost activities (ongoing coaching and support and delivery of The 4Rs curriculum) were negatively correlated, which can be valuable information for multiyear resource allocation and budgeting.
In New York City, the average cost per student in an elementary school was $13,125 between the years of 2004–2007 (New York City Department of Education, 2013a , b ). Therefore, The 4Rs Program would increase the cost per student slightly less than 1% (0.98%) in New York City. Furthermore, with this transparent and extensive methodology of the costs, school administrators can anticipate where they may most be able to meaningfully reduce the costs of The 4Rs Program. For example, a school may have a large number of volunteers who could take the place of The 4Rs Staff Developers. A standardized cost methodology, such as the one used in this study, can help evaluators conduct CEAs and BCAs, by ensuring that cost comparisons between several programs are both accurate and extensive.
Limitations
The main limitation of this study was that it is a retrospective cost analysis. The 4Rs Program was implemented approximately seven to ten years ago. This made collecting costs for the intervention a challenge. For example, some of the program’s administrators are no longer available to clarify some of the costs. Also, information regarding costs of some specific resources did not exist and required estimates. Ideally, a cost analysis would be performed the same time the program is being implemented. This would allow for more accurate recording of the costs of the program (Levin & McEwan, Reference Levin and McEwan2001; Yates, Reference Yates1999).
Another limitation of this cost analysis was that it assumed all students enrolled in the schools received the same amount of The 4Rs Program, with the rationale that The 4Rs Program is a school-wide intervention. However, it did not acknowledge any student attendance variation or participation variation.
Another limitation concerns not automatically interpreting a decline in costs over time of some activities as a definitive indicator of the intervention becoming more efficient. For example, it may be that the Morningside Center simply reduced the amount of funding in later years – not that the use of funds became more efficient. To help understand whether the reduced costs in Year 2 and Year 3 were caused by increased efficiency of the intervention or simply a reduction in funds, one must incorporate the year-by-year variation in outcomes; we can’t be certain spending less money is a good thing.
Finally, while The 4Rs Program is not directly government funded, both Heckman, Moon, Pinto, Savelyev and Yavitz (Reference Heckman, Moon, Pinto, Savelyev and Yavitz2010) and Reynolds, Temple, White, Ou and Robertson (Reference Reynolds, Temple, White, Ou and Robertson2011) incorporate an additional cost representing the distortionary consequences of interventions funded even in part using taxation, referred to as deadweight loss or excess burden. Estimates of this loss or burden range from 15% to 100% (Heckman et al., Reference Heckman, Moon, Pinto, Savelyev and Yavitz2010) and have not been incorporated in the current analysis. If other evaluations of programs similar to ours in funding sources made similar adjustments to costs, it seems likely that the advantages of 4Rs would continue to be apparent. Until a firmer consensus develops regarding both routine use of deadweight loss/excess burden adjustment and the specific figure to be used in this adjustment, readers who wish to generalize our cost findings to publicly-funded contexts should be cognizant of the societal effects of using public rather than private funding and adjust estimated costs accordingly, especially if comparing public to private funding.
Next steps for The 4Rs program cost analysis
Establishing the costs of The 4Rs Program is the first step for either a CEA or BCA of The 4Rs Program. In a CEA of The 4Rs Program, student outcomes will be incorporated into the analysis. And hopefully, as other SEL interventions provide extensive and transparent records of costs, a CEA can be performed to better understand the cost–effectiveness of different SEL interventions. Another possibility is a BCA of The 4Rs Program. The 4Rs Program evaluation team is collecting ongoing data on a select group of students from the intervention (Jones et al., Reference Jones, Brown and Aber2011), which will potentially allow the determination of costs and benefits of SEL outcomes later in life of The 4Rs Program.
As this cost analysis continues, there should be ongoing collaboration with researchers focused on economic evaluation. This will ensure that the costs will be measured accurately. Furthermore, the transparency of the costs and standardization of the methodology will help establish better comparability of costs for CEAs and BCAs.
Conclusion
The ultimate goal is for SEL researchers to recognize and utilize a set of standardized and transparent cost methods in their evaluations. This will enhance the value of SEL interventions at the policy level. These cost analyses can help evaluators make a more convincing case for funding for effective SEL interventions.
Increasing both the quality and number of economic evaluations of SEL programs is necessary in ensuring overall support of children’s social and emotional development (Jones & Buffard, Reference Jones and Bouffard2012). As national policy moves in the direction of requiring more schools commit to providing SEL to their students, the schools will also need more guidance on what evidence-based SEL programming is available to them and at what costs. This cost information can help a school make the best selection in a SEL program that will work the best for their school depending on their current resources.
Supplementary material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/bca.2015.6.
