1 Introduction
A key purpose of Benefit-Cost Analysis (BCA) is to inform policy-makers about the value of publicly financed programs. Traditionally, benefits are based on parameter estimates of the average effect of a given program, which are then compared to the associated average costs. However, there will often be substantial heterogeneity in the effects and/or costs across individuals, and BCA based on the traditional approach may therefore hide valuable information about who benefits the most net of costs. Nevertheless, inclusion of heterogeneity in BCA is strikingly absent from both the research literature and consultants’ applied work. The purpose of this paper is to analyze the difference between a traditional average BCA approach and an approach that takes heterogeneity into account.
Usually, the Kaldor–Hicks efficiency criterion is applied in BCA. According to this criterion, a public program can be defended if it, in total, has higher benefits than costs. A program can therefore have negative consequences for some individuals, as long as the overall benefits are higher than the overall costs.Footnote
2
$^{,}$
Footnote
3
We maintain the Kaldor–Hicks criterion, noting, though, that there is still a potential for increased efficiency. Making the most of program evaluations and social experiments involves taking into account heterogeneity in program impacts. This basic point was made in the seminal paper of Heckman, Smith and Clements (Reference Heckman, Smith and Clements1997), who note that
“…many persons would judge programs to be successful if enough persons, or enough of the right kinds of persons, reaped benefits from them even if the average participant did not.”, Heckman et al. (Reference Heckman, Smith and Clements1997, p. 488).
It is difficult not to concur with this point of view, and in a BCA setting it includes potential variations on the cost side. One important consequence of heterogeneous benefits and/or costs might be that public funding can potentially be targeted at those who gain the most net of costs. This will make society better off overall.
Taking account of heterogeneous effects in BCA is a rather basic idea. However, to our knowledge this has never been done. Why is it not standard? In recent years, there has been a surge in methodological developments and applied work related to identification and estimation of treatment effects, see e.g., Imbens and Wooldridge (Reference Imbens and Wooldridge2009). This literature has focused mostly, but not entirely, on average treatment effects (ATEs).Footnote 4
In this paper, we carry out a BCA of a Danish return-to-work (RTW) program, by which stakeholders within a municipality coordinate their efforts. To emphasize the potential gains of distributional effect analysis, we seek to understand who benefits the most from coordinated efforts and how. We study a large Danish municipality in which the RTW experiment was successfully implemented as a randomized controlled trial (RCT).Footnote
5
$^{,}$
Footnote
6
The Danish National RTW program is based on the notion that long-term sickness absentees often face a series of challenges related to social, work and health status and, consequently, that employers, municipal case workers and health and rehabilitation professionals all play an important role in facilitating their RTW (Nielsen et al., Reference Nielsen, Hansen, Aust, Tverborgvik, Thomsen, Bjorner, Mortensen, Rugulies, Winzor, Ørbæk, Helverskov, Kristensen and Poulsen2013). Coordinated efforts among these stakeholders have often been suggested as an effective tool to reduce long-term sickness absence and have been found to generate small, but likely important, relative absolute benefits (Schandelmaier et al., Reference Schandelmaier, Ebrahim, Burkhardt, de Boer, Zumbrunn, Guyatt, Busse and Kunz2012). The Danish National RTW program produced mixed results (Poulsen et al., Reference Poulsen, Aust, Bjorner, Rugulies, Hansen, Tverborgvik, Winzor, Mortensen, Ørbæk and Nielsen2012; Nielsen et al., Reference Nielsen, Hansen, Aust, Tverborgvik, Thomsen, Bjorner, Mortensen, Rugulies, Winzor, Ørbæk, Helverskov, Kristensen and Poulsen2013). For instance, Poulsen et al. (Reference Poulsen, Aust, Bjorner, Rugulies, Hansen, Tverborgvik, Winzor, Mortensen, Ørbæk and Nielsen2012) found that the effects of the intervention in relation to duration of sickness absence differed markedly among the three municipalities examined. They argue that this indicates that contextual factors are of major importance for the success or failure of this complex intervention. Possibly, part of the reason for these mixed results can be related to mixed results within municipalities.
Understanding better how or why the program works is important, because RTW programs can be relatively costly. Hence, in order to be able to induce municipalities to make this additional effort and investment it is of great importance to ascertain whether or not the program is actually self-sustainable.
We find that benefits net of costs are positive, based on estimates of the average effect. However, quantile treatment-effect (QTE) analysis reveals that the program effect is heterogeneous. We define the “efficiency potential” (EP) as the increased benefits net of costs that would accrue to the municipality (or society), if it was possible to screen participants perfectly to those who benefit individually at least as much as the costs involved. Such a screening must necessarily be based on observable characteristics. In our RTW example, the EP is 0.22, which underscores the potential and the scope for future BCAs. However, we also find it difficult to realize the potential in our empirical example. In spite of the data being very rich, none of the observable characteristics interacted with the treatment were found to show significance in reducing the duration of sickness absence. Bitler, Gelbach and Hoynes (Reference Bitler, Gelbach and Hoynes2006, Reference Bitler, Gelbach and Hoynes2014) point out that this type of finding will often occur due to variation not only between but also within subgroups.
The remainder of the paper is organized as follows. In the next section, we describe the RCT experiment and the detailed administrative data records, while Section 3 outlines the methodological framework. In Section 4, we first list the results based on traditional average benefits and costs and subsequently look into heterogeneity and the EP. Finally, we discuss implications and conclude with suggestions for future research and analysis.
2 Data and institutional background
As mentioned, the empirical application builds on experimental data from a randomized control trial of an RTW program in a large Danish municipality combined with detailed administrative panel data and survey data.
2.1 The institutional set-up related to sickness absence
In Denmark, employed, self-employed and insured unemployed people, are eligible for sickness benefits if they are unable to work due to sickness. The employer is obligated to pay either wages or sickness benefits during the first 30 days of the sick-leave period, and the municipality pays for the rest of the sick-leave period up to a maximum of 52 weeks. In some cases, dispensation can be granted up to another 52 weeks. All sickness benefit recipients are classified into three different categories: (1) if the sick person is expected to be able to return to work within 3 months, (2) if long-term sickness absence is assessed to last longer than three months, but the person on sick leave is able to attend labor market programs, and (3) if sickness absence is assessed to last longer than three months and the person on sick leave is unable to attend labor market programs.Footnote 7 For all persons on sick leave, the municipality must perform a sickness benefit management interview within the first eight weeks after the first day of sick leave. After this, the requirement for the frequency of meetings depends on the category. For category 2 individuals, the municipal case manager must carry out a sickness benefit management interview every four weeks. The goal is to develop an RTW plan that promotes the sick person’s return to work, which can include participation in labor market programs.
2.2 Administrative registry data
A wide range of longitudinal administrative registry data from Statistics Denmark were used in the analysis. These records, all at the individual level, allow us to follow each person in both the treatment and the control group over time, both before and after the start of the RTW program. Specifically, the data include monthly hours worked and monthly income or welfare transfer, labor market status and purchases of prescription drugs, with information on date of purchase, price and type of drug. Finally, data include information on primary health care, i.e., all contacts with general practitioners, special practitioners and psychologists covered by the public insurance.Footnote 8
2.3 A first look at the data
The experiment covered 2,310 sick-listed persons, of which 887 were being treated within the coordinated RTW program.Footnote 9 Ideally, we should find no difference in the characteristics between the treatment group and controls, see Table 1.
Generally, we only find few observed differences between the treatment (intervention) and control groups. One difference is that the rate of women among sick-listed persons under the RTW program is significantly higher than in the control group of sick-listed persons receiving ordinary sickness benefit management.
Two sample
$t$
-tests for differences in control versus treated, average values.
Note: *
$p<0.10$
, **
$p<0.05$
, ***
$p<0.01$
.
In addition, the treatment and control groups also differ in that individuals with self-assessed psychiatric illnesses are overrepresented in the treatment group vis-à-vis the control group. Moreover, prior job status and age differ significantly. These differences are not troublesome on their own, because we are able to condition on observed covariates in the treatment-effect analysis. However, the fact that several variables are significantly different raises the concern that, possibly, the random assignment was flawed or there was sample selection after random assignment. Concerning the differences in age the differences found are merely marginal since only the oldest age category differs and the mean age does not differ. As for prior job status and psychiatric illness they both are highly gender correlated.Footnote 10 That is, these differences should not be seen as three independent factors being different, rather they are likely to be a consequence of the share of women being higher in the treatment group vis-à-vis the control group. However, we have no explanation for this gender difference. Both Aust et al. (Reference Aust, Nielsen, Grundtvig, Buchardt, Ferm, Andersen, Lund, Jelle, Andersen, Hansen, Tverborgvik, Helverskov, Bjorner, Rugulies, Ørbæk, Winzor, Bültmann and Poulsen2015) and Poulsen et al. (Reference Poulsen, Aust, Bjorner, Rugulies, Hansen, Tverborgvik, Winzor, Mortensen, Ørbæk and Nielsen2012) have documented the design and implementation of the RCT study and they conclude that this municipality has performed very well. Poulsen et al. (Reference Poulsen, Aust, Bjorner, Rugulies, Hansen, Tverborgvik, Winzor, Mortensen, Ørbæk and Nielsen2012) are also puzzled by the gender difference, but conclude that it must be a simple coincidence. We balance the two groups by including covariates in the treatment-effect analysis.
3 Methodological approach
3.1 Average and quantile treatment effects
The methodological approach was chosen in accordance with the available experimental data described above. The availability of experimental data makes it possible to obtain the ATE without the standard type of selection problem that often hampers treatment analyses. The RCT lends itself to a simple nonparametric analysis of the treatment effect. Using common notation, we may write
$$\begin{eqnarray}\displaystyle \text{ATT}=\text{ATE}=\bar{Y}_{1}-\bar{Y}_{0} & & \displaystyle\end{eqnarray}$$
where
$\bar{Y}_{1}$
is the average number of weeks on sickness absence for the treatment group, while
$\bar{Y}_{0}$
is the average number of weeks on sickness absence for the control group. The nonparametric average effect of the RTW program is simply the difference between the two averages.
As usual, ATT denotes average treatment of the treated, and ATE denotes average treatment effect (for the population). Given the RCT design, these two types of average effects coincide here. The interpretation, therefore, is that for the entire population we estimate what an extension of the program would yield. The results do not carry over to other municipalities, but do carry over to extended versions of the program within this municipality. As shown in Table 1, though, control variables are needed, in order to balance the treatment and control groups on a few of the observable characteristics. We do this by estimating a weighted OLS model.
One important purpose of this paper is to shed light on the mechanisms that make RTW coordination (economically) successful. One avenue is simply to introduce interactions between RTW participation and observable characteristics, for instance, interaction with type of diagnosis.Footnote 11 Here, we make additional use of the tremendously good data available. In order to make the most of the available information, we use confirmatory factor analysis (CFA) to create a combined measure that may capture the severity of the condition of the sick-listed person. In particular, the detailed individual historical records of employment, purchases of prescription drugs and hospital diagnoses. Moreover, the self-assessed diagnoses recorded by the person who did the initial screening of the long-term sick-listed person are combined into a single factor that can be thought of as a measure of complexity combined with sickness history. Such latent factors are computed overall as are within types of reported diagnoses and medicine used in relation to that condition.
It is intrinsically difficult to measure the program impact for each individual, since, by definition, the counterfactual for each person is unobserved. However, we may compute the entire distribution of outcomes for treated persons and compare this with the distribution of outcomes for controls (see Heckman et al. (Reference Heckman, Smith and Clements1997)). If the distance between the two distributions is not constant over the population, this will be sign of heterogeneity. Hence, although we are unable to estimate the impacts individually, we might nevertheless gain knowledge about whether there is heterogeneity in the distribution of effects. It is important to note the limitations of QTE. With this approach, we can ascertain that there is some degree of heterogeneity if the two distributions, treated vs controlled, are not alike, even though QTE does not reveal the individual impact. However, if we invoke an order assumption, whereby individuals with the longest sickness duration without treatment also are the ones with longest duration among the treated (and so forth following the order of all participants), we obtain a measure of individual net benefits. While surely false to some degree in almost any application, this assumption may be approximately true for many cases including our application.Footnote 12
In practice, we do this by estimating QTEs. Following Bitler et al. (Reference Bitler, Gelbach and Hoynes2006), consider two distributions of predicted duration of sick leave for treated and nontreated persons, respectively,
$F_{1}$
and
$F_{0}$
, and define QTE as
$\unicode[STIX]{x1D6E5}_{q}=y_{q}(1)-y_{q}(0)$
, where
$y_{q}(t)$
is the
$q$
th quantile of distribution
$F_{t}$
. Graphically, this is equivalent to the horizontal distance between two cumulative distributions.
3.2 Benefit-cost analysis
We link the estimated effects to the costs of achieving them through a BCA for society as a whole. The RTW program is implemented at the municipal level, and given the economic strain put on municipalities, costs and benefits related to such a decision are likely very important. However, there are returns beyond the municipality level, and we therefore assess the economic effects of the program for society at large. The elements included in the BCA are listed in Table 2.
Elements included in the BCA.
The cost side includes extra costs to the coordinator and marginal costs of public funds. Taxes and transfers between the municipality and citizens do not impact the production in society in general and are therefore not to be included in the BCA at society level, (Mishan and Quah (Reference Mishan and Quah2007)). Still, distortionary effects from collecting taxes, the marginal costs of public funds (MCF), are included at the society level. The additional or reduced distortion expected from this program stems from the additional or reduced use of public transfers due to the program. We use the treatment-effect estimate from the weighted OLS model to calculate the reduced public cost for sickness benefits and the increased public cost for other public transfers. The remaining costs and benefits are measured directly in data.Footnote 13 The range of MCF is discussed further below. Among benefits, wage is the primary component, but reduced (public) costs for medical treatment and medicine are also accounted for.
All BCAs are uncertain, and ours is no exception. For instance, the estimated benefits are essentially random variables, and the municipal costs are partly based on subjective assessments by civil servants and are intrinsically difficult to determine, (see Appendix B). The appropriate approach for a BCA, therefore, is to treat the multiple uncertainties jointly through the use of Monte Carlo simulations. This is also strongly recommended by Vining and Weimer (Reference Vining and Weimer2010); previous examples include Weimer and Sager (Reference Weimer and Sager2009), Nichol (Reference Nichol2001) and Briggs, Mooney and Wonderling (Reference Briggs, Mooney and Wonderling1999). We follow the recommendation and perform a Monte Carlo analysis of the uncertainty related to the BCA.
In practice, we assume that MCF is uniformly distributed across the interval [20%; 120%]. This covers several of the elasticity scenarios in Kleven and Kreiner (Reference Kleven and Kreiner2006).Footnote 14 The use of a uniform distribution in place of a normal or log-normal distribution increases the variance, and this underscores the uncertainty involved for this parameter. The estimated treatment-effect parameter is assumed to follow a normal distribution with standard deviation equal to the estimated standard deviation. It appears reasonable to assume that the uncertainty linked to the municipal assessment of costs is bell-shaped around the “best guess” given. For this reason, we assume a normal distribution for these costs as well, but lacking any prior about the uncertainty we assume that the 95% confidence interval for the “best guess” assessment varies by a factor of plus/minus 100%. In this manner, we combine a continuum of municipal cost scenarios with the Monte Carlo analysis. This approach is useful in situations where one (important) variable is uncertain, and where priors about the uncertainty are arbitrary to some degree and has previously mostly been used for dealing with uncertainty of the social discount rate.
4 Results
4.1 Average treatment and BCA
We first present results from a standard BCA based on the ATEs. We find that, on average, sick-listed individuals under the RTW program have 24 days shorter periods of sickness leave compared to sick-listed persons under the traditional RTW program; see Table 3. The comparison of the treatment and control groups presented in Table 1 showed that the treatment group differed from the control group in various aspects (age, gender, self-assessed cause of sickness absence and prior job status). Hence, we estimate the treatment effect both nonparametrically and in a parametrically weighted OLS model including the covariates that differ significantly between the treatment and control groups. Even though three out of the four covariates significantly affect the duration of sickness leave, including them in the model does not alter the ATE significantly. The new point estimate is 24.6 days.Footnote 15
Average treatment effect of RTW on duration of sickness leave, nonparametric and weighted OLS regression.
Note: Standard errors in parentheses. Duration is measured in days. *
$p<0.10$
, **
$p<0.05$
, ***
$p<0.01$
. Covariates centered on their mean.
Economically, the overwhelming benefits from RTW stem from lower public transfers, and consequently reduced marginal costs for public funds, as well as increased production, see Table 4.
A part of the reduction in sickness benefits is transferred to increased costs for unemployment insurance. This is because being reported fit for duty means that the individual is either back in employment or has been transferred to unemployment insurance or, if the person is not eligible for UI benefits, transferred to social welfare. The major costs are operating costs, but the net effect, the BCA result, is a large positive economic benefit from the RTW program in this municipality. The point estimate is 5,568 DKK per person in net benefit for society. The standard errors, calculated using 50,000 Monte Carlo iterations, indicate a relatively high uncertainty, even though the BCA result is positive at the 95% confidence level.
Benefit-cost analysis, by item (DKK).
Note: Standard errors are bootstrapped based on 50,000 iterations. CI
$=$
95% Confidence Interval. A minus denotes a cost. 5.6 DKK equaled approximately 1 USD, in the years covered by our sampling period. Further details given in Appendix C.
The most uncertain of all inputs in the BCA is the partly subjective assessment of the municipal operating costs. As noted above, we therefore introduce a transparent manner by which one can elicit to what extent the BCA results respond to this uncertainty. A common manner for dealing with uncertainty in the social discount rate is to vary the assessed costs from plus/minus 100% (the 100% level means we use the reported best assessment from the municipality,
$-$
100% means we apply zero cost and
$+$
100% means we apply double the assessed cost); see Moore, Boardman, Vining, Weimer and Greenberg (Reference Moore, Boardman, Vining, Weimer and Greenberg2004).Footnote
16
The normal distribution is truncated to zero in order to avoid that the costs become negative. The results of this exercise are shown in Figure 1.
Net benefits of RTW program.
As the gray area indicates, the horizontal confidence interval broadens as the assessed program cost increases.
4.2 Accounting for heterogeneity
To explore the possibility of a heterogeneous treatment effect, we estimate QTEs. The difference in the distribution of length of absence is informative with regard to how the effect of the RTW program may vary over the distribution. Treatment-effect differences within the distribution of sickness leave duration will indicate whether, for instance, the treatment effects are stronger for the long-term sick-listed than the short-term sick-listed.
Figure 2 shows the cumulative distributions for the treatment groups and the control groups of the RTW program. The horizontal difference in the distributions indicates the impact of the program, and this difference corresponds to the QTE effect, which is presented in Figure 3. A horizontal line for the ATE is presented in Figure 3.
Cumulative distribution of sick-leave duration.
Average and quantile treatment effects.
Figure 3 clearly indicates that the RTW program has varying impact across the distribution. Given the order assumption holds, the largest effects are found approximately between the 40th and 80th percentiles of sickness absence duration. Also, note that the duration of sickness absence is lower for the treatment group at almost all points in the distribution, but not significantly lower at the two extremes.
The comparison between ATE and QTE, shown in Figure 3, reveals substantial heterogeneity in the treatment effect. If we were able to perfectly screen participants and only allocate treatment to individuals with a treatment effect above the average cost level, (in the absence of individualized cost data), i.e., individuals above the break-even line shown in Figure 4, then, obviously, we would be able to improve the bottom line.
Quantile treatment and break-even analysis.
The potential room for improving the bottom line in the BCA is what we define as an EP.
Definition.
The “Efficiency Potential” (EP) is the increased benefits net of costs that would accrue to the society, if it was possible to perfectly screen participants to those who benefit individually at least as much as the costs involved. Formally, the EP is given by
$$\begin{eqnarray}\displaystyle \text{EP}=\frac{(\text{BCA Perfect}-\text{BCA realized})}{\text{BCA realized}}. & & \displaystyle \nonumber\end{eqnarray}$$
In our application, the BCA results in an economic net effect of 7,923,264 DKK (Table 4). The corresponding sum of BCA among the share of individuals with a significantly positive treatment effect is 9,630,864 DKK.Footnote 17 Hence, the implied EP in this application is 0.22.Footnote 18
The heterogeneity can be split between observed and unobserved heterogeneity and, clearly, the EP can only be harvested if we are able to screen individuals based on observable characteristics. One may expect heterogeneity in the treatment effect to arise from specific observable characteristics of the treated. For instance, coordination could very well be of greater use for sick-listed persons with more complicated diseases. Next, we therefore seek to identify which observable covariates drive the heterogeneous treatment effect.
We apply three different strategies to investigate the hypothesis regarding heterogeneous treatment effects.
First, we interact the RTW treatment with prior employment status and level of education. Part of the coordination involves maintaining the individuals’ connection with the labor market (for those who were in employment prior to their illness). One may hypothesize that such a connection would mean that the sick-listed person could benefit from being able to return to his/her former job once ready, and that this could even shorten the sickness spell. Along similar lines, highly educated persons might have a greater degree of empowerment, (Kristensen, Andersen and Pedersen (Reference Kristensen, Andersen and Pedersen2012)), and this could perhaps result in greater benefits from coordination.Footnote 19 However, these interaction terms are insignificant, and so the heterogeneity observed in Figure 2 is not linked to these covariates, see Table 5.
Duration of sickness spell, weighted OLS regression with different interactions.
Note: Standard error in parentheses. *
$p<0.10$
, **
$p<0.05$
, ***
$p<0.01$
. Balanced with covariates, as in Table 3.
The second approach is to investigate the hypothesis that individuals with complicated sickness spells would gain more from a coordinated RTW program. In order to capture the level of complication for each sickness spell, we make use of factor analysis. This allows us to exploit the wide range of good quality data available by summarizing them according to different measures of sickness severity. Using CFA, we estimate factors that are based on a predefined structure.
Hence, three separate measures, one each for severe psychiatric diseases, severe somatic diseases and severe pain diseases, are estimated. In addition, a joint measure for general severe sickness spells is estimated and applied to the treatment-effect estimation. The estimated factors draw upon information on previous hospitalizations, previous medical costs, diagnosis, previous sickness leave periods and previous psychiatric costs.
The estimated factor-specific treatment effects are found to be insignificant for all four factors, see Table 6.
Average treatment effects and interactions with different factors for severity, weighted OLS regression.
Note:
$t$
statistics in parentheses. *
$p<0.10$
, **
$p<0.05$
, ***
$p<0.01$
. Based on 2,310 observations. Balanced with covariates as in Table 3.
Hence, although the QTEs clearly predict the existence of heterogeneity, our measure for sickness severity is apparently not able to capture this heterogeneity. According to the findings in Table 6, having a history of either psychiatric illness, somatic illness or pain-affected illness does not increase the effect gained from being part of the RTW program.
Third, using individual characteristics (including measures of illness) we fit a model of duration, including a dummy for treatment but not including interactions, and then use the predicted values of this model to interact with the treatment. Once again (results now shown) we find very insignificant estimation results of the RTW interactions.
5 Discussion
Based on observable covariates, nothing seems to indicate that the RTW program has different effects on sickness beneficiaries across observable differences. Following Bitler et al. (Reference Bitler, Gelbach and Hoynes2006, Reference Bitler, Gelbach and Hoynes2014), this finding is not surprising. They argue that mean subgroup impacts, which is what we estimate using subgroup interactions, may be poorly suited to capturing treatment-effect heterogeneity when both within-group and between-group treatment-effect heterogeneity are present. Our results indicate that within-subgroup heterogeneity may drive the QTE results. Bitler et al. (Reference Bitler, Gelbach and Hoynes2014) also build on RCT data in their case related to labor supply. They find that adjusting subgroups’ mean impacts for extensive margin impacts of reforms leads to a substantial improvement in the ability of cross-subgroup mean impact treatment effects to explain distributional effects of welfare reform. In our data, labor force participation might also be an important issue, and hence the extensive margin may also be relevant here. Our outcome, however, is limited to observing whether or not the sick-listed persons return to work, with no further analysis of whether they return to work, say, 12 or 37 hours a week.
Our findings corroborate the general message found in Bitler et al. (Reference Bitler, Gelbach and Hoynes2006, Reference Bitler, Gelbach and Hoynes2014) and may consequently draw a rather pessimistic picture of the true potential of accounting for heterogeneity in BCAs. However, while it may be impossible to harvest close to the maximum EP subgroup analyses have nevertheless often been argued to be highly important, (e.g., in the vast literature on ALMP, see the handbook chapter in Heckman, LaLonde and Smith (Reference Heckman, LaLonde, Smith, Ashenfelter and Card1999) and the meta-analysis in Card, Kluve and Weber (Reference Card, Kluve and Weber2010)). Furthermore, our sample size of 1,423 treated individuals and 887 individuals in the control group is relatively modest and consequently provides relatively little statistical power.
Making the most of BCA by including heterogeneity in the analysis ought to include not only heterogeneous treatment effects but preferably also heterogeneous costs. In our application, the municipality had a relatively difficult task in coming up with a credible estimate of the average costs, and individualized cost data may generally be hard to collect. This aspect of heterogeneity in BCA might in fact be just as important as the heterogeneity in treatment effects, though the potential seems to be even more difficult to realize.
6 Conclusion
Heterogeneity in BCA is strikingly absent, even though it is well known that one should expect treatment-effect heterogeneity, and hence expect a scope for targeting publicly funded programs better. Our example here – a large scale RTW program in a large Danish municipality achieving significant average net benefits – shows that there might indeed be a large potential. However, it also shows that, even with the extremely rich data available, we have not been able to gain knowledge about how any screening system could tap into the EP. In part, this lack of significant subgroup effects is a result of a relatively small sample size and the consequent low test power.
The fundamental idea to take away from this analysis remains: There is a large potential for improved efficiency and increased knowledge from including heterogeneous treatment effects in BCAs. Future research, as well as applied consultancy work, should seek to collect the type of data needed for individualized (or subgroup-dependent) costs, and treatment effects should allow for more than a simple average.
Appendix A. The Danish National RTW program
The Danish National RTW program was targeted at sickness benefit receivers in category 2. With a view to improving RTW, it consisted of three core elements that aimed at changing the procedures of sickness benefit management. The core elements were: (i) establishment of a multidisciplinary RTW team consisting of two RTW coordinators (municipal case workers), a psychologist, a physiotherapist or ergotherapist, a part-time psychiatrist and a part-time physician of occupational, social or general medicine, (ii) implementation of standardized workability assessments tools, and (iii) a comprehensive training course for all multidisciplinary R teams prior to the program (for details see Aust et al., Reference Aust, Helverskov, Nielsen, Bjorner, Rugulies, Nielsen, Sørensen, Grundtvig, Andersen, Hansen, Buchardt, Nielsen, Lund, Andersen, Andersen, Clausen, Heinesen, Mortensen, Ektor-Andersen, Ørbæk, Winzor, Bültmann and Poulsen2012, Reference Aust, Nielsen, Grundtvig, Buchardt, Ferm, Andersen, Lund, Jelle, Andersen, Hansen, Tverborgvik, Helverskov, Bjorner, Rugulies, Ørbæk, Winzor, Bültmann and Poulsen2015). Both the assessment tools and the training course had a special focus on supporting better detection and handling of psychological health problems. In contrast to ordinary sickness benefit management, RTW coordinators in the RTW program had the option of referring sick-listed persons to the RTW team for further assessment. However, it was expected that only half of the category 2 sickness beneficiaries would need to be referred to the interdisciplinary RTW team. Hence, part of the task of the RTW coordinator was to assess whether a referral was needed. If referred to the RTW team, the sick-listed person was asked to attend a consultation with a psychologist and with a physiotherapist or ergotherapist. In this way, the RTW team members were able to identify sick-listed persons with both somatic and psychological symptoms, which generally serves as a predictor of risk of long-term sickness.
If further assessment was needed, sick-listed persons could be referred to the psychiatrist and/or the physician of occupational, social or general medicine. The RTW program was implemented in 21 Danish municipalities (Aust et al., Reference Aust, Nielsen, Grundtvig, Buchardt, Ferm, Andersen, Lund, Jelle, Andersen, Hansen, Tverborgvik, Helverskov, Bjorner, Rugulies, Ørbæk, Winzor, Bültmann and Poulsen2015). Only in three of these municipalities was the evaluation design a randomized controlled trial (RCT). This paper builds on the larger of the three municipalities, which gives a highly homogeneous sample with only within municipality variation. The experimental design in this municipality was especially successful, as the randomization was conducted centrally before the cases were divided into two physical divided units, one carrying out the coordinated RTW program, the other offering the ordinary sickness benefit management. Although the caseworkers were placed in geographically separate units, the sick-listed persons were spread geographically independently of being in the treatment group or not.
Appendix B. Municipal economic data
The economic program costs are provided by the municipalities. The costs are assessed both for the treatment and the control group, and the difference thus lies in measuring the extra cost per sickness benefit case for providing the RTW program compared to the ordinary sickness benefit management. The total costs are separated into initial start-up costs and operating costs. Initial start-up costs consist of start-up expenses, on education, office arrangement etc., and operating costs in the form of administrative expenses for case administration and program expenses as labor market training and activation. The cost of the sickness benefit itself is not included.
The costs associated with the program are very difficult to trace out for the municipalities, as these are not registered on specific accounts, and as a set of locally estimated algorithms for distributing personnel costs between RTW and other activities have to be applied. Consequently, the reported costs for the individual municipalities are fraught with uncertainty. In order to reduce sensitivity, and as a conservative approach, we apply the average costs across the 18 municipalities that took part in the program. The cost estimate is conservative in the sense that the costs here are higher than the cost assessment made by the municipality in question.Footnote 20
Appendix C. Shadow prices and BCA
Weekly shadow prices for public transfers are:
-
∙ Sickness benefit: 3,830 DKK
-
∙ UI benefit: 3,830 DKK
-
∙ Social welfare: 3,069 DKK
-
∙ Other transfers: 3,657 DKK
When program participation reduces the duration of sickness absence it enters the BCA as the difference between these weekly shadow prices. If a person makes a transition from sickness benefits to social welfare, the net impact on public funds is
$(\text{3,830}-\text{3,069})=761~\text{DKK}$
. If a person makes a transition from sickness benefits to work the net impact becomes both the reduced sickness benefits (3,830 DKK) and the increased production (
$=$
the wage). In both cases, change in the need for public funds is also included as a change in the marginal cost of public funds.
