We would like to thank Whiteley et al. (Reference Whiteley, Clarke, Sanders and Stewart2015) for a robust response to our critique of their influential valence model of party choice. As some parts of their paper address somewhat different concerns to those examined in our article, our reply focuses on those elements that connect most clearly with the arguments and evidence we have presented. To this end, we shall point to some of the problems in their critique and the possible concerns their current arguments raise for their previous work making the case for the valence model. We nonetheless welcome their willingness to engage in a scholarly debate on this important question.
So, what do Whiteley et al. claim? There are five key points, the first three of which are interconnected so we will consider them together:
First, EC do not actually test the valence politics model. The model is motivated by a theory of electoral choice, not a theory about the relationship between a generalized measure of “party preference”—a sort of “super partisanship”—on the one hand, and a composite measure of “party performance”, on the other. Second, they use inappropriate variables in their analyses in the sense that their measures do not appear in our specification of the valence politics model. Third, their argument provides no theoretical grounding for the key variable which they claim is strongly exogenous, i.e., partisanship (Whiteley et al. 2015).
Variable Selection
According to Whiteley et al. (2015): “the valence model has three key variables that drive voting behaviour and party support between elections: evaluations of party performance on valence issues, partisanship and leader images.” In our analysis, we focus on two key elements of this model: evaluations of party performance and leader images. Given this focus, Whiteley et al.’s criticism of our variable selection is so inaccurate as to raise concerns as to whether they have actually read the paper. They claim that: “key explanatory variables in our valence politics model are missing (e.g., leader images, party best on most important issue)” (Whiteley et al. Reference Whiteley, Clarke, Sanders and Stewart2015, 20). Clearly, as the reader can see, both leader images and “party best on most important issue” are in our models (Evans and Chzhen Reference Evans and Chzhen2015, 8).Footnote 1
Whiteley et al. also seem to think that we include party identity as an indicator of “party preference” (see footnote 1), which is not correct. Throughout their reply they repeatedly refer to our analysis as one focusing on partisanship:
The fundamental problem with EC’s critique is their subtle, yet seductive, invitation to seek a magic variable that could account for the observed dynamics in voting, leader images and party performance judgments. EC believe that the variable is partisanship which, by their account, drives all the other variables under consideration. However, if EC are right, then partisanship must itself exhibit dynamic properties (Whiteley et al., 2015).
However, we do not examine partisanship. Indeed, on pages 14–5 of our paper, we explain why we are not including party identity in our analysis and what would happen if we did. Moreover, we at no point say that the latent party preference measure was supposed to be “super-partisanship.”
Whiteley et al. elaborate at some length on this issue:
When we “round up the usual suspects”, use our standard conceptual and operational definitions of variables of interest, and employ appropriate methods for investigating EC’s claim that partisanship is strongly exogenous, the results strongly support our version of valence politics model. Partisanship is not strongly exogenous in this system (2015).
This means that the variable which is thought to anchor the analysis because it is a strongly exogenous determinant of the other variables has no plausible theoretical grounding. Partisanship has large-scale dynamics but where these come from is a mystery since EC do not account for this “moving mover” (2015).
However, we do not claim at any point that partisanship is strongly exogenous. Partisanship is not in our model. The whole premise of the Whiteley et al. reply with respect to “hunting the snark” is misconceived. Indeed, in other work (Evans and Chzhen under Reference Evans and Chzhenreview), we show why partisanship has no place in valence models—as we note in the paper.
From reading their critique, we can only conclude that when Whiteley et al. argue that we are not testing the “valence” model, they are right only in that we do not operationalize and estimate the very same model using the very same indicators they proposed in some of their work (specifications of the valence model vary even in Whiteley et al.’s own published analyses). However, if their valence model can only be specified in such a restricted way, it does not have a very wide applicability. Given the general claims made for the model, this is a major weakness.
Theoretical Bias?
Whiteley et al. also devote a substantial part of their response to pitting valence theory against spatial models, implying that we are in some way wedded to this particular alternative. That spatial considerations are themselves exogenous influences on vote choice is something we have demonstrated to be inaccurate except under certain conditions, using the type of panel evidence and analysis presented here (Evans and Andersen Reference Evans and Andersen2004).Footnote 2 As with hunting the snark, this is something of a straw man. Let us repeat that we are quite explicitly not privileging any approaches in our analyses of the endogeneity of survey predictors, or outcomes, of party choice. Our studies content themselves with exposing the limitations of the assumed exogeneity of “influences” on party choice such as economic perceptions (Evans and Andersen Reference Evans and Andersen2006; Evans and Pickup Reference Evans and Pickup2010; Pickup and Evans 2013) and issue proximities (Evans and Andersen Reference Evans and Andersen2004) and, in the paper discussed here, elements of the valence model itself. Though we do note that leadership images are relatively robust to our tests, both in this paper and in previous and ongoing work (Evans and Andersen Reference Evans and Andersen2005; Evans and Chzhen Reference Evans and Chzhen2013; Evans and Mellon Reference Evans and Mellon2015). We have also identified the conditions when economic perceptions are relatively more exogenous (Chzhen, Evans and Pickup Reference Chzhen, Evans and Pickup2014). It is likely that almost all survey variables are potentially endogenous under some conditions and ultimately we need to move beyond approaches to voting that rely on perceptions, evaluations, appraisals, or whatever else we call them, of parties measured contemporaneously in surveys, while routinely assuming that causality flows in only one direction.
The remaining two points raised by Whiteley et al. are also interconnected:
Fourth, there are methodological problems with their structural equation model and their statistical analysis. Finally, their approach to exogeneity testing is misleading and the data they employ for this purpose is inadequate to the task (2015).
Model Estimation
Whiteley et al. criticize our model for not including contemporaneous values, arguing that this results in a biased estimation.Footnote 3 However, we are pains to acknowledge in the paper that cross-lagged dynamic structural models do not allow for non-recursive contemporaneous effects and that we are only estimating “Granger causality.” Even if the data generating process did contain such contemporaneous effects, we would still obtain unbiased estimates of cross-lagged effects from a reduced form model. For example, consider the following data generating process for two variables y1t and y2t:
These two variables are a contemporaneous function of each other and their own lags in the previous period. We can apply the following algebraic transformation of y1t:
This can be re-written more concisely as
Note that this models y1t as a function of a lag of y2t, with a12=b11b22/(1−b11b12). This is called a reduced form model. The estimate of a12 (which we obtain from our cross-lagged structural models) is not biased, it just would not be the causal effect(s), which are b11 and b12. Unfortunately, it is not possible to calculate b11 and b12 from the parameters estimated in reduced form unless you can make some sort of identifying assumption, such as b12=0. Note that such an assumption amounts to making a causal assumption that y1t has no contemporaneous effect on y2t. In short, causal effects cannot be estimated without some sort of causal assumption.
This is important because Whiteley et al. propose using a vector error correction model (VECM) as an alternative to our model. However, they do not mention that the VECM is also a reduced form model. It is simply an algebraic transformation of a vector autoregression (VAR) model, which does not explicitly estimate contemporaneous relationships and the contemporaneous relationships cannot be calculated from the parameters of the VECM (or the equivalent VAR) without making an (causal) identifying assumption. There is no suggestion that Whiteley et al. have done this. They have provided a more complicated model, which does nothing to resolve the problem for which they criticize our paper. It may resolve the issue of the data being first-order integrated I(1) if the data were in fact I(1), but even then it still does not resolve the causal problem.
On page 15, Whiteley et al. note that VARs cannot incorporate long-term effects and then later suggest that this is why a VECM is required. This is perplexing. The VECM is an algebraic transformation of the VAR. If the data are all stationary, the VAR and the VECM produce exactly the same short-term and long-term effects, although an algebraic calculation may be required. If the data are integrated, the VAR cannot be used, but this is an estimation problem and has nothing to do with its ability to incorporate long- or short-term effects. The dynamics of a model with stationary data are different than those of a model with integrated data, but this is an issue of stationary versus integrated and not one of VAR versus VECM.
Whiteley et al. claim that because the data appear to be trending, it is likely first-order integrated I(1). However, there are many sources of trending data and an I(1) process is just one possibility. In fact, on observation the data do not appear to be simply I(1). Another possibility is that the data series contain deterministic trends (which is probable). A further possibility is that the data are I(1) with drift and yet another possibility is that the data are I(1) with trend. Because of these possibilities, the Dickey–Fuller test used to test if the data are I(1) must account for the possibilities of trends and drifts. This also has to be done for any test of cointegration. It does not appear that Whiteley et al. have done this, making their testing incomplete and unconvincing. Some, in fact, would argue that as some of these data series are in proportion they are bounded and cannot be truly I(1). This is important because it is the argument that the data are cointegrated that is used as a justification for the use of a VECM. Along the same lines, it is pretty clear that no trend (and there are a variety of possible versions of trends in a VECM) is accounted for in the VECM estimation. So again, the VECM results are flawed.Footnote 4
On Exogeneity Tests
Whiteley et al. claim that weak exogeneity is the sufficient exogeneity requirement for an unbiased estimation:
reliable estimates can be obtained from contemporaneous regressions as long as the key predictors are weakly exogenous. If variable y is thought to be explained by variable x, then x is said to be weakly exogenous to y if current values of y do not explain x (2015).
However, the definition given in their comment on our paper is that of contemporaneous exogeneity. Actually, it is not a very precise definition of contemporaneous exogeneity, but is closer to contemporaneous exogeneity than anything else. It is true that weak exogeneity is usually the sufficient exogeneity requirement (Engle, Hendry and Richard Reference Engle, Hendry and Richard1983), but contemporaneous exogeneity may or may not be. It depends on many other things (Wooldridge Reference Wooldridge2012). The important point here is that the conditions that Whiteley et al. spell out are not the conditions of weak exogeneity and may not therefore be sufficient for unbiased estimation.
Data (i): Aggregate Versus Individual Level
Surprisingly, Whiteley et al. make a plea to privilege aggregate data when evaluating their valence model. This is somewhat noteworthy, not only because aggregate data are likely to be subject to ecological fallacy, but also because Whiteley et al.’s own work—the very research that bolstered the valence model and led to such positive appraisals by commentators such as Johnston (Reference Johnston2011)—was based on face-to-face cross-sectional surveys, augmented by a small T large N panel analysis (i.e., Sanders et al. Reference Sanders, Clarke, Stewart and Whiteley2011). It now seems that Whiteley et al. reject the very basis of their own model, as reported in their major publications (Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2004; Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2009) and elsewhere (Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2011). Instead, they implore the use of aggregate survey data.
We do not fully understand Whiteley et al.’s reasoning here. There are strengths and weaknesses in both individual- and aggregate-level data. The strength of using aggregate data is primarily that we can often obtain a greater number of measured time points. In an ideal world, we would have individual-level data at many time points, but this is less common. Yet, the acknowledged weakness of aggregate-level data is always the potential for ecological fallacy. It almost seems as if Whiteley et al. are arguing that the possibility of committing an ecological fallacy is a strength of using aggregate data:
aggregate analyses of closely spaced time series observations in which individual idiosyncrasies are averaged out is likely to provide a more accurate picture of dynamics than do individual-level analyses (2015).
This assumes differences average out. However, our heterogeneity analysis shows very different processes between non-randomly selected types of voters and aggregate data do not enable the differentiation between different types of voters. Consider the assertion (Whiteley et al., p. 6) that: “The costs of information processing are much lower for valence compared with spatial issues” and thus open to use by the bulk of voters who are not motivated to take great interest in politics (a reiteration of a point they have made many times, for example, Whiteley et al. Reference Clarke, Sanders, Stewart and Whiteley2009, 30–52; Sanders et al. Reference Sanders, Clarke, Stewart and Whiteley2011). Our individual-level analysis shows that valence does not work for preference updating among the less politically informed. An aggregate analysis would simply obscure this fundamental problem with the assumptions underlying the theory. Thus, individual-level analysis allows us to demonstrate the inapplicability of the simplifying heuristic principle underlying the valence model by demonstrating systematic differences between individuals in updating processes. We leave judgment about the desirability of moving to ecological inference, with all of its well-known limitations, to the reader.
Likewise, the timing of surveys in relation to processes of voter decision making, as noted earlier, is not overcome with aggregate data. For some reason, Whiteley et al. seem to think that aggregate survey analysis gets beyond this problem:
It is important to note that the number of waves and the intervals between them in BES and other national inter-election panel surveys have been imposed by the practicalities of data collection and research funding constraints. Lags between adjacent waves of these panel surveys do not necessarily accord with—indeed, may have nothing to do with—the dynamics of voter decision-making (p. 12).
They thus assume that the lags between aggregate-level surveys do not also make this assumption, which is clearly not true. As noted earlier, our own approach to this issue has been to accept that this indeterminacy applies to both causal directions estimated in our models. This does not therefore disadvantage the valence model.
Data (ii): The Use of Cross-Sectional Survey Analysis?
A final somewhat perplexing claim in Whiteley et al.’s comment is that:
EC’s critique illustrates the kind of analysis that perpetuates theoretical and methodological problems that beset the field of electoral studies. In particular, their critique nicely illustrates the limitations of traditional election study data-gathering methods. Reluctant or unable to break free from the survey research techniques pioneered in Ann Arbor nearly 70 years ago, many political scientists do not conduct the very high-frequency, very large N, panel surveys that would provide needed leverage for studying the kinds of tightly inter-related dynamic systems of beliefs, attitudes and behaviour that drive individual and collective electoral choice and other forms of citizen political action (p. 21).
This is a rather odd point to make. As any reader can see, we utilize the large N panel studies implemented by Whiteley et al. in our analysis, and have likewise used earlier panel studies devised by previous election teams—which included Evans, who has also been involved in the design and implementation of panel studies of ~30,000 cases as part of the current BES team. Unsurprisingly, at no point in EC is there any reference to the use of a cross-sectional face-to-face survey. In fact, our argument is very much that we need to go beyond such data for these sorts of questions. There are, it should be made clear, questions of long-term social and political change that cross-sectional surveys (which have sampling frames, and are not plagued by panel attrition) are suited for, but the disentangling of short-term reciprocal relationships through time is clearly not a task for which they can be used.
Conclusion
In summary, for these reasons we reject Whiteley et al.’s criticisms of our paper. There are certainly points of agreement, in that both we and Whiteley et al. concur on the idea that leader images matter for voters. The analysis in EC shows this quite clearly, as do previous and ongoing analyses of leadership effects using a range of statistical techniques and both cross-sectional and panel data. However, crucially, our analysis shows that party preference dictates party performance evaluation, not vice versa. This is a major problem for a model of performance voting. Whiteley et al.’s response fails to dispel this problem. Indeed, by bundling together disparate elements that can influence voters’ decision making, such as leadership, performance on various issues and, of course, the economy, the valence model prevents the identification of elements that are consequential for party choice as distinct from those that are not.Footnote 5 The net impact and interconnection, or otherwise, of these distinct elements of “valence” evaluations is best identified via empirical analysis, not assertion.
You have
Access
We would like to thank Whiteley et al. (Reference Whiteley, Clarke, Sanders and Stewart2015) for a robust response to our critique of their influential valence model of party choice. As some parts of their paper address somewhat different concerns to those examined in our article, our reply focuses on those elements that connect most clearly with the arguments and evidence we have presented. To this end, we shall point to some of the problems in their critique and the possible concerns their current arguments raise for their previous work making the case for the valence model. We nonetheless welcome their willingness to engage in a scholarly debate on this important question.
So, what do Whiteley et al. claim? There are five key points, the first three of which are interconnected so we will consider them together:
First, EC do not actually test the valence politics model. The model is motivated by a theory of electoral choice, not a theory about the relationship between a generalized measure of “party preference”—a sort of “super partisanship”—on the one hand, and a composite measure of “party performance”, on the other. Second, they use inappropriate variables in their analyses in the sense that their measures do not appear in our specification of the valence politics model. Third, their argument provides no theoretical grounding for the key variable which they claim is strongly exogenous, i.e., partisanship (Whiteley et al. 2015).
Variable Selection
According to Whiteley et al. (2015): “the valence model has three key variables that drive voting behaviour and party support between elections: evaluations of party performance on valence issues, partisanship and leader images.” In our analysis, we focus on two key elements of this model: evaluations of party performance and leader images. Given this focus, Whiteley et al.’s criticism of our variable selection is so inaccurate as to raise concerns as to whether they have actually read the paper. They claim that: “key explanatory variables in our valence politics model are missing (e.g., leader images, party best on most important issue)” (Whiteley et al. Reference Whiteley, Clarke, Sanders and Stewart2015, 20). Clearly, as the reader can see, both leader images and “party best on most important issue” are in our models (Evans and Chzhen Reference Evans and Chzhen2015, 8).Footnote 1
Whiteley et al. also seem to think that we include party identity as an indicator of “party preference” (see footnote 1), which is not correct. Throughout their reply they repeatedly refer to our analysis as one focusing on partisanship:
The fundamental problem with EC’s critique is their subtle, yet seductive, invitation to seek a magic variable that could account for the observed dynamics in voting, leader images and party performance judgments. EC believe that the variable is partisanship which, by their account, drives all the other variables under consideration. However, if EC are right, then partisanship must itself exhibit dynamic properties (Whiteley et al., 2015).
However, we do not examine partisanship. Indeed, on pages 14–5 of our paper, we explain why we are not including party identity in our analysis and what would happen if we did. Moreover, we at no point say that the latent party preference measure was supposed to be “super-partisanship.”
Whiteley et al. elaborate at some length on this issue:
When we “round up the usual suspects”, use our standard conceptual and operational definitions of variables of interest, and employ appropriate methods for investigating EC’s claim that partisanship is strongly exogenous, the results strongly support our version of valence politics model. Partisanship is not strongly exogenous in this system (2015).
This means that the variable which is thought to anchor the analysis because it is a strongly exogenous determinant of the other variables has no plausible theoretical grounding. Partisanship has large-scale dynamics but where these come from is a mystery since EC do not account for this “moving mover” (2015).
However, we do not claim at any point that partisanship is strongly exogenous. Partisanship is not in our model. The whole premise of the Whiteley et al. reply with respect to “hunting the snark” is misconceived. Indeed, in other work (Evans and Chzhen under Reference Evans and Chzhenreview), we show why partisanship has no place in valence models—as we note in the paper.
From reading their critique, we can only conclude that when Whiteley et al. argue that we are not testing the “valence” model, they are right only in that we do not operationalize and estimate the very same model using the very same indicators they proposed in some of their work (specifications of the valence model vary even in Whiteley et al.’s own published analyses). However, if their valence model can only be specified in such a restricted way, it does not have a very wide applicability. Given the general claims made for the model, this is a major weakness.
Theoretical Bias?
Whiteley et al. also devote a substantial part of their response to pitting valence theory against spatial models, implying that we are in some way wedded to this particular alternative. That spatial considerations are themselves exogenous influences on vote choice is something we have demonstrated to be inaccurate except under certain conditions, using the type of panel evidence and analysis presented here (Evans and Andersen Reference Evans and Andersen2004).Footnote 2 As with hunting the snark, this is something of a straw man. Let us repeat that we are quite explicitly not privileging any approaches in our analyses of the endogeneity of survey predictors, or outcomes, of party choice. Our studies content themselves with exposing the limitations of the assumed exogeneity of “influences” on party choice such as economic perceptions (Evans and Andersen Reference Evans and Andersen2006; Evans and Pickup Reference Evans and Pickup2010; Pickup and Evans 2013) and issue proximities (Evans and Andersen Reference Evans and Andersen2004) and, in the paper discussed here, elements of the valence model itself. Though we do note that leadership images are relatively robust to our tests, both in this paper and in previous and ongoing work (Evans and Andersen Reference Evans and Andersen2005; Evans and Chzhen Reference Evans and Chzhen2013; Evans and Mellon Reference Evans and Mellon2015). We have also identified the conditions when economic perceptions are relatively more exogenous (Chzhen, Evans and Pickup Reference Chzhen, Evans and Pickup2014). It is likely that almost all survey variables are potentially endogenous under some conditions and ultimately we need to move beyond approaches to voting that rely on perceptions, evaluations, appraisals, or whatever else we call them, of parties measured contemporaneously in surveys, while routinely assuming that causality flows in only one direction.
The remaining two points raised by Whiteley et al. are also interconnected:
Fourth, there are methodological problems with their structural equation model and their statistical analysis. Finally, their approach to exogeneity testing is misleading and the data they employ for this purpose is inadequate to the task (2015).
Model Estimation
Whiteley et al. criticize our model for not including contemporaneous values, arguing that this results in a biased estimation.Footnote 3 However, we are pains to acknowledge in the paper that cross-lagged dynamic structural models do not allow for non-recursive contemporaneous effects and that we are only estimating “Granger causality.” Even if the data generating process did contain such contemporaneous effects, we would still obtain unbiased estimates of cross-lagged effects from a reduced form model. For example, consider the following data generating process for two variables y1t and y2t:
These two variables are a contemporaneous function of each other and their own lags in the previous period. We can apply the following algebraic transformation of y1t:
This can be re-written more concisely as
Note that this models y1t as a function of a lag of y2t, with a12=b11b22/(1−b11b12). This is called a reduced form model. The estimate of a12 (which we obtain from our cross-lagged structural models) is not biased, it just would not be the causal effect(s), which are b11 and b12. Unfortunately, it is not possible to calculate b11 and b12 from the parameters estimated in reduced form unless you can make some sort of identifying assumption, such as b12=0. Note that such an assumption amounts to making a causal assumption that y1t has no contemporaneous effect on y2t. In short, causal effects cannot be estimated without some sort of causal assumption.
This is important because Whiteley et al. propose using a vector error correction model (VECM) as an alternative to our model. However, they do not mention that the VECM is also a reduced form model. It is simply an algebraic transformation of a vector autoregression (VAR) model, which does not explicitly estimate contemporaneous relationships and the contemporaneous relationships cannot be calculated from the parameters of the VECM (or the equivalent VAR) without making an (causal) identifying assumption. There is no suggestion that Whiteley et al. have done this. They have provided a more complicated model, which does nothing to resolve the problem for which they criticize our paper. It may resolve the issue of the data being first-order integrated I(1) if the data were in fact I(1), but even then it still does not resolve the causal problem.
On page 15, Whiteley et al. note that VARs cannot incorporate long-term effects and then later suggest that this is why a VECM is required. This is perplexing. The VECM is an algebraic transformation of the VAR. If the data are all stationary, the VAR and the VECM produce exactly the same short-term and long-term effects, although an algebraic calculation may be required. If the data are integrated, the VAR cannot be used, but this is an estimation problem and has nothing to do with its ability to incorporate long- or short-term effects. The dynamics of a model with stationary data are different than those of a model with integrated data, but this is an issue of stationary versus integrated and not one of VAR versus VECM.
Whiteley et al. claim that because the data appear to be trending, it is likely first-order integrated I(1). However, there are many sources of trending data and an I(1) process is just one possibility. In fact, on observation the data do not appear to be simply I(1). Another possibility is that the data series contain deterministic trends (which is probable). A further possibility is that the data are I(1) with drift and yet another possibility is that the data are I(1) with trend. Because of these possibilities, the Dickey–Fuller test used to test if the data are I(1) must account for the possibilities of trends and drifts. This also has to be done for any test of cointegration. It does not appear that Whiteley et al. have done this, making their testing incomplete and unconvincing. Some, in fact, would argue that as some of these data series are in proportion they are bounded and cannot be truly I(1). This is important because it is the argument that the data are cointegrated that is used as a justification for the use of a VECM. Along the same lines, it is pretty clear that no trend (and there are a variety of possible versions of trends in a VECM) is accounted for in the VECM estimation. So again, the VECM results are flawed.Footnote 4
On Exogeneity Tests
Whiteley et al. claim that weak exogeneity is the sufficient exogeneity requirement for an unbiased estimation:
reliable estimates can be obtained from contemporaneous regressions as long as the key predictors are weakly exogenous. If variable y is thought to be explained by variable x, then x is said to be weakly exogenous to y if current values of y do not explain x (2015).
However, the definition given in their comment on our paper is that of contemporaneous exogeneity. Actually, it is not a very precise definition of contemporaneous exogeneity, but is closer to contemporaneous exogeneity than anything else. It is true that weak exogeneity is usually the sufficient exogeneity requirement (Engle, Hendry and Richard Reference Engle, Hendry and Richard1983), but contemporaneous exogeneity may or may not be. It depends on many other things (Wooldridge Reference Wooldridge2012). The important point here is that the conditions that Whiteley et al. spell out are not the conditions of weak exogeneity and may not therefore be sufficient for unbiased estimation.
Data (i): Aggregate Versus Individual Level
Surprisingly, Whiteley et al. make a plea to privilege aggregate data when evaluating their valence model. This is somewhat noteworthy, not only because aggregate data are likely to be subject to ecological fallacy, but also because Whiteley et al.’s own work—the very research that bolstered the valence model and led to such positive appraisals by commentators such as Johnston (Reference Johnston2011)—was based on face-to-face cross-sectional surveys, augmented by a small T large N panel analysis (i.e., Sanders et al. Reference Sanders, Clarke, Stewart and Whiteley2011). It now seems that Whiteley et al. reject the very basis of their own model, as reported in their major publications (Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2004; Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2009) and elsewhere (Clarke et al. Reference Clarke, Sanders, Stewart and Whiteley2011). Instead, they implore the use of aggregate survey data.
We do not fully understand Whiteley et al.’s reasoning here. There are strengths and weaknesses in both individual- and aggregate-level data. The strength of using aggregate data is primarily that we can often obtain a greater number of measured time points. In an ideal world, we would have individual-level data at many time points, but this is less common. Yet, the acknowledged weakness of aggregate-level data is always the potential for ecological fallacy. It almost seems as if Whiteley et al. are arguing that the possibility of committing an ecological fallacy is a strength of using aggregate data:
aggregate analyses of closely spaced time series observations in which individual idiosyncrasies are averaged out is likely to provide a more accurate picture of dynamics than do individual-level analyses (2015).
This assumes differences average out. However, our heterogeneity analysis shows very different processes between non-randomly selected types of voters and aggregate data do not enable the differentiation between different types of voters. Consider the assertion (Whiteley et al., p. 6) that: “The costs of information processing are much lower for valence compared with spatial issues” and thus open to use by the bulk of voters who are not motivated to take great interest in politics (a reiteration of a point they have made many times, for example, Whiteley et al. Reference Clarke, Sanders, Stewart and Whiteley2009, 30–52; Sanders et al. Reference Sanders, Clarke, Stewart and Whiteley2011). Our individual-level analysis shows that valence does not work for preference updating among the less politically informed. An aggregate analysis would simply obscure this fundamental problem with the assumptions underlying the theory. Thus, individual-level analysis allows us to demonstrate the inapplicability of the simplifying heuristic principle underlying the valence model by demonstrating systematic differences between individuals in updating processes. We leave judgment about the desirability of moving to ecological inference, with all of its well-known limitations, to the reader.
Likewise, the timing of surveys in relation to processes of voter decision making, as noted earlier, is not overcome with aggregate data. For some reason, Whiteley et al. seem to think that aggregate survey analysis gets beyond this problem:
It is important to note that the number of waves and the intervals between them in BES and other national inter-election panel surveys have been imposed by the practicalities of data collection and research funding constraints. Lags between adjacent waves of these panel surveys do not necessarily accord with—indeed, may have nothing to do with—the dynamics of voter decision-making (p. 12).
They thus assume that the lags between aggregate-level surveys do not also make this assumption, which is clearly not true. As noted earlier, our own approach to this issue has been to accept that this indeterminacy applies to both causal directions estimated in our models. This does not therefore disadvantage the valence model.
Data (ii): The Use of Cross-Sectional Survey Analysis?
A final somewhat perplexing claim in Whiteley et al.’s comment is that:
EC’s critique illustrates the kind of analysis that perpetuates theoretical and methodological problems that beset the field of electoral studies. In particular, their critique nicely illustrates the limitations of traditional election study data-gathering methods. Reluctant or unable to break free from the survey research techniques pioneered in Ann Arbor nearly 70 years ago, many political scientists do not conduct the very high-frequency, very large N, panel surveys that would provide needed leverage for studying the kinds of tightly inter-related dynamic systems of beliefs, attitudes and behaviour that drive individual and collective electoral choice and other forms of citizen political action (p. 21).
This is a rather odd point to make. As any reader can see, we utilize the large N panel studies implemented by Whiteley et al. in our analysis, and have likewise used earlier panel studies devised by previous election teams—which included Evans, who has also been involved in the design and implementation of panel studies of ~30,000 cases as part of the current BES team. Unsurprisingly, at no point in EC is there any reference to the use of a cross-sectional face-to-face survey. In fact, our argument is very much that we need to go beyond such data for these sorts of questions. There are, it should be made clear, questions of long-term social and political change that cross-sectional surveys (which have sampling frames, and are not plagued by panel attrition) are suited for, but the disentangling of short-term reciprocal relationships through time is clearly not a task for which they can be used.
Conclusion
In summary, for these reasons we reject Whiteley et al.’s criticisms of our paper. There are certainly points of agreement, in that both we and Whiteley et al. concur on the idea that leader images matter for voters. The analysis in EC shows this quite clearly, as do previous and ongoing analyses of leadership effects using a range of statistical techniques and both cross-sectional and panel data. However, crucially, our analysis shows that party preference dictates party performance evaluation, not vice versa. This is a major problem for a model of performance voting. Whiteley et al.’s response fails to dispel this problem. Indeed, by bundling together disparate elements that can influence voters’ decision making, such as leadership, performance on various issues and, of course, the economy, the valence model prevents the identification of elements that are consequential for party choice as distinct from those that are not.Footnote 5 The net impact and interconnection, or otherwise, of these distinct elements of “valence” evaluations is best identified via empirical analysis, not assertion.