DO WE DISAGREE?

Before addressing the methodological points of critique raised by Górecki, we pose a very simple question: Do we actually disagree on the effect of the Polish gender quota in 2011? Górecki's response seems to imply so, but we do not think that this is the case.

Despite the different modeling strategies, our study and the one by Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) come to similar conclusions about the impact of the gender quota on woman's electoral success. For the election of 2007, in which the gender quota was not implemented, Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74) found no statistically significant effect of gender for any party. We reported very similar findings, and the only difference was that some of our gender effects with respect to the Sejm elections 2007 were negative, whereas all of the 2007 gender effects reported in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74) were positive.Footnote ³ However, in both analyses, the effects were not statistically significant for all parties, which implies that a candidate's gender was not a decisive candidate characteristic in the election of 2007.

After the implementation of the gender quota in the election of 2011, we reported statistically significant negative gender effects for three of four parties included in our analysis. The fourth gender effect (for the left-wing party Democratic Left Alliance or SLD) was also negative, but the effect was not statistically significant. Therefore, all of the gender effects we identified for the election of 2011 pointed in the same direction, and most of them were significant. Again, this finding is in line with the results reported in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74), who also reported negative effects for all parties. In sum, both studies agree that gender was more or less irrelevant in the election of 2007 and that there were negative effects for female candidates in the election of 2011 (see the comparison of the results in Table 1).

Table 1.

Comparison of gender effects reported in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) and Jankowski and Marcinkiewicz (Reference Jankowski and Marcinkiewicz2017)

Note: A positive (negative) effect means that women received, on average, more (fewer) votes than men. Entries for Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) are based on their ‘basic model’ (see Górecki and Kukołowicz Reference Górecki and Kukołowicz2014, 74). In their ‘full model,’ the effects are only significant for two of the four parties, and effect sizes are substantially smaller. A comparison between the two studies is presented only for the four parties included in both analyses: Civic Platform (PO), Law and Justice (PiS), Polish People's Party (PSL), and Democratic Left Alliance (SLD).

The magnitude of the reported negative gender effect in 2011 (and consequently the framing of the article) is at least somewhat different between the two articles. We detected a negative effect for women between −.005 and −.01, meaning that a female candidate received a 0.5% to 1% lower list vote share after the introduction of the gender quota. To make these quantities comparable to the effects reported in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74), our effects must be multiplied by the number of votes cast for a list. For example, on a list with 100,000 voters, we would expect the number of votes cast for a female candidate to be 500 to 1,000 votes lower compared to those for a male candidate.Footnote ⁴ Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) reported much stronger negative effects, at least in some of their models. As a consequence, they put more emphasis on these strong negative effects, whereas we described our findings conservatively.Footnote ⁵

As a consequence, the debate largely breaks down to a discussion of different effect sizes. Górecki argues that the small effect sizes reported in our analysis were potentially caused by “unobvious, debatable and potentially controversial” (Górecki, Reference Górecki2019) methodological choices. We agree with Górecki (Reference Górecki2019) that the methodological differences between both studies explain the different effect sizes, but we disagree that the approach chosen in our original analysis is as problematic as he claims. Later in this article, we discuss in detail our disagreement with the methodological critique of Górecki (Reference Górecki2019). Here, we only want to demonstrate that the different model specifications have important consequences for the debate about the different effect sizes in both articles. In Table 1, we compare our results to the model specification of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) in which they found the by far strongest negative effect sizes for female candidates in the election of 2011 but in which they only controlled for a very limited set of potential confounders. This model specification is called the ‘basic model’ in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014). But these large effect sizes vanish in the so-called ‘full model’ of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) in which they additionally control for candidate's background and list position. Under this model specification, the effect sizes are substantially smaller for all parties and the effect of gender is statistically significant for only one of the four relevant parties that we also analyzed in our 2017 article. In Table 2, we compare these different effect sizes to the results reported in our article.Footnote ⁶ This comparison makes clear that (a) the results by Górecki and Kukołowicz strongly depend on a particular ‘basic’ model specification and (b) that the results from Górecki and Kukołowicz's (Reference Górecki and Kukołowicz2014) “full model” are comparable to our results.

Table 2.

Marginal effects for female candidates as reported in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) and Jankowski and Marcinkiewicz (Reference Jankowski and Marcinkiewicz2017)

Note: Table reports marginal effects for female candidates on the number of votes received in the election of 2011 in Poland. Estimates for the models of Górecki and Kukołowicz are taken from Table 4 in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74). Estimates for Jankowski and Marcinkiewicz are computed by multiplying the marginal effect for the list vote share (as reported in Jankowski and Marcinkiewicz Reference Jankowski and Marcinkiewicz2017) with the respective average number of votes for a party list. Confidence intervals (95%) are shown in brackets.

Put differently, the article by Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) itself provides substantial evidence that the effect of gender strongly depends on certain model specifications. Therefore, in the remainder of the present article, we do two things. On the one hand, we respond to Górecki's (Reference Górecki2019) critique of our model specification and justify it in more detail. On the other hand, in discussing Górecki's (Reference Górecki2019) critique, we also highlight some major drawbacks of Górecki and Kukolwoicz's (Reference Górecki and Kukołowicz2014) methodological approach.

THE CHOICE OF THE DEPENDENT VARIABLE: VOTE COUNTS OR VOTE SHARES?

Górecki mentions a number of points as to why he is convinced that using the raw number of votes as the dependent variable is a reasonable choice (the approach chosen in Górecki and Kukołowicz Reference Górecki and Kukołowicz2014) and is potentially superior to using a candidate's list vote share (our approach). In general, we think that this choice is not that consequential. Within each list, the vote shares and the raw number of votes are perfectly correlated; thus, it would be rather surprising if using one of the two measures led to drastically different results. However, Górecki sees many advantages to using the raw number of votes that we perceive to be disadvantages. In response to Górecki's critique, we focus on one aspect we consider the strongest drawback of using the raw number of votes as dependent variable.

Górecki argues that one has to control for the number of votes cast for a list when using the raw number of votes for a candidate as the dependent variable because the number of votes received by a candidate depends on the total number of votes that have been cast for a list. This is correct. However, this perspective ignores that the marginal effects obtained from the regression model can only be meaningfully interpreted when compared to the total number of votes cast for the list. For example, Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) report an effect size of −2,473 votes for women on the list of the liberal party Civic Platform (PO) in the election of 2011 (Table 2). It is very challenging to interpret this effect size substantially without knowing how many votes have been cast for a list. For example, the effect size would probably be interpreted as large when the total number of list votes is 10,000, but the effect size would be considered rather small for a list with 300,000 votes. This poses a challenge because the number of votes cast for a party varies considerably between districts; for example, in 2011 the Civic Platform received between 63,000 votes in the Eastern Polish Chełm district and 498,000 votes in Warsaw 1. Therefore, the effect size can only be interpreted meaningfully when being compared to the total number of votes cast for the list.

The problem not only affects the interpretation but also the estimation of the effects. The example in the previous section makes clear that the effect size depends on the total number of votes cast for a list. When fewer (or more) votes have been cast for a list, we can expect that the effect size will also be smaller (or larger). Otherwise, the effect size is not proportional to the total number of votes cast for a list, which would imply that the effect is actually different depending on the number of votes cast for the list.Footnote ⁷

This characteristic regarding the effect size when using the raw number of votes as the dependent variable has important implications for model estimation. It implies that the effects are actually a function of the total number of votes cast for a list and that there is a strong interaction effect between the total number of votes cast for a list and the gender effect. Thus, it is not sufficient to simply control for the total number of votes cast for a list. Instead, an interaction effect should be included between the total number of votes for a list and the gender variable.

To demonstrate empirically that this heterogeneity in the effects exists, we replicated the models by Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74), but we included an interaction term between the gender effect and the total number of votes cast for a list. Figure 1 displays the results of this interaction. The effects were indeed stronger (weaker) the more (fewer) votes a list received. Notably, this heterogeneity does not mean that women were more disadvantaged on lists with more votes. Instead, the observed effect heterogeneity is artificial and results from the choice of the dependent variable. Furthermore, this artificial effect heterogeneity exists for every variable included in the model and could only be accounted for by including an interaction between the total number of votes for the list and each independent variable. This method would lead to overwhelmingly complex model specification.

Estimating a model with artificially induced effect heterogeneity should be avoided because it can lead to serious problems. Suppose, for example, that the effect of gender is truly heterogeneous with respect to the number of votes cast for a list (or a variable that strongly correlates with the total number of list votes, e.g., the number of candidates on a list). It becomes very challenging to analyze such effect heterogeneity because the effects are already artificially heterogeneous. Therefore, we are convinced that using the raw number of votes as the dependent variable is not a better choice compared to using the list vote share. The list vote share, by design, already accounts for the total number of votes cast for a list. Consequently, no artificial effect heterogeneity exists when using this approach.Footnote ⁸

Figure 1.

Replication of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014): Interaction effect between total number of votes for a list and gender. Note: This is a rough replication of the ‘basic model’ described in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) for the election of 2011, but with an additional interaction effect between the total number of list votes and the gender of a candidate. The plot displays the marginal effect of gender conditional on the total number of votes cast for a list (divided by 10,000). As expected, the effect of gender strongly depends on the number of votes cast for a list. This heterogeneity does not necessarily imply that the effect of gender becomes stronger. Instead, this heterogeneity occurs naturally when using the number of votes as the dependent variable. The x-axis in the plot reflects the observed range of the total number of votes cast for a list for each party.

We are not claiming, however, that accounting for this effect heterogeneity in the study of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) would necessarily lead to different results with respect to the effect of gender in the elections of 2007 and 2011. Instead, our argument is that there are important drawbacks of the model specification chosen by Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) that have more general implications for the analysis of vote casting patterns in open-list PR systems.

Finally, we want to respond directly to Górecki's argument that modeling the raw number of votes takes the different performances of parties into account. Of course, we agree with his argument that it might be better for a candidate to receive 5% of the list vote share when party magnitude is large compared to receiving 10% of all list votes when party magnitude is small. But we disagree that this is relevant for the analysis. We are not interested in the question of whether a candidate is eventually elected or not. If we were interested in this question, we could have modeled this directly by regressing being elected on a set of independent variables. Górecki writes that “[w]hen calculating party-list vote shares, the only thing that is actually taken into account is a candidate's performance relative to her district-level running mates.” We fully agree. We are interested in the question of whether a female candidate performs better or worse compared to an otherwise similar candidate from the same (or a very similar) list.Footnote ⁹ Therefore, using the list vote shares captures the concept we are interested in (i.e., intraparty competition) very well.

LIST PLACEMENT: ENDOGENEITY VERSUS OMITTED VARIABLE BIAS?

Górecki claims that our analysis suffers from the fact that we controlled for the ballot position of a candidate. Instead, he advocates controlling only up to a certain degree for the ballot position because the ballot position is a potentially endogenous predictor, and thus, the estimated effects are potentially biased. We found this critique a bit surprising given that Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 76) already discussed the potential endogeneity problem of ballot position effects. In their article, they concluded that they “do not believe the potential endogeneity distorts … [the] results substantially” (Górecki and Kukołowicz Reference Górecki and Kukołowicz2014, 76). They also provided several arguments against the endogeneity of the ballot position, such as surveys in which 16% of Polish voters admit to relying on the ballot position as an information shortcut when selecting a candidate.Footnote ¹⁰ In his response to our article, Górecki is now much more critical of ballot position effects and describes them as “overwhelmingly endogeneous” that only “might contain some genuinely causal components” (Górecki Reference Górecki2019). We disagree with Górecki's interpretation of this strong endogeneity and agree more with the original arguments presented in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 76). In the following section, we explain why.

In his response, Górecki interprets the existing literature as providing only very little evidence for a causal component of the ballot position. We agree that many studies of ballot position effects are not based on a causal identification strategy, and therefore, the reported effects should be interpreted with caution.Footnote ¹¹ However, some studies are based on a causal identification strategy. A very good and recent example from a relatively high-salience election, Bavarian state elections in Germany, is provided by Däubler and Rudolph (Reference Däubler and Rudolph2018). They rely on a quasi-experiment and find strong causal evidence for ballot position effects. In fact, they estimate that “approximately every second vote is altered due to ballot position” and that “ballot order affects both the sequence of consideration and the assessment of individual candidates” (Däubler and Rudolph Reference Däubler and Rudolph2018, 24). They also demonstrate that a ballot position effect can be observed up to the fifteenth ballot position but that the effects are substantially stronger for top-placed candidates. In their supplemental material, Däubler and Rudolph (Reference Däubler and Rudolph2018) further demonstrate that approximately 75% of all votes cast for the first ballot position can be attributed to the ballot-position effect. From this perspective, Górecki's claim that the ballot position is “overwhelmingly” endogenous appears highly disputable.Footnote ¹²

Given such evidence, we think it is reasonable to argue that more than just “some genuinely causal compenent” (Górecki, Reference Górecki2019) is captured by the ballot position and that the effect is not “overwhelmingly” endogenous. However, we also see that the inclusion of the ballot position does not only capture true ballot position effects. We agree with Górecki's argument that parties tend to nominate candidates to top positions that are more likely to attract a large number of preferential votes. Thus, the strong effects observed for the ballot position are not just due to the fact that voters simply prefer to vote for candidates with a better list placement. Instead, these candidates also receive more votes because they are often of higher quality than their lower placed list competitors. In this regard, the ballot position is highly informative about candidate characteristics that are very challenging to control for. Usually, one does not have access to detailed information about a candidate's quality (aside from factors such as incumbency). The nice aspect of the ballot position is that it probably captures many of these factors (e.g., the rhetorical abilities of a candidate, the number of supporters within the party, and so on). In this regard, the relationship between a candidate's list placement and electoral success is definitely not causal on its own but also reflects all these additional aspects that arise from differences in candidate quality. Therefore, controlling for the ballot position captures various different aspects regarding a candidate's quality that are all worth controlling for but cannot be directly observed.

However, controlling for the ballot position can be a problem for the identification of the gender effect when parties nominate women of similar quality to less favorable ballot positions than men, that is, if parties are systematically biased against female candidates in their nomination strategies. If this is the case, then controlling for the ballot position leads to a situation in which we are essentially comparing more qualified women to less qualified men. Therefore, we acknowledge that there is a potential mechanism of how controlling for the ballot position might underestimate the gender effect, as argued by Górecki (Reference Górecki2019). However, this mechanism strongly depends on the assumption that parties are indeed biased against female candidates in their nomination strategies. Whether this is actually the case cannot be comprehensively tested here, but we can provide a few arguments that speak against this assumption. First, the fact that women were placed more frequently in less favorable ballot positions in the election of 2011 cannot be used as evidence for a gender bias in candidate nomination patterns. Many of the newly nominated women lacked political experience and were running for a national office for the first time. As a consequence, it is only natural that these women were not directly placed in favorable ballot positions. Second, in the prequota period, there is very little evidence that women were placed systematically in less favorable positions compared to men.Footnote ¹³ Such patterns should have been observed if women were indeed disadvantaged in the nomination process. Third, in contrast to a negative gender bias in candidate nomination, at least the Civic Platform (PO) has adopted an internal voluntary gender quota in 2011 that went beyond the official quota regulations and provided women with an additional advantage in list placement. If at all, women seem to be advantaged and not disadvantaged in this party. Given these arguments, there is little evidence of a large gender bias in parties' candidate nomination strategies.

Górecki also mentions the problem of ignoring the ballot position in the analysis because this likely causes an omitted variable bias. Therefore, he argues that it can make sense to control for the ballot position of a candidate. His concern regarding our analysis is, however, that we controlled too much for the ballot position by including dummy variables for the first four positions and a continuous predictor for the remaining positions (see Jankowski and Marcinkiewicz Reference Jankowski and Marcinkiewicz2017, 19). In contrast, Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) used a rather rough indicator for a candidate's list placement. Their measure for list placement only distinguished between a ‘viable’ and ‘nonviable’ position, whereas ‘viable’ was defined as every ballot position that was smaller or equal to twice the party magnitude. So, for example, the first and the twentieth ballot positions were treated equally in the analysis of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) when the party magnitude equalled 10. Given the vote distribution patterns in open-list PR systems, it is highly debatable whether such an operationalization of the ballot position is appropriate.Footnote ¹⁴ The operationalization of the ballot position in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) also contrasts with the argument provided by Górecki (Reference Górecki2019) that ballot position effects are “largely limited to the top (first) position” (Górecki Reference Górecki2019). If this is the case, then it makes sense to control intensively for the top ballot positions by including dummy variables. However, this was not done in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014), where a large number of ballot positions were often treated as ‘viable.’

Finally, Górecki argues that a trade-off exists between a model specification with the potentially endogenous ballot position predictor or an omitted variable bias due to the exclusion of the ballot position. In the previous section, we argued that we do not think that the endogeneity problem is so severe. Here, we emphasize that the omitted variable bias, on the contrary, is potentially quite severe. That an omitted variable bias exists when excluding the ballot position from the analysis is rather obvious. We and Górecki agree that the gender quota has led parties to place additional female candidates predominantly in the middle or close to the lower end of the party lists. This means that gender and list placement are strongly correlated. Given how votes under open-list PR are distributed, there can also be no doubt that the ballot position and electoral success are correlated (although Górecki puts the causal relationship into question). Therefore, if one does not control for the ballot position, one ends up comparing dissimilar entities in the regression analysis. Essentially, one compares the electoral success of a top-placed male candidate to a female candidate closer to the end of the ballot paper or vice versa. This concept contrasts starkly with the fundamental idea that effects should be estimated by comparing candidates that are most similar. In our study, we followed this principle as closely as possible; we wanted to compare the electoral success of a woman with the electoral success of a man placed in comparable ballot positions and who were also comparable in regard to other candidate characteristics (e.g., age). We think that the severity of the omitted variable bias caused by controlling inefficiently or even not at all for the ballot position is already apparent in the analysis of Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014, 74). Comparing their ‘basic model’ and their ‘full model,’ the effect of gender decreased drastically, up to −90% for the Civic Platform (PO).Footnote ¹⁵ Such large instability in the estimated effects should be a strong warning sign that the effects in the basic model did not capture the effect of a candidate's gender. In fact, it can be seen as evidence of how strongly the results in Górecki and Kukołowicz (Reference Górecki and Kukołowicz2014) were dependent on a certain model specification.

In sum, there are good reasons why one has to control rather extensively for the ballot position of candidates to avoid an omitted variable bias. We acknowledge that there might be an endogenous component of the ballot position, but this problem is not as severe as described in Górecki's critique. Even if it is, then we should start thinking about how to address the endogeneity problem appropriately, for example, by using an instrumental variable approach. Simply excluding the ballot position would have led to omitted variable bias and thus to biased estimates.