1. INTRODUCTION
This paper is concerned with how an individual decision maker updates his or her beliefs in the light of others’ beliefs. Its core is an empirical investigation where we vary how much the decision maker knows about other people’s beliefs: whether they know what course of action others have taken (in this case how a problem is solved) and whether they know why this course of action was taken (why a particular solution is preferred). It is thus a paper about judgement aggregation; how individuals aggregate others’ judgements and their own previous judgement into a new revised one. However, since group decisions can be substituted with individual decisions based on information from group members, results concerning these are also relevant.
Conceptual research into fields such as judgement aggregation, deliberation and testimony ask the normative question how beliefs should be combined in order to track truth, or yield optimal outcomes. In this context, results are generated through proofs and simulations. Whenever this is the case, substantial simplification is required. Simplifying assumptions always run the risk of glossing over important distinctions. For instance, most research focuses on access to others’ beliefs (the what above) and not on the reasons for the beliefs (the why). It has, however, been found that the truth tracking capacity of majority votes based on premises differ from that of majority votes based on conclusions and this seems to generalize also to other decision strategies (Bovens and Rabinowicz Reference Bovens and Rabinowicz2004; List and Puppe Reference List, Puppe, Anand, Puppe and Pattanaik2009).
Collective wisdom, as it is described by Helene Landemore (Landemore and Elster Reference Landemore and Elster2012), is nowadays considered to be primarily a matter of nondeliberative judgement aggregation based on three separate, but related, mechanisms: the Condorcet jury theorem (Condorcet Reference Condorcet1785), numerical belief aggregation cancelling noise (Sunstein Reference Sunstein2006), and cognitive diversity results, indicating that a diverse group under some circumstances outperforms a homogenous, more competent, group, since group members’ prediction models in the former case are unlikely to make identical errors (Hong and Page Reference Hong and Page2004).
In what follows we will use the terminology of epistemic and non-epistemic social information in order to capture, but broaden, the difference between premise- and conclusion-based decisions. Epistemic social information, in our terminology, denotes the reasons accompanying advice or opinions (i.e. I recommend action X, since I believe Y), whereas non-epistemic social information stands for mere recommendations (do X), or observations (seeing someone else choose X). Psychological research indicates that decision makers are sensitive to this difference. For instance, advisors that can explain how they arrived at their recommendations are preferred over those that cannot (Jonas and Frey Reference Jonas and Frey2003; Yates et al. Reference Yates, Price, Lee and Ramirez1996), and advice accompanied by irrelevant reasons is given less weight than advice with no accompanying reasons (Simonson et al. Reference Simonson, Nowlis and Simonson1993).
We know from conceptual work that majority votes based on epistemic social information (premises) behave differently than ones based on non-epistemic social information (conclusions). It even seems as if the premise-based procedures often are better truth trackers than the conclusion-based ones are (Bovens and Rabinowicz Reference Bovens and Rabinowicz2004). If this is true, and if people are reasonably rational, we should expect decision makers to be sensitive to epistemic social information, to prefer to use it, and perhaps also to be more successful when they do so. The matter is, however, far from settled. For instance, participants confronted with the doctrinal paradox (pitting premise- and conclusion-based procedures against each other) appear to prefer whichever procedure is positively framed, that is, leads to a good outcome for the defendant (Bonnefon Reference Bonnefon2007, Reference Bonnefon2010). Moreover, a number of results in judgement aggregation are based on majority votes and different implementations of the Condorcet jury theorem. These results depend on simplifying assumptions, such as independence and competent voters (Dietrich and Spiekermann Reference Dietrich and Spiekermann2013). The same assumptions are important for numerical belief aggregation to succeed (to become ‘the miracle of aggregation’), and, to some extent, for Hong and Page’s cognitive diversity results (Hong and Page Reference Hong and Page2004; Page Reference Page2007). Since independence is difficult to maintain in real life, and relaxing it diminishes the power of majority votes etc., we cannot count on judgement aggregation to be generally successful. In addition, the cognitive costs associated with majority rules are high since they require information about all group members. It is therefore interesting to see whether actual decision makers can take advantage of these mechanisms.
In psychological research, socially inspired decision-making is mainly studied in the context of group decision-making (Davis Reference Davis1973) or advice taking (Sniezek and Buckley Reference Sniezek and Buckley1995), where individual decision makers are given information about other individuals’ choices. We will follow the second tradition since it allows us to compare an individual decision maker’s pre-advice opinion to his or her post-advice opinion. The focus on epistemic social information will, however, make our experimental set up differ from that of traditional advice taking studies, where advice is typically quantitative, such as probabilities or price estimates, and fabricated. Simplifying advice allows researchers to get a firm grasp on the factors affecting advice taking, but it may also change how decisions are made. It has, for instance, been suggested that decision makers discount advice and give less weight to others’ opinions than to their own because of an informational asymmetry between decision maker and advisor (e.g. Yaniv and Kleinberger Reference Yaniv and Kleinberger2000; Yaniv Reference Yaniv2004). When we know little about the decision-making process of someone else, it is hard to know whether that person is our epistemic peer, that is, that we are equally rational, well-informed and conscientious. It appears, however, as if adding epistemic social information to the equation may decrease informational asymmetry, and thus increase the weight given to others’ views. We thus agree with Ilan Yaniv that understanding the role of qualitative (premise-based) advice is the next important step also for the psychological research tradition (cf. Yaniv and Milyavsky Reference Yaniv and Milyavsky2007).
The importance of qualitative advice is highlighted by the different outcomes of premise- and conclusion-based decision processes. It is important to note that including epistemic social information changes the scenery not only in relation to previous empirical work, but also in relation to judgement aggregation and collective wisdom. The Condorcet jury theorem is constructed for binary choice, and the miracle of aggregation and cognitive diversity models concern numerical predictions. When we investigate how individual decision makers use epistemic social information we will try to apply non-epistemic models (such as conformity, or majority rules) also to this type of advice.
In our experiments we vary access to epistemic social information and investigate to what extent it increases the weight given to others’ opinions and, more importantly, the way in which these opinions are combined with the decision makers’ own. We first ask participants to solve a problem task individually (the Wason selection task; for a description, see section 3) and then to re-solve it with access to either epistemic or non-epistemic social information while keeping track of improvement between rounds. Participants gain social information from each other and correct responses are rewarded. For each treatment we use model comparison techniques to evaluate which advice taking strategies participants use. We estimate whether strategies vary with access to epistemic social information, as well as parameters of those strategies that give weights of reliance on both epistemic and non-epistemic social information. In this way we hope to better understand the relationship between research on premise- and conclusion-based decision procedures and actual decisions made with access to epistemic and non-epistemic social information. The strategies are introduced in the following section, and come mainly from the social learning literature. The literature is handy since it provides us with exact mathematical models for advice taking, take cognitive costs into account, and investigate decision-making strategies with respect to their long-term effects. They are thus based on more realistic assumptions than the conceptual results described above.
2. ADVICE TAKING STRATEGIES FOR NON-EPISTEMIC AND EPISTEMIC SOCIAL INFORMATION
In this section, we present common candidate strategies for advice taking based on non-epistemic social information. We will not cite mathematical expressions here, but refer to original literature containing mathematical analysis. In the data analysis section we construct formal versions of these individual-based models of advice taking, and these formalizations can be found in section 3.2.2.
Formal models of social learning strategies suggest that individuals are predisposed to learn socially (i.e. take advice) when individual learning is costly (Boyd and Richerson Reference Boyd and Richerson1985; Rogers Reference Rogers1988). Costs may be due to information search, or because of risks associated with mistakes during learning. Individual learning should be preferred when costs are low and when tasks are easy, i.e. when errors are infrequent (for empirical results consistent with the formal predictions, see Deutsch and Gerard Reference Deutsch and Gerard1955; Baron et al. Reference Baron, Vandello and Brunsman1996). This is because social learning is likely to lag behind changes in the task environment. For instance, the information on which an advisor’s behaviour or opinion is based may be irrelevant at the time he or she is consulted. Nevertheless, a good amount of social learning is predicted to be on average adaptive, despite the risks associated with it.
Simple (linear) imitation. The simplest version of social learning leading to the above prediction is unbiased imitation: learning socially from an advisor when he or she is not chosen for any particular characteristic. The mathematical form of such a rule turns out to be linear in the frequency of advice (of behaviour that can be imitated). As a result, if individuals use this strategy, the expected proportions of the different behaviours do not change from one time period to the next. This is the most basic social learning strategy and we will refer to it as simple imitation.
Conformity. Under some circumstances, social learning can lead to more correct outcomes than individual learning does. Condorcet’s jury theorem (Condorcet Reference Condorcet1785) is an early demonstration of this fact. The intuition the theorem provides is that when each advisor has a better than random chance of arriving on their own at the correct, or appropriate, response to a task, and advisors’ opinions are independently formed, it is adaptive to adopt the majority response. Since effective individual and social learning strategies make more-adaptive behaviours increase, conformity is likely to be a widely used decision-making strategy. On the other hand, formal models of conformity indicate that the strategy will be less common for novel tasks, where advisors cannot be assumed to be competent (above chance), and when individual learning is cheap and efficient. In addition, when groups of individuals use a conformist strategy they risk lagging behind changes in the choice task (Henrich and Boyd Reference Henrich and Boyd1998; Kameda and Nakanishi Reference Kameda and Nakanishi2002). When the best response varies over time, perhaps due to changes in the environment, and each individual in the group conforms to the others, the group will not be able to track these changes. When the long-term effects of conformity are studied, as in this case, independence is, of course, violated. Models of conformity suggest that the dilemma is solved by individuals employing several strategies with different frequencies and at different times (Henrich and Boyd Reference Henrich and Boyd1998).
Other strategies. One often-discussed strategy that we will omit is payoff-biased imitation. There is evidence, both experimental and natural (Henrich and Gil-White Reference Henrich and Gil-White2001; Camerer Reference Camerer2003), and good theoretical justification (Boyd and Richerson Reference Boyd and Richerson1985; Schlag Reference Schlag1999, Reference Schlag1998) for the expectation that people bias their social learning by estimates of the payoffs others get from their choices. Since the experiments we present do not provide participants with information about the payoffs of other individuals, payoff-biased strategies are irrelevant to this paper.
Epistemic social information. Since social learning based on epistemic social information has not been previously investigated, there are no results to report here. We will construct social learning strategies for epistemic social information based on the strategies presented above, and these will be presented in section 4.2.2.
There are, however, several reasons for us to expect that epistemic social information will change social learning. In particular, adding epistemic social information may increase the likelihood that the advantages of deliberation are achieved, so that participants get access to more arguments and possible solutions than they produce themselves. Epistemic social information can also help them identify the better arguments, or, through the arguments, the better solutions (and be more influenced by these). Potentially epistemic social information could even make participants agree on the best solution to the task.
3. EXPERIMENT 1: INDIVIDUAL USE OF NON-EPISTEMIC SOCIAL INFORMATION
In our first experiment we present participants with a problem-solving task, and ask them to solve it first individually and then to re-solve it with or without access to other participants’ solutions (advice). We estimate the weight participants give to this non-epistemic social information, and the predominant advice taking strategy. Results will then be compared with Experiment 2 in which participants have access also to epistemic social information.
Since social learning is reduced when tasks are too easy, we needed a difficult experimental task that allows for a range of advice taking strategies. We also wanted a task in which both epistemic and non-epistemic social information could be provided, so that we could later design a comparable experiment for epistemic social information. The Wason selection task (Wason Reference Wason and Foss1966, Reference Wason1968) fulfils both of these requirements. In the original task participants are presented with a picture of four cards, lying face up, with letters or numbers printed on them (typically A, D, 4 and 7), and a statement of the form: ‘If a card has a vowel on one side, it will have an even number on the other side’. Participants are asked to select only the cards necessary to check in order to find out whether the statement is true. In what follows we will use the material implication, ‘if p then q’, to describe the task (we will use this as a handy abbreviation, but are aware that there are other interpretations of the statement). Typically the most common response to the task is to turn the vowel and the even number (p and q), but, according to Peter Wason, turning the vowel and the odd number (p and not-q) is the correct response. For simplicity, we will stick with Wason’s interpretation of the task, but which interpretation, if any, is correct is less important for our purposes. What matters is that participants can communicate behaviour – a vector of card selections – as well as justifications for this behaviour to other participants.
There are also other aspects of the task that makes it ideal for our purposes: Letting participants discuss card selections dramatically alters the distribution of responses to the task (Moshman and Geil Reference Moshman and Geil1998), while having individuals merely write down the reasons for their selection does not (Griggs Reference Griggs, Newstead and Evans1995), and in addition, the reasons given for particular solutions to the task are imperfectly related to participants’ actual solutions (Wason and Evans Reference Wason and Evans1975; Evans and Wason Reference Evans and Wason1976). Thus, providing epistemic social information does not seem to change performance on this task, while being exposed to it does.
We paid participants quite generously with approximately €2 or USD 3 (at the time of the experiments) for each correct card selection in the final round of the experiment. An individual who turned only p and not-q thus received €8 or USD 12 in total, for correctly turning two cards, and avoiding to turn the additional cards. We paid such rates to make sure that participants were highly motivated to reach the correct answer, and only used social information to improve their decision quality. In addition, all answers were turned in individually and privately.
3.1. Design and procedure
This first experiment consists of two treatments, a control treatment and a treatment involving non-epistemic social information. We used the control treatment to estimate patterns of individual decision making in our population. Thirty-two participants performed the Wason selection task, twice in succession (two attempts at solving the task, with 10 minutes available for each choice), and were rewarded for correct answers. They received no feedback about correct or incorrect card selections between round 1 and round 2, and were paid according to their choices in both rounds after the experiment had been concluded. As in all treatments we report here, participants learned of the entire procedure before beginning, and there was no deception.
In the non-epistemic social information treatment, participants were allowed to see the card selections of other individuals between rounds. This non-epistemic social information was collected and distributed anonymously. The procedure took place in front of participants so that it was clear to them that there was no deceit, and that responses were representative. We ran the treatment with two experimental group sizes (N), N = 19 (1 group) and N = 5 (4 groups). Otherwise procedures were identical for the control group and the non-epistemic group.
Participants were 71 students recruited through introductory classes in the humanities. They were given a show-up fee of €4 or USD 6 in addition to rewards for correct responses. No participant was familiar with the task prior to taking part in the experiment.
3.2. Results
We first present an outcome-based analysis that computes the expected card selections given initial selections. This analysis suggests that non-epistemic social information did not tend to move participants toward correct card selections (under Wason’s interpretation of the task). Instead, overall patterns of card selections remain similar to the controls, who had no access to social information. We do not quantify support for different hypotheses in this section. However, the easy-to-digest presentation of results in this section is supported by the detailed, but harder to digest, statistics of the next.
We then present a process-based analysis using maximum likelihood and multi-model comparison to investigate what strategies our participants used. This analysis demonstrates that, while participants did not reach dramatically different answers as a result of having access to social information, they did make use of this information. Furthermore, the way in which they used the non-epistemic social information sheds light on why it failed to change the overall patterns of card selections, compared with the controls.
In none of our analyses do we use traditional null-hypothesis significance tests. We are primarily interested in quantifying relative support for multiple social learning processes, and these cannot be estimated using null-hypothesis tests. We instead compare the different statistical models’ abilities to explain the data, using a common information-theoretic model comparison metric, Akaike Information Criteria, or AIC (Burnham and Anderson Reference Burnham and Anderson2002), that is described in the section on Model comparison results.
3.2.1. Outcome analysis
Figure 1 plots the results of Experiment 1. Overall, introducing non-epistemic social information (SI) had little apparent effect on final card selections, compared with control participants. This is also seen when we look at the data in terms of modelled probabilities of card selections between rounds 1 and 2. Across treatments, individuals began in round 1 with different patterns of card selections and, therefore, different frequencies of normatively correct p and not-q selections. Thus it is potentially misleading to simply compare raw frequencies of card selections at the end of the experiment. We used the control subjects to estimate the conditional probabilities of ending round 2 with a given card selection, given a card selection in round 1. For example, those who end round 1 with the p card un-selected have a lower probability of selecting p in round 2. Taking the maximum likelihood estimates of these probabilities, we computed the expected round 2 card selections for the participants who had access to social information, given their card selections in round 1, before they saw any answers from other individuals.
Frequencies of individual card selections in Experiment 1. White bars indicate round 1 card selections. Grey bars indicate round 2 card selections. The column ‘p, not-q’ indicates the frequency of normatively correct card selections across individuals.
While we included one large (N = 19) group in this summary, disaggregating that group from the smaller groups does not change the interpretation of the outcomes. Round 1 frequencies of card selections ({p, not-p,q, not-q}) were {1,0.1,0.55,0.4} for the four-person groups and {0.89,0.32,0.53,0.58} for the 19-person group. Round 2 frequencies were {1,0.1,0.4,0.65} and {1,0.11,0.53,0.63}, respectively. While there are small differences between the two sub-samples, neither changes much from round 1 to round 2. (We had originally included the large group, in order to have a powerful test of conformity. As we show in the likelihood-based analysis, however, there is little evidence of it.)
Figure 2 shows the final card selections from Experiment 1, treatment 2, relative to expectations derived from the control participants. Overall, there is very little difference in the expected and observed frequencies of round 2 card selections, either on a per-card basis (the first four columns in Figure 2) or regarding the frequency of normatively correct answers (the fifth column). If access to non-epistemic social information in this task had much of an effect on decisions, it is not apparent from an analysis of outcomes.
Final card selections in Experiment 1, non-epistemic SI treatment, relative to the expected frequencies, given that participants behaved as individuals did in the control treatment. The column (p, not-q) indicates the frequency of normatively correct card selections across individuals.
3.2.2. Process analysis
The card selections participants made can tell us about the outcome of their decision-making strategies, but a more nuanced analysis is needed to directly address the strategies themselves. In this section, we present a model comparison analysis of several candidate strategies by which participants used non-epistemic social information to make their card selections. The only information available to participants was the card selections of other individuals in their group. Knowing what information was available to each participant at each point in time, we used maximum likelihood estimation to fit variants of the models described in section 2. In addition, we fit a model in which individuals completely ignore social information and instead make recurrent selections according to the patterns we observed in the control treatment. We then compare the models on the basis of Akaike information criteria (AIC) and present weight-of-evidence measures for each.
Since the models presented below make predictions about the effect of group size, the two group sizes in the non-epistemic social information treatment are used together in the analysis.
Each model we considered was translated into a formula for predicting the probability of turning each card, given previous card selections and the card selections of other group members. In principle, these analyses are similar to logistic regression with different predictor variables. However, logistic regression predicts a single binary (0 or 1) event. These data require predicting four simultaneous binary events (the card selections). Each model is labelled with an integer x that identifies it in the formulas and the results tables to follow. Thus we define for each model x a probability 
$\Pr _{i,c,t}(1|x)$ of individual i turning card c in round t. The joint probability we are interested in for a vector of card selections θi, t is the likelihood of observing θi, t, under the given model x. This is defined as:
\begin{equation}
{\cal L}(\theta _{i,t}|x) = \prod _{c=1}^{4} \Pr _{i,c,t}(o_{i,c}|x) ,
\end{equation}
where o i, c ∈ {0, 1} is the observed card selection for individual i and card c. We use 
${\cal L}(\theta _{i,t}|x)$ as the basis for model comparison. Using this value computed for each individual in each round, we produce a total likelihood across all observations for each given model:
\begin{equation}
{\cal L}_{x} = \prod _i{\prod _t{{\cal L}(\theta _{i,t}|x)}}.
\end{equation}
We considered models that appeared to be plausible a priori, those presented earlier, in addition to a set of strategies suggested a priori by our colleagues. We have not attempted to exhaustively examine every possible model in the region of these.
Asocial (x = 1). For the basic asocial model, in which individuals ignore the choices of others, the probability that individual i turns card c in round t is given by:
\begin{equation}
\Pr _{i,c,t}(1|x=1) = \Pr _{c}(1| \ell _{i,c}),
\end{equation}
where ℓi, c ∈ {0, 1} is the previous (lag) card selection by individual i for card c in round t − 1. 
$\Pr _{c}(1|y)$ is the observed probability of turning card c given a previous selection y ∈ {0, 1} for that card. These conditional probabilities are the maximum-likelihood estimates derived from the control participants, who had no access to social information. Thus the asocial model is a test of the predictive accuracy of the individual behaviour in the control subjects, applied to subjects in other treatments. This model is our closest approximation of a ‘null hypothesis’. Although the asocial model presented here assumes that the probability of each card turn is independent of other cards, we have also tried an asocial model with a complete set of conditional probabilities. Since this did not alter the results presented below (the conditional probability model has a higher AIC than the unconditional), it is omitted from the paper.
Simple imitation (x = 2). The probability individual i turns card c under the simple imitation rule is defined by:
\begin{equation}
\Pr _{i,c,t}(1|x=2) = ( 1-T(\alpha ) ) \Pr _{c}(1| \ell _{i,c}) + T(\alpha ) \frac{\sum _{j}{\ell _{j,c}} - \ell _{i,c}}{N-1},
\end{equation}
where N is the number of individuals in the experimental group and α is a parameter to be fit to the data that indicates the frequency with which individuals use the rule. We transform α with the logit function T(α) = 1/(1 + exp (α)), so that there is always some chance the agent does not follow the social learning strategy (this is required in all statistical frameworks). As α → ∞, the above model reduces to the asocial model.
Simple imitation, as it is expressed here, describes both the behaviour of decision makers that randomly pick an advisor to imitate, and decision makers whose likelihood to adopt a behaviour (a particular card turn) is proportional to how frequent it is among advisors, i.e. some sort of probability matching strategy (Vulkan Reference Vulkan2000). Since our modelling is based on card turns, we cannot distinguish between these strategies.
Simple imitation, keepp and not-q (x = 3). In discussion of our analyses with colleagues, several people suggested that the results could be explained by simple imitation, except when people have the correct card selections (turning p and not-q only). The intuition behind this suggestion seems to be that those with the normatively correct answer realize this and are less subject – or immune – to social influence. The model is derived from model 2 and is defined by:
\begin{eqnarray}
\Pr _{i,c,t}(1|x=3) &=& ( 1-T(\alpha ) ) \Pr _{c}(1| \ell _{i,c}) \nonumber\\
&&+\, T(\alpha ) \left\lbrace \begin{array}{l@{\hspace*{7pt}}l}\theta _{i,t-1} = \lbrace p {\rm and not-}q\rbrace , & \ell _{i,c} \\[8pt]
\theta _{i,t-1} \ne \lbrace p {\rm and not-}q\rbrace , & \frac{\sum _{j}{\ell _{j,c}} - \ell _{i,c}}{N-1}
\end{array} \right..
\end{eqnarray}
Conformity (x = 4). The simplest conformity model implies that individuals adopt the majority card selection for each card. The probability individual i turns card c under this model is defined by:
\begin{eqnarray}
\Pr _{i,c,t}(1|x=4) &=& ( 1-T(\alpha ) ) \Pr _{c}(1| \ell _{i,c}) \nonumber\\
&&+\, T(\alpha ) \left\lbrace \begin{array}{l@{\hspace*{7pt}}l}\sum _j{\ell _{j,c}}>\frac{N}{2}, & 1 \\[7pt]
\sum _j{\ell _{j,c}}<\frac{N}{2}, & 0 \\[7pt]
\sum _j{\ell _{j,c}}=\frac{N}{2}, & \Pr _{c}(1| \ell _{i,c})
\end{array} \right. .
\end{eqnarray}
The parameter α again specifies the frequency with which individuals use the rule.
Model comparison results. Results of the maximum likelihood estimations are shown in Tables 1 and 2. For each model in the considered set, we compute wx, the Akaike weight (Burnham and Anderson Reference Burnham and Anderson2002), which measures the relative support for each model, in terms of its ability to predict out-of-sample data. wx is defined as:
\begin{equation}
w_{x} = \frac{ \exp \lbrace -\frac{1}{2}(\mathrm{AIC}_{x}-\mathrm{AIC}_{min})\rbrace }{ \sum _y{\exp \lbrace -\frac{1}{2}(\mathrm{AIC}_{y}-\mathrm{AIC}_{min})\rbrace } },
\end{equation}
where AICx is the Akaike Information Criterion for model x, defined by:
\begin{equation}
\mathrm{AIC}_{x} = -2\log ({\cal L}_{x}) + 2k_{x},
\end{equation}
and AICmin is the minimum AIC among the set of models. The variable kx is the number of free parameters to be estimated in model x. AIC corrects for the explanatory bias introduced by models with more parameters, and wx quantifies the relative support for each model, compared with the other models considered. Thus models with different numbers of parameters can be compared.
Maximum likelihood results for the models discussed in the text, fit to the data from Experiment 1 (n = 39). kx is the number of free parameters and wx is the Akaike weight, for a given model x. Best model shown in bold.
Maximum likelihood parameter estimates for the models discussed in the text, fitted to the data from Experiment 1 (n = 39). 95% confidence intervals shown in parentheses. Best model, according to Akaike weight, shown in bold.
While these weights do not indicate the estimated proportion of participants likely to have adopted each strategy, it is very probable that different individuals used different strategies in the experiments. Without more data on each participant, it is impossible to tightly estimate individual strategy use. Instead, we have produced overall fits for each model, and a model-averaging approach would lead us to predict that models with higher Akaike weights were used by more participants or, similarly, more often (cf. Burnham and Anderson Reference Burnham and Anderson2002).
The simple imitation model (x = 2) has much more support than the other models in the set. Among all the models considered, the asocial model (x = 1) ranked last. The strategy Simple + keep p and not-q, in which the normatively correct selection of the p and not-q cards was kept, did rather poorly compared with the simple imitation strategy. The parameter estimate for the best-fitting model suggests a 60% reliance on social information (95% confidence interval 0.85–0.34).
3.3. Discussion
The model fitting evidence suggests that individuals that have access to the answers of other participants use this information to modify their decisions. The asocial model performed worse than every strategy involving the use of available social information. Of the strategies examined, simple imitation, in which another participant is chosen at random and copied, fits the data best, and its relative support is greater than that of the other models. The result is counter-intuitive, because previous analyses have suggested that in situations in which simple imitation of this kind is adaptive, a majority rule should perform at least as well (McElreath et al. Reference McElreath, Lubell, Richerson, Waring, Baum, Edsten, Efferson and Paciotti2005). Yet conformity did a comparatively poor job of fitting the data, suggesting either that it was not present in our participants or that very few participants used such a strategy. The result resembles findings by Larrick and Soll (Reference Larrick and Soll2006) where participants were unwilling to average advice, and instead trusted a single, expert, advisor, also under circumstances in which the average response would have been more successful. The important difference is, however, that our participants had few means to identify the better advisors, except perhaps by comparing the number of card turns they had in common which each potential advisor. The distribution of card turns, with very little variation in p and not-p card turns, unfortunately makes any such strategy difficult to detect, and also less useful for participants. The lack of conformity is a puzzling finding that, perhaps, can be explained by the cognitive cost associated with looking at all social information, or the general difficulty of the task, which could have made participants doubt that advisors in general were above chance. There may, of course, be other models better suited to predict participants’ behaviour, but it seems relatively clear that they used social information, but not through conformity, despite its advantages.
There is additional support for the inference that some kind of simple imitation is at work in Experiment 1. Despite the poor fit of the asocial model to the data, participants’ advice taking did not substantially alter the distribution of card selections, compared with the controls. This is to be expected if the best-fit model, simple imitation, is at work, because it does not change the frequencies of answers in a group. Rather, it tends to preserve previous variation in choice.
4. EXPERIMENT 2: INDIVIDUAL USE OF EPISTEMIC SOCIAL INFORMATION
Our second experiment gave participants access to epistemic, as well as non-epistemic, social information, in an effort to see both how the decision outcomes differ as a result and what advice taking strategies decision makers use when both types of social information are present.
4.1. Design and procedure
The experiment consisted of two treatments, varying how participants were presented with epistemic social information.
In the epistemic social information treatment, epistemic social information consisted of a short written motivation for card turns, typically one or two sentences. As in Experiment 1 all social information was anonymized, and the non-epistemic social information consisted of other participants’ card turns. In contrast with Experiment 1, participants provided and received social information not once, but three times. That is, participants first solved the task individually, while providing written reasons for their choice. They then exchanged answers and reasons and re-solved the task. This process was iterated two more times, before participants handed in their final answer. The answers provided during the first (initial answers) and last round were rewarded, as in Experiment 1. These two rounds were 20 minutes, as before, but the intermediate rounds (round 2 and 3) were shortened to 5 minutes so as not to make the task too long (the total already being 30 minutes). Out of the four rounds, rounds 1 and 2 are directly comparable to the two rounds in the non-epistemic social information treatment in Experiment 1 in terms of the number of iterations, and round 3 in terms of time spent on the task. To make sure that the available epistemic social information was of good quality, we also rewarded participants for group improvement. Participants who answered the first round correctly received €4 or USD 6 for any increase of correct answers in the last round (i.e. for any individual that correctly solved the problem in the last round, but not in the first round). We ran four groups with five members each.
In the full interaction treatment, groups were allowed a maximum of 20 minutes of free discussion after they first had completed the 10 minute individual decision round that all treatments begin with. Thus full interaction and the epistemic social information treatments were both allowed 30 minutes on the task in total. Groups were not instructed to reach a consensus, and participants’ final answers were turned in individually. Participants were rewarded as in the epistemic social information treatment. We videotaped the group discussions, but have not used this material in the analysis. Again, we ran four groups with five members each.
Participants were 40 students recruited through introductory classes in the humanities (20 in each treatment). They were previously unfamiliar with the task.
4.2. Results
Again, we present the results in two parts: outcome-oriented results about the card selections individuals made, compared with the control treatment in Experiment 1, and process-based analysis about the strategies driving these decisions. The outcome analysis considers both treatments, full-interaction and epistemic social information. The process analysis considers only the epistemic information treatment, where we have detailed coding for the information available to each participant at each stage. Again, the outcome analysis is easier to digest than the process analysis. However, we quantify statistical support for different hypotheses in the process analysis, not the outcome analysis.
4.2.1. Outcome analysis
Unlike in Experiment 1, the last round in Experiment 2 shows a dramatic change in card selections, both relative to the controls (Figure 1) and to the card selections in round 1, before participants had access to any social information. Figure 3 shows the card selection frequencies for the full interaction and epistemic information treatments of Experiment 2. There are large shifts in decisions over the course of the experiment, with the full interaction treatment showing the larger changes. The direction of change is the same in both cases, however.
Card selections in Experiment 2, both treatments, all rounds. The column ‘p, not-q’ indicates the frequency of normatively correct card selections across individuals.
Again, a better way to analyse the outcomes is by comparing final decisions to expected decisions, under the model that participants behaved in the same way as the controls, who had no access to social information. Figure 4 plots the observed frequencies of card selections minus the expected frequencies, for all cards and the normatively correct p and not-q pattern. This way of looking at the results tells the same story: Experiment 2 induced dramatic changes in the card selection patterns, relative to no-social-information controls. The directional effect of the change was to induce more p and not-q decisions. We quantify the support for these interpretations in the next sections.
Final card selections in Experiment 2, relative to the expected frequencies, given that participants behaved as individuals did in the control treatment. Grey bars indicate results of the Epistemic Social Information treatment. White bars indicate the results of the Full Interaction treatment.
4.2.2. Process analysis
In order to analyse the epistemic social information available in the epistemic information treatment, we coded the reasons participants provided into a set of six potential cases. Reasons were coded in terms of possible interpretations of the rule. We termed the two most common reasons equivalence and material implication, and it is only these that are modelled in the analysis. Equivalence roughly corresponds to reasons denoted as ‘matching’ and ‘verification’ by Wason and Evans (Reference Wason and Evans1975) and material implication to what they termed ‘falsification’.
By material implication, we denote reasons that imply that p and q do not have to be exclusively paired. An example of a reason suggesting material implication is:
If there is a consonant or a vowel behind 4 makes no difference. If there is an even or odd number behind K makes no difference!
The most common card turns associated with this type of reason are ‘turn p and not-q’, 17 out of 23 card turns associated with material implication involve these cards.
By equivalence we mean reasons that imply that p and q should be exclusively paired. An example of a reason suggesting this is:
A and 4 can prove the rule (or disprove). K and 7 can disprove the rule if there is a vowel or an even no. on the back.
Common card turns associated with equivalence are ‘turn p, not-p, q and not-q’ (7 out of 12 possible equivalence card turns) and ‘turn p and q’ (4 out of 12 turns). That neither equivalence reasons, nor material implication reasons are always associated with these card turns is comparable to previous studies (Wason and Evans Reference Wason and Evans1975; Evans and Wason Reference Evans and Wason1976).
To examine the strategies our participants used for social learning when exposed to both non-epistemic and epistemic social information about card selections, we fit the models from the previous analysis, as well as eight new models incorporating possible use of epistemic social information contained in the explanations. Here we describe the eight models incorporating use of epistemic social information, defining for each a function 
$\Pr _{i,c,t}(1|x)$, as before.
Simple answers and reasons imitation (x = 5). This model posits simple imitation of both non-epistemic (answers) and epistemic (reasons) social information, adjusting the weight or frequency of use of each parametrically. Adopting a reason, either material implication or equivalence, increases the probability of adopting the corresponding card selections, {p and not-q} or {p and q}, respectively. While no single vector of card selections was associated with either interpretation, these two patterns are motivated by the interpretations, so using them is perhaps the harshest test of this model. The probability individual i turns card c in round t is given by:
\begin{eqnarray}
\Pr _{i,c,t}(1|x=5) &=& ( 1-T(\beta ) ) \Pr _{i,c,t}(1|x=2) \nonumber \\
& & +\, T(\beta ) \lbrace (1-T(\alpha ))\Pr _c(1|\ell _{i,c}) + T(\alpha )( f_m s_{m,c} + f_e s_{e,c} ) \rbrace ,
\end{eqnarray}
where fr is the frequency of reason r among all reasons, excluding individual i, presented in round t − 1, and s r, c ∈ [0, 1] is the card selection suggested by reason r for card c. Again we use the logit transformation T(α) = 1/(1 + exp (α)) to ensure well-behaved probabilities.
Simple answers and reasons, keep material implication or keep equivalence (x = 6 andx = 7). Epistemic social information might change the probability of particular card turns, depending upon what reason is adopted. To model this, we assumed that individuals behave according to model 5 above, except when they have already adopted material implication or equivalence as a reason for their card selections. In that case, they make the card selections implied by the reason they have adopted, with no chance of imitation.
Simple answers imitation, conformity reasons (x = 8). To examine the possible use of a majority rule in adopting reasons, we fit a model that uses the simple imitation of answers with a flexible use of conformity. We do not combine conformity applied to reasons with other answers-based (i.e. non epistemic social information) strategies because the results of Experiment 1 strongly suggest simple imitation of answers, when participants attend to non-epistemic social information in this task. (We also fit such strategies to the data, and found this intuition was correct. We omit them here because they add little to the presentation of an already-complicated analysis.) The probability of turning card c under this model is given by
\begin{eqnarray}
\Pr _{i,c,t}(1|x=8) &=& (1-T(\beta ))\Pr _{i,c,t}(1|x=2) \nonumber \\
& & +\, T(\beta ) \left( (1-T(\alpha ))\Pr _c(1|\ell _{i,c}) + T(\alpha ) \left\lbrace \begin{array}{ll}f_m > 0.5, & s_{m,c} \\
f_m < 0.5, & s_{e,c} \\
\end{array} \right. \right),
\end{eqnarray}
where fr is defined as in model 5. The multiple-choice expression on the far right indicates the card selection for the most common reason. (A continuous function that can accomplish the same kind of branching choice is fzm/(fm z + fze)s m, c + (1 − fzm/(fm z + fze))s e, c, where z ≫ 1 is a large power that creates a steep step-function.)
Simple answers imitation, conformity reasons, keep material implication or equivalence (x = 9 andx = 10). Analogous to models 6 and 7, we modified model 8 to include keeping material implication or equivalence answer vectors, but having a chance to imitate a reason otherwise.
Model comparison results. The results of fitting all 10 candidate models to the 60 vectors of card selections and provided-reasons from Experiment 2, restricted-interaction treatment, are shown in Tables 3 and 4. This analysis finds that reasons had a large influence on card selections, as suggested by the outcome analysis. The only model ignoring epistemic social information that ranks above any model incorporating it is model 2 (simple imitation), the best-fit model from Experiment 1. Still, model 2’s fit to the data is very poor, compared with several of the epistemic models (models 5 through 10). There is more uncertainty here which of the epistemic models is correct. However, model 6 carries the majority of the weight of evidence.
Maximum likelihood results for the models discussed in the text, fitted to the data from Experiment 2 (n = 20). kx is the number of free parameters and wx is the Akaike weight, for a given model x.
Maximum likelihood parameter estimates for the models discussed in the text, fit to the data from Experiment 2 (n = 20). 95% confidence intervals shown in parentheses.
The parameter estimates in Table 4 show that the better-fitting models contain substantial weights given to social information, of both types. Examining the best-fitting model, number 6, the total weight of use of social information is estimated at 80% (95% confidence interval 0.93–0.61). This is an increase of about 20% from the estimate for the best-fitting model in Experiment 1 (Table 2). Additionally, the estimated weight of epistemic social information within this 0.80 is about half (95% confidence interval 0.596–0.167). It is possible that these model fits would be different for just the second round of data. Computing the Akaike weights for just the 20 vectors of card selections in round 2 of restricted interaction treatment of Experiment 2, we find the same ordering and approximate spacing of Akaike weights for these models. This suggests the processes were similar throughout each session.
4.3. Discussion
The outcome analysis demonstrates that access to epistemic social information in this task can tremendously change the characteristic decisions individuals make in a way that non-epistemic social information could not in Experiment 1. This is not a result of having more time, as just round 2 data from Experiment 2 (restricted interaction) shows the same drift towards p and not-q card selections. The process analysis supports this by providing much more support for strategies using epistemic social information than those using non-epistemic social information only. Furthermore, the finding does not depend upon having more than two rounds in which to work on the task.
A robust summary of the model comparison results is that material implication as a reason either persists or spreads better than equivalence, and that this feature of it explains the very different outcome results we find in Experiment 2. All leading models involve epistemic social information and a relatively simple advice taking process that ignores large portions of the available social information. Additional processes may be at work in the full-interaction treatment. Indeed, we expect many subtle aspects of persuasion and signalling to be at work in such groups. But our restricted-interaction groups demonstrate that access to epistemic, in addition to non-epistemic, information can explain large portions of the differences in decisions between Experiments 1 and 2.
5. GENERAL DISCUSSION
The goal of this paper has been to initiate research on how decision makers integrate others’ opinions with their own, both when it comes to epistemic and non-epistemic social information.
In Experiment 1 the experimental group of decision makers were given access to non-epistemic social information. Although the outcomes of the experimental and control groups (with no advice) did not differ, analysis revealed that participants that received advice were influenced by it. In particular, the asocial model (ignoring advice) had a poor fit to data, whereas a simple imitation model provided the best fit. The Akaike weights used to compare models guarantee that simple imitation’s advantage is not due to its additional free parameter.
Experiment 2 introduced epistemic social information into the advice-taking scenario. Advice taking strategies including this information had superior fit to strategies ignoring the epistemic part of advice. Even simple imitation, explaining most of the responses in Experiment 1, did relatively poorly in the presence of reasons. The advice-taking model that fit data best was, however, a version of simple imitation directed at both epistemic and non-epistemic social information. The best-fitting Model 6 has an additional quirk, however: if participants have already adopted a reason advocating a material implication interpretation of the Wason selection task, they are inoculated against social influence. Such decision makers instead select the correct response. Thus this advice-taking model allows the right reasons to persist or spread among participants.
In both experiments, some version of simple imitation outperformed the asocial and conformity models. Thus, decision makers take advice into account, but they do not take advantage of the well-documented benefits of conformity. On the other hand, epistemic social information improved advice taking: ‘Correct’ reasons either persisted or spread among decision makers.
The models evaluated here are only a subset of possible advice taking strategies, and other models could have produced a better fit. Nevertheless, also when the experiments are conservatively interpreted the following conclusions appear valid: First, when available, participants take epistemic social information into account, even if this calls for more advanced (but not necessarily more successful) cognitive processing. Second, epistemic social information affects advice taking substantially. Different strategies were used in the epistemic and non-epistemic experimental groups, and it is unlikely that any reasonable advice-taking model could explain both response patterns. Furthermore, the presence of epistemic social information affects the weights participants assign to advice. Making social information epistemic increases its impact.
A first lesson to draw from these results is that majority rules, such as the conformity model we have tested here, do not seem to be as common as one might expect given their advantages. This might be because real life violates many of the assumptions made in, for instance, the Condorcet jury theorem. Although parts of our experiments did fulfil the independence criterion (Experiment 1, and the first steps in Experiment 2), participants may not be used to this, since most social interactions are blatant violations of independence. The notorious difficulty of the Wason selection task also helps us understand the results. If participants do not believe that other group members are better than random in selecting the correct response (and as a matter of fact, decades of experiments support this assumption, in particular for the q and not-p card turns), they have no reason to resort to some kind of majority rule. Under such circumstances it could be better to imitate one, particularly smart, group member, getting the advantages of pay-off biased social learning. However, this strategy appears better suited for Experiment 2 than Experiment 1, where participants have little means of identifying the better group members. A confusing and difficult task might, however, lead participants to clutch at straws in the first experiment. Perhaps it is time for also the conceptual research into deliberation to start looking into decision procedures that are aimed at identifying a few, better, decision makers.
A second lesson that can be drawn is that epistemic social information certainly matters. It matters both with respect to outcomes and with respect to how participants use social information. Thus, we would like to emphasize that research into, for instance, premise- and conclusion-based majority rules are on the right track. Real people are sensitive to this difference, and knowing more about when each type of social information yields optimal outcomes would also be very helpful with respect to research on advice taking and group decision making.
ACKNOWLEDGEMENTS
We would like to thank Maria Larsson, who helped us conduct the experiments, the Max Planck Institute for Human Development for funding the study, Christian List, Wlodek Rabinowicz, Stefan Schubert and audiences at various presentations for their helpful comments and suggestions.
