1. Introduction
Adolescents are growing up in an information landscape that Stanovich and Toplak (Reference Stanovich and Toplak2023) describe as a hostile thinking environment, saturated with misinformation such as fake news and pseudoscientific claims. In such increasingly demanding conditions, deciding whom and what to trust, and whether to do so reluctantly or enthusiastically, requires a criterion or standard by which to evaluate someone’s credibility.
This question is civically important. On the verge of adulthood, adolescents start making decisions about which people and institutions are trustworthy enough to represent their interests. At the same time, they are going through a developmental stage marked by identity formation, including ideologies, beliefs, and attitudes toward the world and society (Schwartz, Reference Schwartz2001). To navigate a hostile thinking environment while simultaneously managing the developmental tasks before them, adolescents need standards of reasoning that can guide how they search for evidence, evaluate alternatives, and decide how much confidence a conclusion deserves. Actively open-minded thinking (AOT) represents one such set of standards.
1.1. Actively open-minded thinking (AOT)
AOT is a prescriptive model of good reasoning and decision-making based on the fair consideration of evidence and openness to revising one’s conclusions, even when they contradict one’s preferred views (Baron et al., Reference Baron, Gürçay, Metz, Toplak and Weller2016). It can be seen as an operationalization of the broader concept of critical thinking, or its gold standard (Metz et al., Reference Metz, Baelen and Yu2020; Stanovich and West, Reference Stanovich and West1997).
The concept of AOT is particularly relevant to how the process of searching for new information unfolds. Thinking often goes astray either because the search is prematurely terminated or because it is biased. Bias may manifest as superficial or limited searching for evidence which contradicts one’s preferred conclusions or as downplaying its importance once encountered, a tendency known as myside bias. A person who thinks in an actively open-minded way not only accepts evidence that contradicts their initial stance but actively seeks it out (Baron et al., Reference Baron, Gürçay, Metz, Toplak and Weller2016).
Along with insufficient (or, more rarely, excessive) search and myside bias, a third common error is overconfidence, which occurs when the confidence one feels in a judgment is not proportional to the amount and quality of reasoning invested in it, given the available opportunity for depth and breadth of investigation (Baron et al., Reference Baron, Isler, Yılmaz, Ottati and Stern2023). The consequences of myside bias and overconfidence are suboptimal decisions, such as endorsing conspiracy theories or unproven alternative medicine remedies. AOT, which entails both an optimal amount and direction of thought, is intended as an antidote to such reasoning errors (Baron et al., Reference Baron, Gürçay, Metz, Toplak and Weller2016). Emphasizing the direction of thinking is the defining feature that distinguishes AOT from related but qualitatively different constructs, such as cognitive reflection.
The popularity of AOT largely stems from its predictive validity across a wide range of indicators of rational thought. For example, AOT has been found to correlate positively with resistance to the framing effect, the conjunction fallacy, the sunk cost fallacy, and outcome bias, as well as with success in solving belief-bias syllogisms and the Wason card selection task (Erceg et al., Reference Erceg, Galić and Bubić2022; Rachev et al., Reference Rachev, Geiger, Vintr, Kirilova, Nabutovsky and Nelsson2022; Toplak et al., Reference Toplak, West and Stanovich2011; West et al., Reference West, Toplak and Stanovich2008). These associations remain significant even when controlling for intellectual ability (Stanovich and Toplak, Reference Stanovich and Toplak2023). Beyond classic heuristics and biases tasks, AOT is negatively related to various forms of unwarranted beliefs, such as paranormal beliefs, conspiracy theories, and anti-scientific attitudes (Pennycook et al., Reference Pennycook, Cheyne, Koehler and Fugelsang2020; Rizeq et al., Reference Rizeq, Flora and Toplak2021; Svedholm-Häkkinen and Lindeman, Reference Svedholm-Häkkinen and Lindeman2018). The positive relationship between AOT and fake news discernment has also been confirmed in multiple studies (Bronstein et al., Reference Bronstein, Pennycook, Bear, Rand and Cannon2019; Erlich et al., Reference Erlich, Garner, Pennycook and Rand2023; Mirhoseini et al., Reference Mirhoseini, Early, El Shamy and Hassanein2023; Saltor et al., Reference Saltor, Barberia and Rodríguez-Ferreiro2023).
These findings support the assumption that AOT can serve as a protective factor in the hostile thinking environment in which adolescents are growing up. Adopting AOT as a standard of good thinking, applicable both to one’s own reasoning and to the claims of others, entails greater skepticism toward unsupported or overly confident assertions, ranging from fake news and pseudoscience to populist rhetoric. Ideally, such skills should be developed before individuals cast their first vote; accordingly, the present study focuses on AOT among adolescents.
1.2. Actively open-minded thinking in adolescence
Adolescence is a period of identity formation and questioning of learned values and norms, which makes it particularly conducive to the development of AOT (Metz et al., Reference Metz, Baelen and Yu2020). Identity status can be differentiated along two dimensions—exploration and commitment (Marcia, Reference Marcia1966). Adolescence is sometimes referred to as the moratorium period, characterized by high exploration and low commitment, which has been associated with indicators of critical thinking and open-mindedness (Yoder, Reference Yoder2000). According to a meta-analysis, more than one third of adolescents experience progress in their identity status, with the proportion of those in moratorium increasing until adulthood and declining thereafter (Kroger et al., Reference Kroger, Martinussen and Marcia2010). This flexibility in self-concept may enable adolescents to critically reassess norms and beliefs.
However, cognitive-developmental research presents a more cautious perspective regarding adolescents’ capacity for AOT. Research on adolescents’ argumentation and scientific reasoning skills indicates a tendency toward myside bias. Nearly half of them fail to generate alternative theories, counterarguments, or rebuttals to their own assumptions, instead taking them for granted (Kuhn, Reference Kuhn1992, Reference Kuhn, Lerner and Steinberg2009). Compared to adults, adolescents spend more time articulating their own position during discussions than responding to the arguments of the opposing side, although they can do so when explicitly instructed (Felton and Kuhn, Reference Felton and Kuhn2001; Kuhn and Udell, Reference Kuhn and Udell2007).
Only a minority reach the final, evaluativist stage of epistemological development, which involves tolerating uncertainty as well as the ability to apply criteria for evaluating different positions. Still, the proportion of adolescents who attain the evaluativist stage increases steadily into emerging adulthood, after which it typically plateaus (Kuhn et al., Reference Kuhn, Cheney and Weinstock2000). Despite observed shortcomings in adolescents’ critical thinking, early adolescence is considered a ‘second critical period’ for shaping cognitive patterns that remain stable throughout life (Kuhn, Reference Kuhn, Lerner and Steinberg2009).
Few studies have directly examined AOT during adolescence. Metz et al. (Reference Metz, Baelen and Yu2020), in a large and diverse sample of American adolescents, found that most participants endorsed the norms of AOT. Research has also shown a tendency for AOT to increase during late childhood and early adolescence (Kokis et al., Reference Kokis, Macpherson, Toplak, West and Stanovich2002; Toplak et al., Reference Toplak, West and Stanovich2014). Studies examining outcomes that AOT typically predicts well in adults have produced mixed results among adolescents. AOT has been found to be a significant predictor of both inductive and deductive reasoning, even when controlling for cognitive ability (Kokis et al., Reference Kokis, Macpherson, Toplak, West and Stanovich2002). Furthermore, higher AOT has generally been linked to better performance on reasoning tasks involving cognitive biases, such as framing effects and belief-bias syllogisms, and to lower endorsement of unwarranted beliefs, including superstition, anti-scientific attitudes, and paranormal or conspiracy beliefs (Rizeq, Reference Rizeq2019; Toplak et al., Reference Toplak, West and Stanovich2014; Toplak and Flora, Reference Toplak and Flora2021).
Conversely, Svedholm-Häkkinen and Lindeman (Reference Svedholm-Häkkinen and Lindeman2018) found that AOT was significantly but weakly negatively correlated with faith in intuition, while its relationship with trust in alternative medicine was not supported. Toplak and Stanovich (Reference Toplak and Stanovich2024) found no associations between AOT and resistance to the framing effect or knowledge calibration, although its correlations with probabilistic and scientific reasoning were significant but weaker than those observed among adults. The low or nonsignificant correlations reported in these two studies may partly be due to the low reliability and questionable unidimensionality of the measures used. These findings call for caution in interpreting observed relationships but also highlight the need for closer examination of the psychometric properties of AOT scales.
1.3. Conceptualization and measurement of AOT
AOT measurement tradition has relied predominantly on questionnaire-based methods, which makes its ability to predict performance on various heuristics and biases tasks all the more remarkable (Stanovich and Toplak, Reference Stanovich and Toplak2023). Scales of AOT are grounded in the assumption that beliefs about how we should think largely determine how we actually think (Baron, Reference Baron, Voss, Perkins and Segal1991). Within this framework, AOT can be conceptualized in several ways.
Stanovich and West (Reference Stanovich and West1997) emphasize decontextualization, the ability to evaluate new information independently of one’s prior knowledge and beliefs. Their original scale, which Stanovich and Toplak (Reference Stanovich and Toplak2023) describe as a ‘conceptual amalgam’, aimed to capture several dimensions: reflectiveness versus impulsiveness; willingness to consider evidence that contradicts one’s beliefs; openness to alternative explanations and viewpoints; and tolerance of uncertainty combined with a willingness to delay conclusions. Items representing these dimensions form the flexible thinking subscale. The authors also included additional subscales such as dogmatism, absolutism, and categorical thinking. This 41-item scale and its later revisions (e.g., Stanovich and West, Reference Stanovich and West2007) have become the most widely used instruments for measuring AOT, forming the basis for numerous shorter versions.
Baron (Reference Baron2019) proposed a shorter version, also used in the present study, designed to assess beliefs about AOT. The items were primarily drawn from the flexible thinking subscale, which is theoretically closest to the construct (Baron, Reference Baron2019). According to Baron (Reference Baron2024), individual differences in AOT can be reduced to the extent to which people endorse norms that help avoid (a) myside bias and (b) uncertainty aversion, which is the assumed cause of overconfidence. The current version of the scale contains more items addressing myside bias than overconfidence (Baron et al., Reference Baron, Isler, Yılmaz, Ottati and Stern2023).
In the context of adolescents, Metz et al. (Reference Metz, Baelen and Yu2020) describe three facets of AOT: (1) deep search for possibilities and evidence, (2) epistemic empathy—the ability to understand the intellectual position and inner logic of someone with whom one disagrees, and (3) epistemological sophistication—the belief that one can learn something through AOT. Their 8-item Adolescent AOT Scale, which was also used in this study, assesses cognitive norms and beliefs about the usefulness of AOT.
Although diverse conceptualizations of AOT all acknowledge its multidimensional and complex nature, AOT questionnaires are often scored and analyzed using a single total or mean score. This scoring approach does not require AOT to be conceptually unidimensional, but it does require that the items share enough common variance for the composite score to be interpretable, reliable, and useful in predicting relevant outcomes. Factor-analytic evaluations of questionnaires derived from the original Stanovich and West scale have raised questions about the psychometric suitability of such simple composite scoring. For instance, a factor analysis of a 17-item version yielded four factors (Svedholm-Häkkinen and Lindeman, Reference Svedholm-Häkkinen and Lindeman2018). Another study found that a one-factor structure did not fit the data adequately, while the proposed four-factor solution was not replicable and the resulting factors were not meaningfully interpretable (Janssen et al., Reference Janssen, Verkoeijen, Heijltjes, Mainhard, van Peppen and van Gog2020). Similarly, research using an adapted short version for adolescents did not confirm a simple one-factor structure (Rizeq, Reference Rizeq2019).
Shorter scales have often emerged as efforts to refine the measurement of AOT toward its conceptual core, particularly flexible thinking (Baron, Reference Baron2019; Pennycook et al., Reference Pennycook, Cheyne, Koehler and Fugelsang2020; Stanovich and Toplak, Reference Stanovich and Toplak2023). Accordingly, Pennycook et al. (Reference Pennycook, Cheyne, Koehler and Fugelsang2020) developed an 8-item measure of actively open-minded thinking about evidence (AOT-E), which assesses meta-beliefs about whether one’s beliefs should change in response to new evidence. Factor analyses in U.S. and Brazilian adult samples confirmed a single latent factor (Bonafé-Pontes et al., Reference Bonafé-Pontes, Bastos and Pilati2025), supporting the notion that shorter, conceptually coherent scales yield more stable structures.
Overall, the existing literature indicates a lack of suitable AOT measures for adolescent populations, as well as a broader scarcity of research on AOT in this developmental period. To help bridge this gap, one of the aims of the present study was to examine the psychometric properties of two AOT scales in a sample of high school students.
1.4. Teaching actively open-minded thinking
The idea of teaching AOT is as old as the construct itself (Baron, Reference Baron1988, Reference Baron1993). This idea relies on the assumption that thinking dispositions may be more amenable to change than cognitive abilities, which are typically considered the main determinants of good thinking (Baron, Reference Baron1985). Although numerous authors have called for interventions aimed at developing AOT, concrete suggestions on how to design and implement those interventions remain scarce. In rare empirical studies that evaluate intervention effectiveness, not all of them report positive effects. For instance, Noone and Hogan (Reference Noone and Hogan2018) found no evidence that a 6-week online mindfulness intervention could improve AOT in a study with an active control group.
More instructional approaches have produced some encouraging findings. A pioneering work from the 1980s found that so-called experimental reasoning classes, which explicitly emphasized the importance of actively considering the opposing side of a problem, helped students generate more arguments for the unpreferred side of socially controversial issues (Perkins, Reference Perkins2019). In another educational context, participants in a month-long robotics course for gifted girls in Saudi Arabia, which included training in soft skills such as presentation, showed higher levels of AOT at the end of the program compared to its beginning (Ayoub et al., Reference Ayoub, Abdulla Alabbasi, Alsubaie, Runco and Acar2022). However, in the absence of a comparable group, it is difficult to draw firm conclusions about any causal effects of the intervention.
Recent work has focused on efforts to teach AOT using brief online formats. A short online module showed modest short-term success in increasing the number of reasons participants offered for why their answer to a multiple-choice question might be wrong, an indicator of reduced overconfidence (Gürçay-Morris, Reference Gürçay-Morris2016). Another attempt to teach AOT through an online module and a serious game was partially successful in reducing myside bias and overconfidence, and also increased behavioral intentions to use AOT in everyday situations, although only in the short term (Erceg et al., Reference Erceg, Andrić, Bosilj, Britvić, Čeko, Dedić, Diatlovska, Jakopović, Jakopović, Jelić, Komlinović, Kostanić, Leverić, Neznanović, Ostoić, Prevendar, Tadijan, Tokić, Vukasović and Galić2024). The most direct recent evidence comes from Biddlestone et al. (Reference Biddlestone, Ziemer, Maertens, Roozenbeek and van der Linden2025), who tested a norm-enhanced logic-based inoculation message against failure to engage in AOT. Across two experiments, the intervention increased AOT relative to control conditions and, through AOT, indirectly improved misinformation discernment and reduced conspiracy beliefs. However, no follow-up was conducted, which leaves us with limited insight into long-term effectiveness. These findings suggest that AOT may be teachable, but the effects do not necessarily hold up in the long run. In order to sustain AOT gains, continuous or repeated interventions may be needed (Erceg et al., Reference Erceg, Andrić, Bosilj, Britvić, Čeko, Dedić, Diatlovska, Jakopović, Jakopović, Jelić, Komlinović, Kostanić, Leverić, Neznanović, Ostoić, Prevendar, Tadijan, Tokić, Vukasović and Galić2024).
This raises the question of whether existing, sustained educational activities can provide opportunities to practice thinking in an actively open-minded way. In his essay Why teach thinking? Baron (Reference Baron1993) offers a critique of the enduring educational trend of using class discussions as a teaching method. Because most students are not sufficiently equipped with domain-specific knowledge nor discussion skills, most classroom discussions fall short of their goal of cultivating critical thinking. In the present study, we examine debate as a specific form of discussion, which may be better suited to address these shortcomings.
1.5. Debate
Debate is a structured, argument-based discussion in which participants propose or oppose a given motion within a defined set of rules (Akerman and Neale, Reference Akerman and Neale2011). These rules, known as debate formats, determine the flow of the debate (e.g., the number of teams, speech duration) and the criteria judges use when evaluating arguments. The existence of such rules is what fundamentally distinguishes debate from ordinary, informal discussion (Snider and Schnurer, Reference Snider and Schnurer2006).
In Croatia, debate as an extracurricular activity follows the model of competitive debate, which emphasizes participation in tournaments and preparation for them. Currently, more than 130 debate clubs operate in primary and secondary schools across the country (Croatian Debate Society [CDS], n.d.-a). High school debate clubs are most often led by volunteers who were debaters themselves.
To assess whether debate might address Baron’s (Reference Baron1993) critique of classroom discussions, it is important to understand how a typical high school debate program operates. Although clubs may differ in how they structure meetings and preparation, several features of the activity are relatively formalized. Each debate begins with a motion, which is a statement designed to be balanced so that both sides have a fair chance of winning. Debaters are expected to argue for either side, regardless of their personal beliefs. Unlike a broad topic, a motion is specific enough for its key aspects to be covered in a debate lasting about an hour (Quinn, Reference Quinn2009). Some motions are prepared, meaning they are announced weeks or months before the tournament. For these motions, debaters research the topic and prepare arguments for both sides. Because they are not told in advance which side they will defend, and usually debate both sides of the same motion during a tournament, they must anticipate arguments and rebuttals from prospective opponents. Impromptu motions, by contrast, are announced (along with assigned sides) only one hour before the debate begins. Debaters must prepare for impromptu motions independently, without access to technology or their coaches’ assistance (The English-Speaking Union, 2019). Tournaments may include only prepared or only impromptu motions, but most feature a combination of both.
Preparation for a prepared motion begins with research (Quinn, Reference Quinn2009). For example, in last year’s regional competitions, high school students debated the motion ‘This HouseFootnote 1 would abolish voting rights for the diaspora in the Republic of Croatia’ (CDS, n.d.-b). Research might begin with exploring the historical context, voting patterns among the diaspora, and the impact of diaspora votes on past elections. Debaters then proceed to develop their arguments, guided by the coach, who provides feedback and facilitates collaborative learning (Snider and Schnurer, Reference Snider and Schnurer2006).
To perform well on impromptu motions, debaters must also stay up to date on current affairs and develop broad general knowledge. Outside of tournament preparation, club meetings often focus on practicing specific debating skills. For instance, a session devoted to logical reasoning may include exercises on induction and deduction, identifying logical fallacies, and learning how to effectively refute an argument (see CDS, 2021, for an example of a debate lesson). Debate involves both formal and informal logic, as well as abductive reasoning. Abductive reasoning begins with a complex question (e.g., ‘Should the diaspora be allowed to vote?’) and seeks the simplest and most plausible explanation or solution—in other words, the argument that most convincingly explains the benefits or harms of diaspora voting. Because absolute certainty is unattainable, abduction is often described as inference to the best explanation (Douven, Reference Douven, Zalta and Nodelman2025). In debates, trying to prove something ‘beyond doubt’ is usually a poor strategy; instead, debaters aim to persuade the judge that their side provides the more plausible explanation.
Research has linked participation in debate to a variety of desirable outcomes. Debaters tend to exhibit higher levels of political knowledge and stronger political participation intentions (Krpanec and Huić, Reference Krpanec and Huić2023). They also show higher need for cognition, though it is likely that students with a naturally higher need for cognition are more drawn to debate in the first place (Ajduković and Čorkalo Biruški, Reference Ajduković and Čorkalo Biruški2008). Of particular relevance to the present study are findings concerning critical thinking. Qualitative research indicates that both debaters and debate coaches frequently report perceived improvements in critical thinking skills (Fine, Reference Fine2000; Hamaker, Reference Hamaker2019).
A meta-analysis of research across communication disciplines found that forensics and competitive debate lead to the greatest gains in critical thinking (Allen et al., Reference Allen, Berkowitz, Hunt and Louden1999). The authors reported an average increase of 44%; however, drawing conclusions about the relationship between debate and AOT from these findings would likely be unwarranted. The main issue lies in the operationalization of critical thinking, which in most cases does not correspond to the concept of AOT. The same issue arises in the categorization of communication disciplines: within forensics, for instance, debate coexists with activities such as acting and storytelling. Furthermore, the meta-analysis included studies conducted during the latter half of the 20th century, whose relevance may be questionable given the societal changes of recent decades.
We are not aware of any studies that have directly examined the relationship between participation in debate and AOT. The most conceptually similar research was the previously mentioned study that tested the effects of different types of informal reasoning instruction, including debate, on myside bias (Perkins, Reference Perkins2019). Interestingly, the results showed that debaters generated even more arguments for their own position in the post-test, with no improvement in the number of arguments generated for the opposing side. While this may indicate that debaters simply became better at elaborating their own stance, it could also suggest a negative effect of debating.
One possible explanation is that debate may encourage the ‘intuitive lawyer’ mode of thinking, especially when participants are preoccupied with winning. In this mode, reasoning is directed toward building a persuasive case for a preselected conclusion. This differs from the ‘intuitive scientist’ mode, in which the goal is to identify the correct or best-supported answer (Baumeister and Newman, Reference Baumeister and Newman1994). Perkins (Reference Perkins2019) explicitly compares debate to law programs by pointing out that debaters, much like lawyers, need both high-quality evidence and persuasive language to be successful.
Another key distinction between Perkins’s study and the present study is that Perkins used a direct measure of myside bias, namely the generation of otherside arguments. In contrast, we turn our focus to AOT norms and standards, which may be particularly relevant for evaluating the trustworthiness of others’ claims and therefore for reasoning and decision-making in societal and political contexts.
Despite this tension, key elements of debate programs align theoretically with the principles of AOT (Baron, Reference Baron1988):
-
(1) an optimal amount of search, proportional to the importance of the problem, is encouraged through preparation for tournaments and regular engagement with relevant topics,
-
(2) fair consideration of alternatives, especially those that contradict one’s initial preference, is essential in debate since debaters do not choose which side of the motion they will defend,
-
(3) calibrated confidence, aligned with the quality and quantity of reasoning, may be supported more indirectly. Debate training requires students to evaluate the strength of their own and others’ arguments, both for case-building and for effective rebuttal. Through repeated engagement in this process, debaters may become more sensitive to the strength and limitations of their own arguments, and thus to whether strong confidence in a conclusion is warranted.
In sum, debate strives to be a more rigorous form of discussion that encourages participants to engage in desirable modes of thinking, but sometimes it may also reward persuasiveness over open-mindedness. This provides a compelling rationale for empirical examination of debate as a potential tool for strengthening AOT.
2. Aim, research questions, and hypotheses
In the existing literature on AOT, we identify a research gap in three key areas. First, the lack of research on AOT during adolescence prevents a deeper understanding of its developmental trajectory during this critical developmental stage. Second, the absence of validated AOT scales adapted for adolescents hinders the empirical examination of its associations with relevant outcomes. Third, although the potential and value of teaching AOT have been recognized, there is still a striking scarcity of studies investigating effective approaches and methods for its development.
Building on these gaps, the present study pursued two main objectives:
(1) to evaluate and refine existing AOT scales for use with adolescent samples, and (2) to explore the relationship between participation in debate and longitudinal AOT changes among high school students. Accordingly, two research problems and corresponding hypotheses were formulated.
Problem 1: To examine the psychometric properties of AOT items drawn from two existing scales and evaluate their suitability for measuring AOT in adolescents.
Hypothesis 1a: We expected that the items comprising the two scales would share a common, meaningful latent structure, but we did not specify a particular assumption about the number of factors.
Hypothesis 1b: The resulting AOT measure was expected to demonstrate criterion validity through its relationships with relevant variables. Higher levels of AOT were expected to be positively related to the ability to recognize fake news and negatively related to epistemically suspect beliefs.
Problem 2: To examine whether participation in a debate club over one school semester is associated with change in AOT among high school students.
Hypothesis 2: The increase in AOT levels over the course of one semester was expected to be significantly greater among debaters than among students in the non-debate comparison group.
3. Methodology
The research employed a longitudinal correlational design with 2 measurement points. At both time points, participants completed an online questionnaire via the Google Forms platform. The study was approved by the Ethics Committee of the Department of Psychology (no. EPOP_2023_24_55).
3.1. Participants
Participants were part of a convenient sample of high school students, mainly from the Croatian capital city and its metropolitan area. In the first wave of data collection, 426 participants completed the questionnaire, of whom 420 were retained for data analysis related to the first research problem. Exclusion criteria are described in the Results section. The sample consisted of 69.5% girls and 28.6% boys, while 8 participants (1.8%) either chose not to report their gender or provided uninterpretable responses. The average age was 16.2 years (SD = 1.28). Participants mostly rated their socioeconomic status as average (38.6%) or somewhat above average (49%), and most reported that their parents had completed higher education. Debaters made up 30.9% of the sample, 15.3% attended journalism or media clubs, 25.4% were involved in other extracurricular activities (which they could specify in an open-ended question), and 28.5% did not participate in any extracurricular activities. Members of journalism/media clubs were initially selected as a theoretically relevant comparison group because such activities may involve information seeking and attention to multiple perspectives. However, they do not necessarily involve the core element of debating, which is defending arguments on opposing sides regardless of personal convictions. Among other activities, sports, music school, and foreign language learning were the most common.
For the second research problem, only participants who took part in both waves of data collection were included in the analysis. A total of 165 participants completed the questionnaire at both time points. Four participants were excluded from further analysis. The final sample consisted of 73.3% girls, 24.8% boys, and 3 participants (1.9%) who chose not to report their gender. The average age was 16.55 years (SD = 1.27). Of the total 161 participants, 40 were debaters, 20 attended journalism group, 51 were involved in other activities, and 50 reported no extracurricular involvement. Sociodemographic characteristics by extracurricular activity group are reported in Table S2 in the Supplementary Material. The groups were broadly comparable, although debaters were on average slightly younger than the rest of the longitudinal sample and had a somewhat higher proportion of mothers with higher education. More detailed sociodemographic information about the sample is provided in the Section B of the Supplementary Material.
3.2. Debate activities
Debate clubs did not follow a fully standardized curriculum, and we did not collect detailed meeting logs. However, all participating clubs were part of a broader competitive debate community and were organized around preparation for and participation in tournaments. Typical debate activities therefore included analyzing motions, searching for relevant information, constructing arguments and rebuttals, practicing speeches or full debates, and receiving feedback from coaches or peers.
On average, debaters had been involved in their high school debate club for 13.8 months (SD = 12.37), and 32.5% had previously participated in debate during primary school. In most clubs, meetings were held approximately once per week, and debaters attended them regularly. On average, they spent 2.86 hours (SD = 2.12) per week on debate activities, including club meetings and individual preparation. At T1, 40% of debaters reported that they had not yet participated in a debate tournament. However, because T1 was conducted early in the school year, this likely underestimates their competitive debate exposure during the study period.
3.3. Procedure
The first wave of data collection took place during October and November 2024, and the second between February and June 2025. Debater participants were recruited through debate club coaches, volunteers of the CDS. In agreement with the coaches, the researcher attended debate club meetings while participants completed the questionnaire. When an in-person visit was not possible, coaches were asked to allocate 15–20 minutes during a meeting for debaters to fill out the survey.
Participants were informed about the purpose of the study and their rights, which were also stated in the informed consent form at the beginning of the questionnaire. Since the participants were 14 years of age or older, they were able to decide independently whether to take part in the study; however, their parents were notified about the research. The notification was sent through the coaches, and a link to it was also available on the first page of the questionnaire.
The second group of participants, who were not debaters, was recruited through school journalism and media clubs, as well as by psychology students in exchange for course credit. Psychology students could recruit up to 4 participants, regardless of their involvement in any extracurricular activities. They were instructed to explain the purpose of the study to participants and to supervise questionnaire completion to ensure independent responding.
For the second wave, the same recruitment channels (debate coaches and psychology students) were contacted again, having been notified in advance that two measurement points were planned. The same data collection procedure was followed as in the first wave. Responses from the two waves were matched using anonymous codes that participants generated and entered at both time points.
Participants did not receive any material compensation for their participation. At the end of the questionnaire, they could optionally leave their contact information in a separate form if they wished to receive a summary of the results upon completion of the study.
3.4. Instruments
The Adolescent Actively Open-Minded Thinking Scale (AAOT; Metz et al., Reference Metz, Baelen and Yu2020) contained 8 items rated on a 6-point scale, 2 of them reverse-scored. An example item is: ‘I can learn things by listening to people I disagree with.’
Actively Open-Minded Beliefs Scale (AOT; Baron et al., Reference Baron, Isler, Yılmaz, Ottati and Stern2023). The AOT scale consists of 11 items (5 reverse-keyed) rated on a 5-point scale. An example item is: ‘Changing your mind is a sign of weakness.’
Fake news discernment was measured using real examples taken from online news portals and social media. Participants were presented with 5 examples of true and 5 examples of false news stories, displayed as screenshots of either social media posts or news portal headlines and subheadings. True news items were taken from mainstream news sources (e.g., BBC), while false news items were sourced from fact-checking websites such as faktograf.hr and admohub.eu. Some of the provided items were written in English, while others were in Croatian language. Participants were asked to rate how accurate or credible each news item was on a 7-point scale (1 = not at all accurate, 7 = completely accurate). The final score was calculated as the difference between the mean rating for true news items and the mean rating for false news items, with a theoretical range from –6 to +6.
The Epistemically Suspect Beliefs Questionnaire (adapted from Šrol, Reference Šrol2022) measures beliefs in the paranormal, conspiracy theories, and pseudoscience. Items that could be difficult for adolescents to understand were excluded, resulting in a 22-item version. Participants rated their agreement with statements (e.g., ‘The horoscope accurately tells a person’s future,’ ‘Childhood vaccines are one of the causal factors in the development of autism’) on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). In our sample (N t1 = 420), exploratory factor analysis (EFA) showed that items related to paranormal beliefs formed a separate factor, while conspiracy theory and pseudoscience items loaded on a shared factor, labeled unconventional beliefs. Items 5, 6, 17, 18, and 20 had low factor loadings and were therefore excluded. Factor loadings are listed in Table S4 in the Supplementary Material. After these modifications, both factors demonstrated satisfactory reliability (Table 1).
Descriptive statistics for actively open-minded thinking, fake news discernment, paranormal beliefs, and unconventional beliefs at first data collection point
Table 1 Long description
The table contains seven columns: Variable, N, Mean, S D, Min, Max, and omega.
* A A O T super a: N 417, Mean 4.13, S D 0.658, Min 1.60, Max 5.00, omega .81.
* A O T: N 411, Mean 3.76, S D 0.508, Min 2.55, Max 5.00, omega .69.
* True news: N 412, Mean 4.64, S D 0.955, Min 1.40, Max 6.80, omega .56.
* Fake news: N 409, Mean 2.86, S D 0.943, Min 1.00, Max 6.00, omega .58.
* Fake news discernment: N 403, Mean 1.78, S D 1.199, Min minus 2.00, Max 5.20, omega forward slash.
* Paranormal beliefs: N 414, Mean 1.49, S D 0.571, Min 1.00, Max 4.67, omega .78.
* Unconventional beliefs: N 405, Mean 2.02, S D 0.704, Min 1.00, Max 4.45, omega .86.
Note: A A O T refers to Adolescent Actively Open-Minded Thinking Scale. A O T refers to Actively Open-Minded Thinking Beliefs Scale. Fake news discernment is the difference between true and false news credibility ratings. Super a indicates A A O T was rescaled from a 6-point to a 5-point scale.
Note: AAOT—mean score on the Adolescent Actively Open-Minded Thinking Scale; AOT—mean score on the Actively Open-Minded Thinking Beliefs Scale; true news—mean perceived credibility of true news items; fake news—mean perceived credibility of false news items; fake news discernment—difference between the mean credibility ratings for true and false news items; paranormal beliefs, unconventional beliefs—mean scores on the paranormal and unconventional beliefs factors from the Epistemically Suspect Beliefs Questionnaire.
a Responses on the AAOT scale were rated on a 6-point scale. Before calculating descriptive statistics, we rescaled them to a 5-point scale to facilitate comparison with the AOT scale.
AOT items are listed in the Appendix and other instruments are available in the Supplementary Material. The reliability coefficients (ω) for all instruments are presented in Table 1.
4. Results
All statistical analyses were conducted using jamovi version 2.6.44, including the SEMLj module (syntax submodule), and R 4.5.2. To address the first research problem, data from the full sample collected at the first measurement point were used, while analyses for the second research problem included only participants who completed the questionnaire at both time points. Participants who did not respond to any AOT items, as well as those with >20% of missing responses on individual items, were excluded from the final samples (n t1 = 6, n t1t2 = 3). Participants with missing values on other scales were retained for further analyses.
One participant was excluded from longitudinal analyses addressing the second research question because their extracurricular activity classification was inconsistent across measurement points: they reported involvement in both debate and journalism at T1, but other activities at T2. As we could not determine whether they had attended debate activities between T1 and T2, or for how long, we decided against arbitrary categorization. The participant was retained in analyses based only on the T1 sample, i.e., pertaining to the first research question.
Journalism/media group participants were also excluded from primary longitudinal analysis. Although journalism was selected as a theoretically relevant comparison activity, the small size of this group (n = 20), together with limited information about activities that students were exposed to, made meaningful comparison between groups difficult. This decision is further elaborated in the Discussion. The primary longitudinal sample therefore consisted of 40 debaters and 101 ‘others’, i.e., students involved in non-debate/journalism activities or no extracurricular activities. Thus, the final analytic samples comprised 420 participants at T1, while the primary longitudinal analytic sample comprised 141 participants. Supplementary longitudinal analyses including the journalism/media group were based on 161 participants.
4.1. Psychometric evaluation of AOT scales
The results of the descriptive analyses for the first measurement point are presented in Table 1. Participants, on average, reported relatively high levels of AOT. The difference between the mean ratings of true and false news items indicates that most participants were generally successful in recognizing fake news. It should be noted that the average ratings were low to moderate for both true and false news, which may suggest a general distrust of the media. Mean scores on the paranormal beliefs and unconventional beliefs factors were low, with positively skewed distributions. This pattern indicates that participants generally did not endorse epistemically suspect beliefs.
We first assessed some psychometric properties of the two AOT scales. Reliability was moderately high for the AAOT but low for the AOT scale. Confirmatory factor analyses (CFAs) assuming a one-factor solution model showed unsatisfactory model fit for both scales, with particularly poor fit for the AOT scaleFootnote 2 (we report details in the Supplementary Material). As expected, these results point to significant item heterogeneity within the scales. This could limit the suitability of simple composite scores, especially when trying to assess subtle changes over time or between groups, which was our primary research aim. Therefore, we tried to refine the existing item pool in order to identify a more internally coherent item subset without compromising validity.
We proceeded to examine the joint latent structure of the two administered AOT measures using a split-sample approach. On the first subsample (na = 210) we conducted an EFA, followed by a CFA on the second subsample (nb = 210) to test the stability of the structure. Because AOT item 4 and AAOT item 8 were identical in content, we averaged their responses after rescaling and treated them as a single item in the joint analyses.
The EFA resulted in a two-factor solution. Factors with eigenvalues greater than 1 were retained, as supported by the scree plot (Figure S1 in the Supplementary Material). All regularly keyed items loaded on one factor and all reverse-scored items loaded on the otherFootnote 3 (Table S7 in the Supplementary Material). This was aligned with the results from preliminary scale-level CFAs, which revealed systematically lower loadings for reverse-keyed, as opposed to regularly keyed items (Table S6 in the Supplementary Material). Furthermore, the two factors were weakly correlated (r = .12).
In addition to construct multidimensionality, these results may indicate method effects associated with item keying. Positively keyed items may partly cluster together because of acquiescence, or a general tendency to agree with statements regardless of their content, whereas reverse-keyed items may introduce additional method variance related to negation, item valence, or social desirability. We therefore tested whether item-keying method factors could account for this pattern in the CFA subsample.
For the CFA conducted on the second half of the sample, we specified a model with one broad substantive factor (theoretically capturing the common core of AOT standards) on which all items loaded, and two additional orthogonal method factors. One method factor was defined by positively worded items, and the other by negatively worded items (Figure S2 in the Supplementary Material shows the path diagram with standardized factor loadings). The bifactor model showed good fit to the data after allowing two items from the AAOT scale, which were highly similar in content to covary. Fit indices are summarized in Table 2.
Model fit indices (robust) for the joint item pool and for the reduced item subset
Table 2 Long description
The table consists of 8 columns and 2 data rows.
Column headers from left to right are: Scale, N, C F I, T L I, S R M R, R M S E A, and R M S E A 95 percent C I (split into Lower and Upper sub-columns).
Row 1: Scale A A O T plus A O T super a (bifactor). N is 210. C F I is .943. T L I is .921. S R M R is .051. R M S E A is .052. R M S E A 95 percent C I is .032 Lower and .069 Upper.
Row 2: Scale A O T minus 9 super b (one factor). N is 420. C F I is .973. T L I is .961. S R M R is .036. R M S E A is .060. R M S E A 95 percent C I is .039 Lower and .082 Upper.
Note: A A O T refers to Adolescent Actively Open-Minded Thinking Scale; A O T refers to Actively Open-Minded Thinking Beliefs Scale; C F I is comparative fit index; T L I is Tucker Lewis index; S R M R is standardized root mean squared residual; R M S E A is root mean square error of approximation; C I is confidence interval.
Note: AAOT—Adolescent Actively Open-Minded Thinking Scale; AOT—Actively Open-Minded Thinking Beliefs Scale; AOT-9—the 9-item AOT composite derived from the AOT and AAOT items based on general factor loadings; CFI—comparative fit index; TLI—Tucker–Lewis index; SRMR—standardized root mean squared residual; RMSEA—root mean square error of approximation; CI—confidence interval.
a Excluding AAOT item 7 and AOT item 2; items 3 and 4 from the AAOT scale were allowed to covary.
b After allowing items 3 and 4 from the AAOT scale and items 9 and 10 from the AOT scale to covary.
After excluding items with low loadings (<.40) on the general factor, the shortened composite (AOT-9) consisted of 9 positively worded items (6 originating from AAOT and 3 from the AOT scale), which we further assessed for internal consistency and validity. Its reliability coefficient was .87 in the total sample. To confirm the one-factor structure of this item subset, we conducted another CFA (Table 2), this time on the entire sample. Due to similarity in content between some items, we allowed two pairs of items to covary, which resulted in satisfactory fit to the data. Table 3 lists the selected 9 items along with standardized factor loadings from the full sample one-factor CFA.
List of items retained in the AOT-9 composite with standardized factor loadings from the one-factor model (N = 420)
Table 3 Long description
The table consists of two columns: Item and Standardized factor loading.
* I can learn things by listening to people I disagree with. (A A O T 1): .725
* People should pay attention to new possibilities. (A A O T 2): .842
* When I disagree with someone, I try to understand why they think as they do. (A A O T 3): .658
* I enjoy conversations with people with different beliefs about the way the world works. (A A O T 4): .579
* I would like to talk to people from other countries about what it’s like where they come from. (A A O T 5): .573
* I respect people who change their mind because of a good argument. (A A O T 6): .688
* People should take into consideration evidence that goes against conclusions they favor. (A O T 1): .657
* When faced with a puzzling question, we should try to consider more than one possible answer before reaching a conclusion. (A O T 9): .610
* Genuine experts are willing to admit to themselves and others that they are uncertain or that they don’t know the answer. (A O T 10): .468
Note: We used Croatian versions of the items, which can be found in the Appendix.
4.2. Correlations between AOT-9 and criterion variables
Table 4 presents the correlation matrix between original AOT scales, the newly derived composite (AOT-9), and theoretically related constructs. AOT-9 showed a significant but weak positive correlation with fake news discernment. Contrary to our expectations, correlations with paranormal and unconventional beliefs were not significant.
Correlations among AOT scales, fake news discernment, and epistemically suspect beliefs
Table 4 Long description
The table presents Pearson correlations among six variables. The variables are A O T 9, A A O T, A O T, F N D, Paranorm. (paranormal beliefs), and Unconv. (unconventional beliefs).
* A A O T correlates with A O T 9 at .921.
* A O T correlates with A O T 9 at .591 and with A A O T at .477.
* F N D correlates with A O T 9 at .215, with A A O T at .168, and with A O T at .239.
* Paranorm. shows non-significant negative correlations with A O T 9 (-.045), A A O T (-.051), A O T (-.065), and F N D (-.070).
* Unconv. shows negative correlations with A O T 9 (-.079), A A O T (-.054), A O T (-.314), and F N D (-.156). It correlates positively with Paranorm. at .356.
Statistical significance is indicated by asterisks: three asterisks for p less than .001, two for p less than .01, and one for p less than .05. Most correlations between A O T scales and F N D are significant at the p less than .001 level.
Note: * p < .05, ** p < .01, *** p < .001, one-tailed. N ranged from 386 to 413 depending on missing data. AOT-9—9-item composite based on general factor loadings from the joint item pool; AAOT—Adolescent Actively Open-Minded Thinking Scale; AOT—Actively Open-Minded Beliefs Scale; FND—difference between the mean credibility ratings of true and false news items (fake news discernment); paranorm., unconv.—mean scores on the paranormal and unconventional belief factors from the Epistemically Suspect Beliefs Questionnaire.
For comparison with AOT-9 as a factorially pure composite, we also examined how the original AAOT and AOT scales correlated with the same variables. The AAOT scale showed a similar correlation pattern, whereas the AOT scale also correlated moderately negatively with unconventional beliefs. All correlations were tested using one-tailed significance tests.
Overall, the 9 retained items formed an internally coherent measure. However, the exclusion of reverse-keyed items and weaker associations with some criterion measures implied that this item subset had more limited validity and construct coverage than expected, which we further explain in the Discussion. Therefore, we decided to prioritize construct breadth over factorial purity for the longitudinal analysis by using the full available item pool.
4.3. Examining the role of debate in adolescents’ actively open-minded thinking
For the longitudinal analyses, we used a combined AOT+AAOT score. Because both scales included an identical item (‘Changing your mind is a sign of weakness;’ AOT4 and AAOT8), we averaged the participant’s responses on those two items after rescaling AAOT8 to match the AOT rating scale. This averaged response was used in subsequent analyses to prevent the same statement from contributing to the combined score twice, thereby reducing the composite to 18 items. The proportion of missing data was low, with only 4 participants having a single missing response across two measurement points. Composite scores were computed as available-data means. The composite showed high reliability in this subsample at both time points (ω t1 = .83, ω t2 = .82). Descriptive statistics are presented in Table 5.
Descriptive statistics for the actively open-minded thinking 18-item composite score across 2 groups and 2 data collections points (Ntotal = 141; ndebaters = 40, nothers = 101)
Table 5 Long description
The table is organized into eight columns. The first column identifies the Group, followed by paired columns for Mean, S D (Standard Deviation), and Min-max range at T 1 and T 2. The final column shows the Mean change score (T 2 minus T 1) with S D in parentheses.
* Debaters row: Mean T 1 is 3.98, Mean T 2 is 4.11. S D T 1 is 0.54, S D T 2 is 0.46. Min-max T 1 is 2.26 to 4.73, Min-max T 2 is 2.76 to 4.76. Mean change score is 0.13 with an S D of 0.27.
* Others row: Mean T 1 is 3.91, Mean T 2 is 3.90. S D T 1 is 0.45, S D T 2 is 0.45. Min-max T 1 is 2.49 to 4.89, Min-max T 2 is 2.82 to 4.89. Mean change score is minus 0.010 with an S D of 0.35.
Note: Change scores were computed for each participant by subtracting their T1 composite score from their T2 composite score.
To examine whether participation in a debate club over the course of one semester was associated with changes in adolescents’ AOT, we compared individual change scores (T2–T1) between debaters and the non-debate/non-journalism comparison group using Welch’s independent-samples t-test. Debaters showed a positive mean change from T1 to T2, whereas the comparison group remained stagnant, which is illustrated in Figure 1. The change was significantly more positive among debaters—t(93.80) = 2.55, p = .013, d = 0.42. The observed effect size is non-trivial given Hattie’s (Reference Hattie2009) benchmark of d = 0.40 for educationally significant effects. Based on meta-analytic evidence, effects exceeding this threshold are typically regarded as desirable and practically relevant. Section D of the Supplementary Material reports additional longitudinal analyses. These include analyses with journalism group participants retained, as well as robustness checks using the original AOT scale, the rescaled AAOT scale, and AOT-9 as alternative dependent variables.
Mean AOT composite scores at T1 and T2 for debaters and other students with 95% confidence intervals (Ntotal = 141; ndebaters = 40, nothers = 101).
With descriptive data showing that many participants’ AOT scores at T1 were high (regardless of the way the composite score was computed), meaning that they could not go much higher between two time points, we were interested in possible ceiling effects. If present, ceiling-related compression could have obscured or distorted the observed group differences in change over time, particularly among those with higher initial scores. To address this, we stretched out the upper end of the scale using a log-odds-style transformation of the AOT+AAOT 18-item composite score and repeated the change score analysis, which yielded the same substantive conclusions as the raw score analysis (t(90.3) = 2.53, p = .013, d = 0.43).
We also plotted the transformed scores to evaluate whether change depended on participants’ initial AOT levels (Figure 2). Assuming compression at the top, we would expect the fitted lines to converge toward the no-change diagonal at the upper end of the scale, or to observe substantial clustering near the top. However, approximately parallel lines indicate that the change remained relatively steady across the range of AOT scores for both groups, suggesting that the observed group difference in change was not noticeably affected by ceiling effects. A similar pattern was observed for the raw scores scatterplot (available in the Supplementary Material).
Scatterplot of T1 and T2 log-odds-transformed scores on the 18-item AOT composite for debaters and other students (Ntotal = 141; ndebaters = 40, nothers = 101).
Note: The solid diagonal line indicates no change from T1 to T2; points above the line indicate increases and points below the line indicate decreases.
Figure 2 Long description
A scatterplot with the x-axis labeled Transformed 18-item A O T composite at T 1 and the y-axis labeled Transformed 18-item A O T composite at T 2. Both axes range from negative 0.5 to 1.5.
At the top, a legend identifies two groups.
* Debaters are represented by solid black circles and a dash-dot regression line.
* Others are represented by open circles and a dashed regression line.
A solid diagonal line starts at the bottom-left origin and extends to the top-right, representing the no-change boundary where T 1 equals T 2.
Data distribution details.
* Most data points for both groups are clustered between 0.0 and 1.5 on both axes.
* The majority of points lie above the solid diagonal line, indicating an overall increase in A O T scores from T 1 to T 2.
* The dash-dot regression line for debaters sits slightly higher than the dashed line for other students, though both show a strong positive linear correlation.
* The debater group shows fewer outliers at the lower end of the scale compared to the other students group.
To gain some preliminary insight into which aspects of AOT might have been most affected by debating, we carried out an exploratory item-level analysis. For each item, we computed individual T2–T1 change scores, along with mean change scores for each group and differences in mean change between debaters and the comparison group. We then correlated item-level change scores with debate club membership. Items were ordered by the size of this association, with positive values indicating greater increase among debaters. Table 6 lists items with strongest and weakest associations with group membership, suggesting that AAOT items related to interpersonal openness to different perspectives, along with an AOT item about scrutinizing one’s own positions, contributed most to differentiating debaters and other students on overall AOT change. Detailed item-level results are reported in the Supplementary Material.
Items with the strongest and weakest associations between item-level change and debate group membership (Ntotal = 141; ndebaters = 40, nothers = 101)
Table 6 Long description
The table consists of three columns: Item, Item content, and r with debate. It is divided into two sections.
Strongest positive associations:
* A O T 5: People should search actively for reasons why they might be wrong. r equals .155.
* A A O T 1: I can learn things by listening to people I disagree with. r equals .147.
* A A O T 5: I would like to talk to people from other countries about what it is like where they come from. r equals .140.
* A A O T 3: When I disagree with someone, I try to understand why they think as they do. r equals .133.
* A A O T 4: I enjoy conversations with people with different beliefs about the way the world works. r equals .116.
Weakest or negative associations:
* A O T 11 (r): It is best to be confident in a conclusion even when we have good reasons to question it. r equals minus .076.
* A A O T 7 (r): When someone disagrees with me, I have less respect for that person. r equals minus .032.
* A O T 6 (r): It is O K to ignore evidence against your established beliefs. r equals minus .007.
* A O T 8: There is nothing wrong with being undecided about many issues. r equals minus .004.
* A O T 1: People should take into consideration evidence that goes against conclusions they favor. r equals .040.
Note: (r) indicates reverse coded items.
Note: (r)—reverse coded.
5. Discussion
The focus of our first research question was the psychometric evaluation of the two administered AOT scales for use with adolescent samples. Through the process of scale refinement, we wanted to identify an internally consistent, but sufficiently comprehensive subset of AOT items, which we further validated by examining associations with relevant criterion variables.
5.1. Internal coherence and criterion validity of AOT measures
Although AOT-9, the shortened composite of positively worded items resulting from factor analyses proved to be reliable and a good fit with the one-factor solution, evidence of validity was weaker than expected. Specifically, the newly derived measure correlated relatively weakly, though comparably to original scales, with fake news discernment, whereas associations with paranormal and unconventional beliefs were nonsignificant. The only measure which showed the expected negative association with unconventional beliefs (consisting of conspiracy and pseudoscientific beliefs) was the original AOT scale.
A possible explanation for this pattern of correlations could be the exclusion of reverse-keyed items from AOT-9. This could have inadvertently led to narrower construct coverage, and subsequently weakened some of the expected relationships. The decision to exclude these items, rather than treat them as a separate subscale, was deemed warranted by low loadings on the general factor in the bifactor model tested on the entire item pool. Furthermore, the reverse-keyed factor did not correlate strongly with the other AOT factor, which would be unexpected unless one assumes that AOT comprises entirely independent subcomponents. Importantly, additional factor analyses and reliability estimates based solely on reverse-keyed or excluded items indicated poor internal consistency and poor fit for a one-factor solution (see Tables S8 and S9 in the Supplementary Material for details). Thus, in a psychometric sense, the reverse-keyed items did not appear to form a coherent or reliable construct, and we attributed the observed factor structure mostly to keying effects.
However, given the superior criterion validity of AOT (which had the largest proportion of reverse-keyed items) relative to the other two measures that we tested, we had to consider the possibility that those items capture a conceptually distinct aspect of AOT. Upon reviewing the content of positively keyed items, we concluded that they may reflect an active component of AOT. This includes seeking out, appreciating, and enjoying in diverse perspectives, as well as beliefs that one stands to gain something from AOT (epistemological sophistication; Metz et al., Reference Metz, Baelen and Yu2020). At the same time, reverse-keyed items may capture resistance to changing one’s stance, rigidity, or a lack of uncertainty tolerance. This seems to correspond to Baron’s (Reference Baron2024) conceptualization of individual differences in AOT as shaped by one’s susceptibility to myside bias, and to uncertainty aversion, which can lead to unjustified overconfidence. The correlation between uncertainty aversion and conspiratorial or pseudoscientific beliefs is theoretically expected, given that such theories have been proposed to function as a buffer against the anxiety arising from a lack of certainty and clarity about the world (Douglas et al., Reference Douglas, Sutton and Cichocka2017). Therefore, reverse-keyed items from the AOT scale could have driven the association between the scale and unconventional beliefs.
Seeing that there was merit to both methodological and theoretical explanations for the reverse-keyed factor, we decided to prioritize construct coverage and criterion validity for the longitudinal analyses, so we went back to the full item pool. The previously excluded items seemed to function well enough within a longer measure, which demonstrated satisfactory reliability and allowed us to detect patterns of change over time and between groups. However, in the interest of future scale refinement efforts, which could be beneficial for studies involving adolescents, we briefly discuss the implications of method effects related to item keying and offer some suggestions for future research.
5.2. Reverse-keyed items: necessity or a nuisance?
The challenge with reverse-keyed items extends beyond grammatical negation or clarity of expression. These items often imply a negative or undesirable trait, producing a valence effect (Kam and Meyer, Reference Kam and Meyer2015). For instance, reverse-scored items on an emotional stability scale essentially measure neuroticism. The negative connotations of such items can evoke different response patterns, partly driven by social desirability concerns, and may lead to spurious factor structures (Kam and Meyer, Reference Kam and Meyer2015). An example from the AAOT scale illustrates this point: ‘When someone disagrees with me, I have less respect for that person.’ This statement suggests hostility or superiority, contrary to normative expectations of politeness and agreeableness. Another item, ‘Being undecided or unsure is the result of muddled thinking,’ can also be interpreted negatively. The reference to indecisiveness may evoke associations with decision paralysis rather than a simple lack of strong opinions.
The apparent coherence of positively keyed items may also partly result from acquiescence, or the tendency to agree with questionnaire items regardless of their substantive content (Krosnick, Reference Krosnick1999). Thus, item-keying effects in our data may encompass both difficulties introduced by reverse-keyed items and biased response tendencies affecting positively keyed items.
Future studies with adolescent samples would benefit from developing well-designed reverse-keyed items to cover the uncertainty aversion component of AOT while preserving psychometric soundness. Consistent with Kam and Meyer’s (Reference Kam and Meyer2015) recommendations, such items should be formulated in a neutral tone, avoiding evaluative or socially undesirable implications. For example, the items noted above could be rewritten as: ‘When someone disagrees with me, I find it hard to understand their point of view’ and ‘Not having strong opinions on many issues is the result of confused thinking.’ Ultimately, balanced scales should include both positively and negatively keyed items across all dimensions to ensure robust measurement without introducing method effects.
5.3. Debate as a potential tool for teaching AOT: implications for practitioners
For our second research question, we hypothesized that students involved in debate would experience a significantly greater increase in AOT during the course of a semester than students not involved in debate. The results generally supported this hypothesis. The difference in change scores between debaters and non-debate/non-journalism students was statistically significant, and the results remained robust to a log-odds-style transformation used to account for ceiling effects. Although many participants showed high AOT scores at T1, it seems that their starting point did not substantially affect the trajectory of change.
Supplementary analyses involving journalism participants also confirmed the observed trend, with debaters differing from non-debate/non-journalism students, while journalists could not be reliably distinguished between the other two groups. In general, comparisons with the journalism group are more difficult to interpret due to subsample characteristics, the most obvious one being group size (n = 20). We had limited information about the actual content and structure of the journalism/media activities in which students participated. Unlike debate, whose core necessarily involves preparing, refining, and responding to arguments on opposing sides of a motion, journalism/media activities are likely to vary deeply between different school settings. This is important because exposure to different viewpoints does not necessarily entail AOT if it is done superficially, without sufficient scrutiny and evaluation of arguments. In some contexts, journalistic practices may take the form of uncritical ‘both-sides’ coverage or giving disproportionate space to poorly supported positions (see Terzian, Reference Terzian2025 for a more detailed discussion of the repercussion of ‘bothsiderism’ in contemporary media).
Additionally, around half of the journalism group participants reported that their group did not meet regularly or met only once a month. This was considerably less frequent than participation reported by debaters (Table S3 in the Supplementary Material), making it less likely that journalism/media activities would produce measurable changes in AOT within a single semester. The observed pattern in this group was therefore unexpected: journalism participants showed the steepest increase, but they also started from an unexpectedly low baseline and only reached the level of the comparison group at T2. Given the small subsample, lower starting point, and uncertainty regarding the nature and duration of participants’ involvement in the activity, it remains unclear whether the observed change reflects the extracurricular activity itself, regression to the mean, or simple measurement error.
Alternative AOT operationalizations yielded similar conclusions as the 18-item AOT+AAOT composite, except for the AOT scale, for which the group difference in change was weaker. The exploratory item-level analysis further suggested that the difference in AOT change between debaters and other students was mostly driven by items from the AAOT scale, although one of the strongest associations was found for an AOT item referring to the active search for reasons why one might be wrong. While not definitive, this analysis offers some insight into which aspects of AOT might have been most strongly influenced by debate training. Items referring to questioning one’s own stance, understanding and seeking out different perspectives, and showing curiosity toward others’ points of view, seemed to have the most pronounced relationship with debate participation.
Somewhat curiously, there was an item with a noticeable correlation in the opposite direction: ‘It is best to be confident in a conclusion even when we have good reasons to question it’ (AOT11, r). One possible interpretation is that this item may have invoked a specific aspect of debate experience. Debaters are routinely put in a position where they need to defend a conclusion despite having good reasons to question it, at least within the context of a debate round. This item may have incurred reflection on that specific experience, instead of demonstrating that debate tends to make those who take part in it stubborn. Overall, our results suggest that debating may primarily encourage endorsement of norms of otherside thinking, interpersonal openness, and engagement with counterarguments, while evidence for change in uncertainty-related items was less consistent.
Furthermore, we discuss the practical significance and implications of our findings. Hattie’s (Reference Hattie2009) d = 0.40 hinge-point for meaningful effect sizes in educational interventions originates from research on academic achievement, so it should not be treated as directly applicable to thinking dispositions. Nonetheless, it merits attention that our crucial result, the difference in the size of AOT change between debaters and non-debate/non-journalism participants, reached this threshold. Given the methodological constraints and limited empirical evidence on the teachability of AOT, as well as the relatively short duration of one school semester, our findings offer grounds for optimism.
Several features of debate may contribute to the development of AOT. Debate programs routinely emphasize and teach the importance of seeking information on current social issues, and considering alternative positions is an essential component of preparation (Snider and Schnurer, Reference Snider and Schnurer2006). Debate also provides a structured framework for evaluating the quality of arguments using both formal and informal logic. This is particularly evident in debate adjudication, which debaters sometimes practice as a peer-learning exercise. A major advantage is the continuity of engagement. Students regularly participate, gain new experience, and practice reasoning skills, which is a key prerequisite for sustaining effects over time (Erceg et al., Reference Erceg, Andrić, Bosilj, Britvić, Čeko, Dedić, Diatlovska, Jakopović, Jakopović, Jelić, Komlinović, Kostanić, Leverić, Neznanović, Ostoić, Prevendar, Tadijan, Tokić, Vukasović and Galić2024).
At the same time, the present design does not allow us to determine which specific elements of debate were responsible for the observed change. Debate clubs, like many informal educational programs, do not strictly follow a unified curriculum, and training methods may vary across coaches, schools, and debate formats.Footnote 4 This calls back to the concern raised in the Introduction. Debate may encourage AOT when students use it as an opportunity to truly engage with competing arguments, but it may be less beneficial when it primarily activates the ‘intuitive lawyer’ mode of thinking, in which reasoning serves the goal of defending a preselected position (Baumeister and Newman, Reference Baumeister and Newman1994). The practical question, therefore, is not only whether debate can encourage AOT, but what kind of debate instruction is most likely to do so.
Based on existing literature, some recommendations can be made regarding how debate (and similar programs) could most effectively foster AOT. First, it is important that debate is not treated solely as a competitive game in which winning is the primary objective. This recommendation is consistent with recent findings from the Croatian high school debate context, showing that mastery-approach goals were positively associated with self-reported development of several debate-related competencies, including those most closely related to AOT, whereas performance-approach goals were generally unrelated or negatively related to competence development (Krpanec and Džafić, Reference Krpanec and Džafić2026). When arguments are used only as strategic instruments, debaters may develop technical argumentative skills without reflecting on broader implications for personal and civic decision-making (Fine, Reference Fine2000). Therefore, debate coaches should model and encourage reflection on personal beliefs and motivations for debating and explicitly communicate why AOT matters for making good decisions and achieving goals, as emphasized by Baron (Reference Baron1993).
It is also important that debaters are exposed to a broader range of ideological perspectives; this is not always the case, as debate communities often lean liberal (Zwarensteyn, Reference Zwarensteyn2012). Ideological homogeneity may limit opportunities to engage with opposing views and may prevent rigorous testing of deeply held beliefs. Although adjudicators are trained to evaluate arguments objectively, biases such as favoring arguments that align with personal beliefs are difficult to eliminate entirely (Stanovich and West, Reference Stanovich and West1997), and this phenomenon is anecdotally recognized by debaters themselves (McKaiser, Reference McKaiser2009). Therefore, awareness of bias should be an explicit component of debate education.
Finally, our findings point to the potential value of integrating debate-based instruction into formal education as a means of developing civic competencies. Such integration could extend benefits to students who do not have access to extracurricular debate or who choose not to participate in competitive formats.
5.4. Study limitations and future research suggestions
The most important methodological limitations of this study stem from the characteristics of the sample and the correlational design. Sample size, particularly within the debate and journalism subgroups, was a constraining factor in terms of statistical power. With a larger number of participants, especially students assessed from the very beginning of their debate involvement, it would be possible to estimate more precisely how AOT changes throughout sustained exposure to debate education. This is important because some participants may already have had substantial debate training before the first measurement, meaning that part of the potential effect could have occurred before the study began.
At the same time, this limitation should be understood in the context of the small population from which the sample was drawn. The total population of active high school debaters is relatively small; approximately 180 students participated in last debate season’s first tournaments (CDS, 2024a, 2024b), which is a reasonable indicator of the overall size. A substantial number of beginners withdraw after the first tournament, contributing to the common problem of attrition in longitudinal studies. Since most students join debate at the beginning of high school, attrition may partly reflect real dropout rates from debate programs.
Self-selection represents another sampling issue. Debate is a voluntary extracurricular activity, and students who choose to participate may differ from other students before any potential effects of debate occur. Even if we assume that students do not join debate because they already possess higher-than-average levels of AOT, it is possible that they represent a group more receptive to efforts aimed at fostering AOT. This limits the generalizability of results to students who would not independently choose debate, such as those who might encounter it in a classroom setting. Additional limitations relate to the convenience sample. Participants were grammar school students, typically of higher socioeconomic status and from the nation’s largest urban area. Future research should include larger and more sociodemographically diverse samples.
In addition to the measurement issues discussed above, the present study relied on self-report scales assessing beliefs about AOT rather than actual reasoning behavior. Participants appear to strongly endorse AOT norms, but this does not necessarily translate into the way judgments and decisions are made in the real world. To better understand the gap between norms and behavior, it would be useful to incorporate more objective measures, such as situational judgment tests, alternative-choice reasoning tasks, or peer and teacher ratings (Erceg et al., Reference Erceg, Vrhovnik, Galić and Ružojčić2025; Metz et al., Reference Metz, Baelen and Yu2020). Ideally, multiple assessment methods should be used.
One final limitation arises from research design. Although repeated measures and comparison groups enabled insight into AOT change across different groups, the correlational nature of the study prevents causal conclusions. Future work would benefit from more measurement points, longer follow-up and controlled interventions to examine long-term effects of debate participation. Qualitative methods could provide richer understanding of how debate is practiced in real settings. The exploratory item-level analyses reported here may help generate hypotheses for such future work, but they should not be interpreted as conclusive evidence about specific effects and mechanisms of debate training.
6. Conclusion
This study examined the measurement of AOT in adolescents and explored whether participation in competitive debate is associated with AOT development over the course of one school semester. The psychometric analyses showed that existing AOT measures are affected by item heterogeneity and method effects related to item keying. Although a shortened positively worded composite showed good internal coherence, its construct coverage and criterion validity were more limited than expected. These findings suggest that future scale refinement should balance reducing method effects with preserving key aspects of AOT, particularly uncertainty aversion.
The longitudinal results supported the hypothesis that debate participation is associated with increases in AOT. Debaters showed greater change than students not involved in debate or journalism/media activities, and the item-level pattern suggested that this change was most visible in openness to opposing perspectives, engagement with disagreement, and active search for reasons why one might be wrong. Although the observational design, convenient sample, and reliance on self-report measures limit causal conclusions, the findings point to debate as a promising context for fostering AOT. Future research should examine this possibility using larger and more diverse samples, additional measurement waves, and multimethod assessments that capture both AOT beliefs and reasoning behavior.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/jdm.2026.10048.
Data availability statement
All data are available at: https://osf.io/jqktp/overview.
Acknowledgements
Generative AI tools were used for language editing, translation, and improving the clarity and style of the manuscript. All intellectual content, research design, analyses, and interpretations were conducted solely by the authors. The authors reviewed and verified all AI-assisted outputs and take full responsibility for the content of the manuscript. This paper is based on a master’s thesis which is available in Croatian here: https://urn.nsk.hr/urn:nbn:hr:131:393667. We would like to thank Prof. Aleksandra Huić for valuable insights regarding the discussion section. We are also grateful to the editor and reviewers for their constructive feedback on earlier versions of the manuscript, including helpful analytic suggestions.
Funding statement
This research received no specific grant funding from any funding agency, commercial, or not-for-profit sectors.
Competing interest
The first author is a member of the CDS and collaborates as an external associate in volunteer training activities. The research was not commissioned or funded by the organization, and none of its representatives had any role in study design, data collection, analysis, or manuscript preparation. The remaining author declares no competing interests.
Appendix
AOT scales with Croatian translation
Adolescent Actively Open-minded Thinking Scale (AAOT, Metz et al., Reference Metz, Baelen and Yu2020)
-
1. Mogu naučiti stvari slušajući ljude s kojima se ne slažem.
[I can learn things by listening to people I disagree with.]
-
2. Ljudi bi trebali obraćati pažnju na nove mogućnosti.
[People should pay attention to new possibilities.]
-
3. Kada se ne slažem s nekim, pokušavam razumjeti zašto oni misle tako.
[When I disagree with someone, I try to understand why they think as they do.]
-
4. Uživam u razgovorima s ljudima koji imaju različita uvjerenja o tome kako svijet funkcionira.
[I enjoy conversations with people with different beliefs about the way the world works.]
-
5. Volio/Voljela bih razgovarati s ljudima iz drugih zemalja o tome kako je tamo odakle dolaze.
[I would like to talk to people from other countries about what it’s like where they come from.]
-
6. Poštujem ljude koji promijene mišljenje zbog dobrog argumenta.
[I respect people who change their mind because of a good argument.]
-
7. Kada se netko ne slaže sa mnom, manje ga poštujem. (r)
[When someone disagrees with me, I have less respect for that person. (r)]
-
8. Promjena mišljenja je znak slabosti. (r)
[Changing your mind is a sign of weakness. (r)]
Actively Open-minded Thinking Beliefs Scale (AOT; Baron et al., Reference Baron, Isler, Yılmaz, Ottati and Stern2023)
-
1. Ljudi bi trebali uzeti u obzir dokaze suprotne zaključcima koje oni favoriziraju.
[People should take into consideration evidence that goes against conclusions they favor.]
-
2. Neodlučnost ili nesigurnost rezultat je zbrkanog razmišljanja. (r)
[Being undecided or unsure is the result of muddled thinking. (r)]
-
3. Suočeni s novim relevantnim informacijama ljudi bi trebali promijeniti svoje zaključke.
[People should revise their conclusions in response to relevant new information.]
-
4. Promjena mišljenja znak je slabosti. (r)
[Changing your mind is a sign of weakness. (r)]
-
5. Ljudi bi trebali aktivno tražiti razloge zbog kojih bi mogli biti u krivu.
[People should search actively for reasons why they might be wrong.]
-
6. U redu je zanemariti dokaze koji se protive našim dubokim uvjerenjima. (r)
[It is OK to ignore evidence against your established beliefs. (r)]
-
7. Važno je stajati uz svoja uvjerenja čak i kad se pokažu dokazi protiv njih. (r)
[It is important to be loyal to your beliefs even when evidence is brought to bear against them. (r)]
-
8. Nema ničeg lošeg u tome ako je netko neodlučan o mnogim pitanjima.
[There is nothing wrong with being undecided about many issues.]
-
9. Kad smo suočeni sa zbunjujućim pitanjem, trebali bismo pokušati razmotriti više mogućih odgovora prije nego što donesemo zaključak.
[When faced with a puzzling question, we should try to consider more than one possible answer before reaching a conclusion.]
-
10. Pravi stručnjaci priznat će sebi i drugima kad su nesigurni ili kad ne znaju odgovor.
[Genuine experts are willing to admit to themselves and others that they are uncertain or that they don’t know the answer.]
-
11. Najbolje je biti uvjeren u svoj zaključak čak i ako postoji dobar razlog da ga dovedemo u pitanje. (r)
[It is best to be confident in a conclusion even when we have good reasons to question it. (r)]
Open access
