Nature of Qualitative Evidence in Authoritarian Settings

There seems to be a basic misunderstanding in some of these guidelines of how, in reality, one extrapolates findings from evidence in qualitative research. The call for publicly posting interview transcripts and field notes for purposes of replication reflects this confusion. There is an assumption that one can look at transcripts of interviews and be able to judge whether the author has correctly interpreted the data in their article as though an interview and interpretation correspond in a one-to-one fashion. Moreover, there seems to be the notion that the interpretation of a quote would be the main problem in interview-based research.

In my experience, rarely does one hear something from one or two interviewees and then take it to be an observation that can be reported or analyzed. Moreover, not every piece of evidence can be treated with equal weight. Findings do not come from one or two sources, nor should they. Rather, they are an evaluation and weighing of multiple sources and experiences, including interviews, newspaper articles, the radio and TV, cultural expressions, informal interactions with people, knowledge of the history of a country, survey data, analysis of statistical data on the country or location, participation in various conferences and events, participant observation and other forms of ethnographic research, as well as a sense of the overall context. Even quotes that are used to illustrate a certain perspective or insight reflect an analysis of a situation based on an interpretation of multiple sources, inputs and experiences.

Providing transcripts of individual interviews, field notes, newspaper articles, or radio/TV transmissions therefore demonstrates only so much. I don’t have any problem with linking articles to media citations since these are public sources, but even these citations represent only partial evidence. To include an example or a citation is no more than one data point out of many that are excluded because of a lack of space and the need for parsimony. But to offer all the evidence available in an appendix or online depository is to ask for the impossible unless one has a very narrow topic and minimal research. Nevertheless, an elaboration of the basis and rationale of how evidence was used to come to one’s conclusions is possible and far more important.

This is emphasized time and again by qualitative researchers. Marc Lynch, who works in the Middle East, explains that “my interview notes tell only part of any story developed through deep knowledge of local context, dense webs of interlocutors engaged over years, and immersion in distinctive narratives and interpretive worldviews.”^{Footnote 12} Kathy Cramer, who has conducted years of award-winning field research in Wisconsin on public opinion, points out that “first, asking me to make my data publicly available assumes that any scholar could use it in the form it exists on my computer. That is, the assumption is that if I provide the transcripts of the conversations, and a key providing an explanation of important characteristics of each speaker, any scholar would be able to treat those words as representations of the conversation. I am just not confident that is possible.”^{Footnote 13} This is because the research is based not just on what people say, but also on the context, the individual’s relationship with their research subjects, and the multitude of ethnographic impressions that are accumulated in the process of first-hand field work. One cannot remove the interviewer from the process any more than one can remove the interviewee from the interview.

Checking to verify that the interview transcript matches the conclusions is not the problem. The problem generally is one of interpretation. Seeing the transcript does not help with that because interpretation rarely hinges on one interview. There isn’t the kind of precise correspondence between raw data—or in this case, interviews—and analysis that one finds in quantitative studies. There is no prior non-relational, non-interpretive moment of raw information to which one can reference and it is nonsensical to try to depersonalize and remove identifying information because this would render the notes unintelligible.

This helps explain the many problems that might arise with publishing transcripts. To think one can go back to a transcript and accurately ascertain that the author has ignored or incorrectly construed something or has asked a leading question misses the point. There are countless reasons this is problematic beyond what has already been said. For example, the researcher’s fieldwork observations initially may be quite simplistic, and with time, his or her understanding and ability to evaluate interviewees deepens based on further observations, readings, and discussions. So making public an early interview with some valuable information may also confuse readers because they may contain some problematic questions and answers due to the limitations of what one knows at the time. It may even be at odds with the findings in the article or book that is being written, which is done at a time when one presumably has a better overall picture of the subject at hand.

Some interviewees have an axe to grind, ideological affinity, organizational affiliation, or some other interest, which affects what they say. The interviewer is working with that and may ask and interpret questions in a particular way because of this. The researcher may have asked a question in a certain way because of the way the interview overall was going, the way the individual was answering the questions, or his or her prior familiarity with the interviewee. For example, someone might parrot the party or government position on something, and the interviewer may be trying to get beyond their standard reply. It is impossible to provide explanations for all these complicated background factors and nuances for each reference and interview and to make explicit all these various important and unimportant mental calculations. Publishing such explanations or comments might burn bridges with informants.

Sometimes one sees researchers adopting the official government position and reproducing it as fact. This is a product of inexperience and limited knowledge of a country or context, not one of incorrectly reflecting the contents of an interview. As a reviewer, I know this is the case, not because I have a copy of the interview transcript, but because I am familiar with the rhetoric of the country’s leaders and of the practices of the government over time.

There are other considerations of context and background that factor in the interpretation of one’s data. There are popular myths and stereotypes that come through in interviews and if someone not familiar with the context reads isolated interviews, they may think that the author has misrepresented a certain reality, when in fact, they have weighed various forms of evidence against one another and come to a conclusion that may contradict the interviews themselves in places. Yet one does not necessarily want to challenge one’s interviewees publicly.

To give one example, I have done enough interviews and found enough statistical data to know that donors and the UN played a role in gender quota adoption in a certain authoritarian country I have worked in. However, some activists, who were funded themselves by donors, said categorically that the donors did not play a role because they wished to provide a certain image of what happened. While I convey in my writing that it was mainly domestic actors that played the dominant role, if I were to provide the transcript of the interviews, it would show that they think domestic actors were the only actors. My written work implicitly challenges their view by providing other evidence. However, someone reading such an interview transcript might assume I misrepresented the situation, when in fact I did not. To explicitly explain such a discrepancy would damage my relations with my informants. The interview is generally partial evidence and cannot be taken as stand-alone evidence in the way one might regard a statistical analysis. Obfuscations are even more common in authoritarian contexts, where agendas are often more veiled on the part of both government and non-governmental actors.

In reviewing work, the author may have cherry picked data, but that cannot realistically be assessed without looking at all the transcripts and all the evidence, in other words redoing the entire study. No one is going to do that in reviewing an article or even a book. There are other means by which one can make such determinations, which I will discuss later.

One’s assessment of the same data can also differ. I have encouraged my research assistants to use the data we have accumulated for their own research projects and I often find that their interpretations differ from mine, even though we agree on the facts as they were present during the interviews. We all bring our biases, past experiences, and emphases to our research, something that positivist researchers often ignore or may even reject in the belief that a good researcher can come to an objective assessment of reality. My assistants may be more invested in a particular perspective because of their ethnic identity, or their religious or party affiliation, leading them to emphasize other aspects I don’t find as compelling, just as I am prone to do, even though we all aspire for balance. Different levels of familiarity with the society leads one to perceive an interviewee or event in different ways. This often comes out in our joint deconstruction of the interview afterwards.

The lack of freedom of press may create a situation where people traffic in rumors and hearsay. Such comments in interviews may in and of themselves be interesting, but they cannot be presented as evidence. Interviewees or people they know may have been forced to engage in illegal activities because of the restrictive nature of the regime. Such activities may be relevant to the interview, but how would one provide evidence about such activities without some form of documentation? There are sometimes restrictions on certain forms of discourse (e.g., on talking about the King or Western Sahara in Morocco, or about the Black Decade or military in Algeria), yet in an interview people may inadvertently or indirectly touch on those subjects, and their comments may be relevant, yet they should not be mentioned publicly even if the individual grants permission to have the interview published.

Sometimes in authoritarian contexts one is asking about things where there is no official or newspaper record, and people’s responses may be necessarily speculative. The silence itself is an important political fact. A 1977 coup in Angola cast a shadow on protest for years to come, and to this day people still speak with veiled references to these events, often lowering their voices, as I discovered when conducting fieldwork in Luanda in 2008. There is still no definitive understanding of what exactly happened, but the Angolan government claims there was an attempted coup by factions seeking to destroy the ruling party. It is estimated that the government, with the help of Cuban troops, killed 28,000–40,000 civilians and jailed and tortured thousands in the two years that followed. About 3,000 disappeared. Some of the killing had to do with localized disputes and vendettas that had nothing to do with the alleged coup. The impact of the coup has been felt to this day as it ended all open debate within the ruling party.^{Footnote 14} People are fearful of demonstrations, and these events contributed to the self-censorship that still exists within Angola and the events cast a pall over race relations in the country in ways that people fear to articulate.^{Footnote 15} This unspoken event is in the background of almost all interviews with elites yet it appears nowhere on the interview transcripts.

Often no one really knows what happened and sometimes people think they know, but with varying degrees of certainty. These speculations can’t be treated as fact, but they may be useful in understanding what people believe. Sometimes knowledge is obtained in a serendipitous fashion, especially if people refuse to talk openly about a subject. One may run across an activity or discussion of interest. How does one provide evidence for such an encounter without revealing sources or one’s location, especially where there are no interview notes?

Publicly Posting Transcripts of Interviews and Field Notes

There are other compelling reasons not to make public interview transcripts. If my interviewees in Algeria, Morocco, Uganda, and elsewhere were to think I was going to make their interviews public, even if anonymized and even if the IRB allowed it (which they didn’t), they would never agree to doing the interviews in the first place. Even if there are no security issues involved, there are privacy issues, issues of reputation, of pride, of not wanting to malign other people needlessly, and even of libel to consider.

Redacting data is mentioned as another way around sensitive material. And while this is time consuming, the main problem is that in certain circles, particularly elite circles, it is easy to figure out the interviewee even if parts of the data are redacted.^{Footnote 16} The use of certain terms or turns of phrases may reveal location, ethnicity, or language. Moreover, redacting alerts authorities to information that is potentially sensitive. It is like waving a red flag over one’s most sensitive data to alert the very people from whom one wishes to keep the data. The risk is simply not worth it.

But even if there are no onerous IRB restrictions, do people who are interviewed really want their interviews made public? Will they want to be interviewed if they know the transcripts will be made public, even if redacted? Some studies have shown that respondents do not mind having their interviews cited and published. This may be the case in an open democratic society and around non-controversial topics that do not reveal overtly private material. But these same assumptions cannot be made in authoritarian societies, especially for those of us who study politics.

I have interviewed people in contexts of war and economic crisis, where people have confided in me about other politicians who tried to kill them or succeeded in killing their loved ones. Some admitted stealing, of being raped, of having affairs with key leaders, of sabotaging industrial production to increase prices, and so on. Most of these specific comments should never be made public in any form for ethical reasons but also, in part, because they are potentially libelous. But one might want to write generally about a certain related phenomenon based on such comments. How would one provide evidence without providing actual texts of interviews that people who made the statements never dreamed would be made public?

I study women and politics and women’s movements in Africa, and I can’t imagine people would want some of the things they say publicly attributed to them or their organization or even to the women’s movement and its opponents. They don’t want their strategies, jealousies, frustrations, or weaknesses revealed to their competitors, opponents, or people they are lobbying. The same is true for those who oppose the women’s organizations.

I also cannot imagine making my field notes public. Apart from the obvious danger they might put my interviewees in, how could these notebooks possibly help anyone else? They are written to myself. They include interviews with people I did not want to tape because I felt the person would not talk freely if I taped the interview or because they refused to be taped. They include frank discussions with my assistant and lunch/dinner conversations with friends and acquaintances I did not interview but found their observations interesting nevertheless. My notes include observations of demonstrations, some illegal, in places where journalists are thrown out of the country over covering such demonstrations. They include reflections on the news, jokes, rumors, poems, proverbs, song lyrics, hearsay, and graffiti. They include comments people make that are racist, sexist, or offensive in some way. The discussions I jotted down into notebooks were with people who were not always aware that I was a researcher because of the casual nature of our interaction if it was on a bus or train or at a party. These interactions were not covered in IRB provisions. My notes also include a lot of ephemera: messy abbreviated lists of what I know and still need to know; outlines for articles and books; interview questions for specific people; book and author references; spellings of interviewees’ names; phone numbers of people to contact; to-do lists; shopping lists; recipes; words that have been translated and spelled out in a foreign language; definitions and interpretations of foreign terms and concepts that cannot easily be translated into English; and so on. How would these notes, which are useful to my overall project, benefit anyone else in such a decontextualized fashion? These notes would require interpretation but to type up and then explain them would be pointlessly time consuming and not worth the effort.

Finally, making public field notes assumes that knowledge and understanding is something acquired in discrete pieces rather than something that evolves over time and builds on previous knowledge. What I once thought was important early on becomes less salient as questions are answered through the interviews and as my understanding improves. Thus publishing early musings in a project may be quite useless and confusing to the reader.

Risk in Authoritarian and Other Challenging Contexts

Contrary to what is implied in the DA-RT guidelines, most IRB requirements would prohibit the types of replication data being discussed in the DA-RT guidelines in an authoritarian context. The requirements have tightened and expanded considerably in the last few years at key research institutions, perhaps more than many realize. But even if these restrictions were not there, there are other reasons these guidelines are problematic.

There is considerable risk for researchers, their assistants, and subjects in authoritarian contexts, more in some places than others.^{Footnote 17} Aremu and MacLean have raised the important issue of risk not just to American authors but to their African collaborators.^{Footnote 18} Just in my little orbit of the world, I have witnessed foreign colleagues and their assistants killed while working in Uganda.^{Footnote 19} The head of my research institute, who was also vice chair of the country’s constitutional commission at the time, was also killed under suspicious circumstances, along with a professor of communications; another colleague at our institute was injured in the same grenade attack; and other foreign scholars were told to leave Uganda.^{Footnote 20} Other scholars have faced other kinds of challenges. Michael Schatzberg, an American political science professor and my colleague at University of Wisconsin-Madison, was deported from Kenya under the Moi regime,^{Footnote 21} and another colleague in my department was harassed by the Rwandan government under President Kagame. I have had colleagues imprisoned at different times in Iran’s infamous Evian prison because of their fairly innocuous research on gender.^{Footnote 22} I have been followed and harassed by security forces in Tanzania under President Mwinyi.

Those working in North Africa were sobered by the brutal torture and murder of Giulio Regeni, a Ph.D. student from Cambridge University, in Cairo in 2016, a stark reminder of the dangers that prevail today in today’s authoritarian contexts. Further afield one hears of researchers in Armenia being detained for working in national archives. The large number of intervention letters by the Committee on Academic Freedom,^{Footnote 23} Middle East Studies Association, and scholars involved in the Scholars at Risk network gives one a sense of just how precarious the situation is for scholars within a country.

These kinds of research realities have forced some researchers to self-censor;^{Footnote 24} to select topics that are more doable rather than ones that are theoretically interesting and less studied; and to avoid questions that might put interviewees in jeopardy. In some cases, outsiders have more leeway and can write about issues that local researchers would have more difficulty tackling because the security costs for an outsider are lower (e.g., the repercussions for a local researcher may bear far greater personal costs).

Even if one works in a “safer” environment, the political climate can change for research and it can change rapidly, as Dina Bishara, Marc Lynch, and others have pointed out.^{Footnote 25} In recent years situations have changed for the worse in countries like Turkey, Hungary, Ukraine, Russia, and Egypt. Informants’ comments at one time can be used against them in a later context. It is not always possible to predict when and how this will happen. This makes the publication of field notes and interviews transcripts problematic even in situations that appear safe at one time.

Human Subjects Requirements of Confidentiality

There seem to be protections in the DA-RT guidelines for people who work in authoritarian and other challenging contexts, but I worry about how much flexibility there will be in practice because many journal editors seem to be treating those who have IRB restrictions as the exception, rather than as an integral part of the discipline. This potentially puts them at a disadvantage compared to people doing quantitative work and it is very possible these restrictions will count against them. I am concerned about how aware those implementing the guidelines are of the ethical considerations in and challenges of doing research in non-democratic and conflicted contexts.

Time and Financial Considerations

Several scholars have written about the time and financial burdens such documentation imposes, particularly for those at liberal arts colleges and under-resourced institutions, as well as junior scholars.^{Footnote 26} If, however, it were essential to the integrity of the research, one would indeed need to find ways to adopt such extreme measures. However, as pointed out earlier, it is not essential to demonstrating the integrity of the research, which can be evaluated in other ways, as I will discuss.

Qualitative research abroad is very time consuming and requires considerable skills that take years to acquire and so these requirements, which are not necessary to the integrity of research, are piled on top of requirements that are burdensome and mounting. There simply isn’t any comparison with quantitative research based in the United States or Europe when it comes to the challenges, hoops, setbacks, and risks. In addition to the political and security constraints, we have to learn numerous languages;^{Footnote 27} obtain research funding and write-up funding; get research clearance and in some cases not just national but local research clearance, along with dozens of other complicated permissions (carte de séjour, etc.); establish contacts; arrange for transport; line up research assistants; and navigate the logistics of leaving one’s own institution, home, and country as well as the logistics of moving to a new country and institutional setting. The institutional paperwork and hoops in our home institutions have expanded exponentially in recent years, including the ever-lengthening IRB forms. And as if this were not disincentive enough, we must now deal with new challenges of getting our work published, especially in leading journals, once we return home. The additional burden placed on researchers conducting fieldwork by the DA-RT guidelines is—simply put—untenable, all the more so for those of us who work in authoritarian contexts.

What Would Qualitative Replicability Mean in an Authoritarian Context?

I recently conducted research in Morocco and Western Sahara (known to Moroccans as the “southern provinces”). During the time I was in Western Sahara, eight foreign journalists were expelled from the area and there continue to be reports of expulsions of academics. I was acutely aware of the political sensitivities of carrying out research in this region even though I was not studying the conflict itself. But it was also one of the most fascinating places where I have done interviews in my three decades of conducting field research throughout Africa, in part because women hold an unusually high position in this matrilineal society. I was only able to gain access because of a serendipitous encounter with a Moroccan who had worked in these provinces for six years with the UN High Commissioner for Refugees and who had exceptionally good access and excellent contacts. It is absurd to think that my exact project could be replicated in the same way as a qualitative project. There are only a handful of political scientists who have worked in Western Sahara and most have looked at the issues from a more macro-perspective or with access primarily to the Algerian camps where POLISARIO is based, not in Morocco. But looking at the DA-RT guidelines, one wonders how a qualitative study of the kind I conducted could be published. Moreover, the kinds of questions I am asking cannot be addressed through a quantitative study and the secondary literature is virtually non-existent.

These contingencies should not affect the overarching claims the researcher makes. But there is considerable ambiguity and room for interpretation in the process of getting there, and a level of imprecision that quantitative scholars may find disconcerting. The bottom line is that it is simply not possible to replicate qualitative work in the same way as quantitative work, especially given how idiosyncratic fieldwork can be. No one is realistically going to replicate interviews that are done either with groups of people or specific individuals in reviewing a journal article or book. Obtaining data sometimes depends on accidentally running into a terrific interviewee, or taking part in specific opportunities provided by events like conferences or observing happenings like a demonstration. Working in dangerous places is even more precarious and contingent.

Although it has happened (as in the recent LaCour case), in general it is extremely rare to find someone replicating a study for purposes of their own publication because there are few if any incentives to do so.^{Footnote 28} Even in the LaCour case, the problem was not the analysis of the data, but rather, the lack of collection of data in the first place.^{Footnote 29} Even experiments in the social sciences are rarely replicated because they are hard to get funded, they are time consuming, they are not seen as innovative and original, and therefore they are harder to publish. In sum, they have less academic reward.^{Footnote 30} Nevertheless, a reviewer can have a sense of the project and the depth of the data based on similar work they have done in other contexts.

While on the face of it the demand for replicability and verification seems reasonable, how it will be interpreted in practice is in doubt, especially since it is creating a system where quantitative research is the implicit standard and qualitative research is being thought of in terms of how it measures up to what is required of quantitative researchers. Elman and Kapiszewski call on qualitative researchers to more explicitly describe how they draw observations and generate data. While they acknowledge that interviews are generally not redone, there should be sufficient information so that the readers can “carefully assess whether the authors’ data generation techniques were aligned with the rules of inference and interpretation they were following.” They say reviewers can do this by reading interview transcripts to see if authors asked leading questions, or were biased in their interviews. “Readers could go to the research context in which authors worked and draw observations from different sources . . . and evaluate the consistency between those observations and those drawn from cited sources.”^{Footnote 31}

Certainly in an ideal world all this could happen, if it were warranted. Researchers sometimes interview people who have been interviewed by others, particularly political and civil society elites. But I have never heard of any reviewer of an article or book redoing political science interviews for the purposes of ensuring transparency. If people interview someone who has been interviewed before, they have their own questions relevant to their own project and their own approach. No one would even think of checking whether someone else interpreted the interview correctly. Moreover, the time period may be different and the views of the interviewee may have shifted with age, experience, and life circumstances.

The main reason interviews are not replicated is because conclusions are generally not based on what one person said. They are based on multiple sources and inputs, as I have explained throughout this article. I have reviewed countless articles and books and I cannot think of one instance where I would have needed to verify an interview. I am a fairly critical reviewer and there are many questions that can be asked about the results based on what others have already done in the same country or context and based on what is known from other contexts. Therefore, one has to ask, what is the payoff of this enormous amount of expended effort that is now being required of authors, especially in authoritarian contexts where security and confidentiality considerations are paramount?

It appears there is a basic misunderstanding of what kind of transparency is possible and desirable in doing qualitative research. These misunderstandings are compounded for those doing research in authoritarian contexts. Many scholars have recognized these challenges. King, Keohane, and Verba caution that “replicability of data may be difficult or impossible in some kinds of research: interviewees may die or disappear, and direct observations of real-world events by witnesses or participants cannot be repeated. Replicability has also come to mean different things in different research traditions. In quantitative research, scholars focus on replicating the analysis after starting with the same data.”^{Footnote 32}

For this reason, Peter Hall has called for an emphasis on integrity, because integrity gets at what is the core value of research, which is that it should reflect “an honest and systematic search for truths about society, the economy, or politics.”^{Footnote 33} Transparency is but one means toward that end.

Strategies for Validity

All inference is imprecise, not generally because the researcher is manufacturing, falsifying, or misrepresenting the data—qualitative or quantitative—but because social, economic and political life is messy. In authoritarian contexts, the researcher often has to contend with higher levels of obfuscation and opaqueness because defenders and detractors of the regime have more to lose.

Qualitative researchers are going to make different kinds of mistakes from quantitative researchers. A qualitative researcher is more likely to make mistakes in generalizing to the broader population than a quantitative researcher, but a quantitative researcher may be more likely to inaccurately measure someone’s attitudes in a survey, for example. Both need to acknowledge the limitations of their study, probability of error, and its scope.^{Footnote 34}

So how does one evaluate qualitative evidence without replicating it? How does one go about ensuring integrity in qualitative work, especially in non-democratic settings? Ideally a reviewer has a familiarity with the context or a similar context.

First, it is important to keep in mind that inferences are drawn from an accumulation of evidence of different kinds, which means that looking at a few interview transcripts or field notes will not confirm or disconfirm the findings of the study.^{Footnote 35} Even though one can’t draw a straight line between evidence and conclusions, there still needs to be a rationale for how one came to one’s conclusions. The main kinds of problems I identify in reviewing are related to cherry-picking one’s data; adopting a confirmation bias in which one interprets data in a way that confirms one’s pre-existing beliefs; being too sympathetic to the regime and taking its propaganda at face value or, alternately, being too critical and failing to see any positive gains by an authoritarian state without recognizing that even authoritarian regimes sometimes get things right. Other problems relate to interviewing only one type of population (NGO leaders and not participants, men not women, elites but not local authorities); extrapolating too much from one case without ensuring external validity; and not accounting sufficiently for alternative views and theories.

To ensure validity, authors are generally required to explain their research question, research design, methodology, and methods; justify their case selection; situate the study in the literature; show how it builds on what is known; and say how they collected their data. They are also expected to explain what procedures they used, the duration of the study, and its general location, along with other parameters of the study. For example, they should explain what kinds of people were interviewed, how many, and how representative are they of the general population or of a specific population if it is a targeted study. Do they represent a variety of demographic groupings and if not, why not, and does it influence the findings? Do the interviews represent a variety of views if they are elite interviews? What do historical or earlier studies tell us and has the pattern changed? If so, why has it changed? Do the findings differ from other neighboring regions, other studies, or other contexts? Can the author account for why their evidence is different? They should discuss remaining puzzles or unexplained findings that derive from the evidence. They must explain the logic behind the conclusions they draw and how the evidence supports these conclusions. They must explain scope conditions on their findings and the limitations of the findings or possible biases in the study. In discussing alternative explanations, the author must explain why are they not applicable and why theirs is better.

There are several other ways for authors to ensure validity for reviewers and readers. A common strategy is to triangulate in a number of ways, through using different data sources, methods, researchers, and theories.^{Footnote 36} Authors can draw on different types of data to show how they support the same conclusion (e.g., interviews, newspaper articles, archives, numerical data, and secondary data). They can employ different or multiple methods for the same purpose. Depending on the project, I have combined in-depth interviews with surveys of my own or of others (e.g., Afrobarometer) with historical archival work, focus groups, or participant observation. At times I have also incorporated cross-national surveys in a nested fashion. Working in a larger project with different investigators allows one to compare notes, to prevent bias and blind spots, but also to allow for multiple perspectives. In the case of coders of surveys or content analysis, they can verify that their coding systems align. Enumerators of surveys can compare their findings to ensure reliability. Various theoretical approaches can be drawn upon to interpret data. All of these strategies of triangulation hedge against bias coming from the single case, single observer, single method, and single theory.

Another strategy is to build validity into the research design. This can be done by paying attention to variance by incorporating positive and negative cases, by looking at change over time as one might in process tracing; and by looking at cross-national variance along the same lines through statistical analysis. One can look for external validity to see to what extent one sees the same patterns in other parts of the world under similar circumstances through comparative analysis of secondary data. One can compare similar outcomes with different time periods as well.

Thus, it is not any one piece of evidence that matters. It is the context that matters and how one explains how the evidence fits into that context. It is also a question of depth of knowledge of that context that is important.