The Survey Instrument

Designing a survey instrument, especially for use in multiple countries and/or linguistic subgroups, requires careful attention to issues of measurement and translation. Survey items may have different cultural meanings or implications across or within countries. Instrument design should include a team of topical, cultural, and linguistic experts as well as survey specialists to address these challenges. Convening a team that includes those knowledgeable about the country(ies) or key target population(s) ensures that concepts and constructs are designed to be meaningful, relevant, and equivalent across contexts.

Translation of the instrument is another critical and time-consuming step. Translation should seek conceptual equivalence rather than a word-for-word approach. Back translations—that is, the process of translating a survey instrument back into the original language—identify certain problems but are likely to miss many others that can limit conceptual equivalence. A more useful approach is a multistep process including a team that first can perform an in-depth assessment and then the initial translation; undertake an adjudication of any issues that arise; pretest the draft translation; and again adjudicate any problems that arise to yield a final translation for fieldwork.

The design of the survey instrument also has a direct bearing on the quality of data collection and the propensity to fabricate data. Sensitive questions that interviewers are reluctant to ask may be omitted by field teams. Lengthy and/or complex questionnaires that test the patience of respondents may lead interviewers to cut short questions or interviews and potentially fabricate data to avoid respondent fatigue. Elaborate skip patterns that are difficult to implement or skips that significantly change the length of an interview (e.g., if a particular response results in skipping a set of follow-on questions) also may lead interviewers to “pre-select” responses that facilitate implementation of the interview. Question wording and translation quality also are important: better questions and translations are easier for interviewers to read and for respondents to answer. That is, interviews will flow more smoothly, thereby reducing the incentive to manufacture data to avoid an uncomfortable process.

To summarize, it is essential to work closely with partners who have expertise in the relevant context, culture, and language(s) and with interviewers—especially during pretests—to obtain feedback on the survey instrument and to identify challenges to implementing it correctly and completely. It may not be possible to eliminate all relevant challenges, but reducing them as much as possible facilitates data-collection processes, thereby enhancing data quality. These conversations also provide insight about which parts of the questionnaire implementation require the closest monitoring to ensure correct implementation.

Partner Selection and Support

Most multinational survey research projects work with in-country partners to lead data collection; therefore, partner selection is critical in all aspects of data quality. There are key points to consider when selecting a partner.

Conducting high-quality, nationally representative surveys—especially when relying on face-to-face interviews in logistically challenging environments—is demanding work. It is ideal to find a partner that has a strong foundation of relevant experience in the topics, the type of research (e.g., market, academic, or electoral/political), and the methods (e.g., computer-assisted personal interviewing [CAPI] and experiments) of the planned survey, as well as other aspects of the survey-management process (e.g., data management and analysis). Inexperienced partners may have unrealistic expectations about what the work will entail, as well as insufficient budgets and inadequate planning, which creates a situation ripe for problems.

Whereas experience is important, it cannot substitute for a partner’s fundamental commitment to the quality and integrity of the survey. Although electronic data-capture methods make it increasingly feasible to monitor partners throughout the survey process, it will never be possible for PIs and/or sponsors^{Footnote 1} to observe and validate every aspect of data quality. It thus is essential to have partners who are committed to meeting high standards not only because it is in the contract or because the PIs and sponsors might be monitoring them, but also because they are committed to the process and the integrity of the data being collected.

Transparency, referred to by Data-OPM—the AmericasBarometer partner in Mexico—as the “hasta la cocina” principle, is another key indicator, especially with new partners. Committed and reliable partners should be willing to grant unrestricted access to all aspects of the research process, even inviting their collaborators “into the kitchen.” Project PIs and/or sponsors should be able to observe any part of the data-collection process, from enumerator selection and training to pilot interviews and fieldwork, as well as data-validation processes. Ideally, no part of the project should be off limits.

Survey PIs and sponsors also have a critical role in building effective partnerships. They must be willing to pay for and incentivize quality. Local partners must have sufficient resources to conduct the research according to specifications, especially when some of these practices may differ from the partner’s normal practices. Simply stated, the project leader should not automatically accept the lowest bid. Sponsors must be willing to provide adequate resources. This includes avoiding unnecessarily contentious bargaining processes that can lead to under-budgeting, which can generate perverse incentives for local partners to relax standards or avoid complying with the required survey specifications.

PIs and sponsors also should become acquainted with their local partners to identify their skills and capabilities and to provide the necessary technical support and supervision. This is especially critical for less-experienced partners that may not be fully prepared for the demands and expense of conducting a nationally representative survey to a high standard. PIs and sponsors also should consider steps that foster partner commitment, such as investing in the organization and incentivizing high-quality work. Investments take many forms, both financial and nonfinancial, and may include provision of hardware or software, technology transfer, and capacity building and human capital.

Interviewer Recruitment, Training, and Support

The goals are similar for the fieldwork team, which is the most critical link in ensuring collection of reliable data. Maximizing commitment to the survey protocols and minimizing incentives to fabricate requires the development of a pool of interviewers who have a clear understanding of the relevant survey methods and their role in implementing them; understand the importance of data quality; are both safe and well supervised in the field; and are fairly compensated.

Achieving these goals begins with recruiting qualified interviewers. Fieldworkers who understand the context of the survey and the ultimate value of the work that they are doing are more likely to commit to the data-collection process. Comprehensive training in the specific protocols of the survey and thorough familiarization with the questionnaire—including pilot interviews—also are critical. Adequate and standardized training is essential even when working with experienced interviewers who have conducted many surveys. This ensures that they follow the particular survey protocols rather than rely on “usual practices” that may not reflect the same methodological standards. It is ideal to have a representative from the sponsor organization participate in the training not only to monitor and observe but also to impart an understanding of the larger context and body of work to which the survey will contribute. It also is advisable to ask local partners to train more interviewers than required so that weak performers can be removed from the team. Ideally, survey sponsors should participate in the selection or approval of enumerators.

Minimizing the risks that interviewers face during fieldwork—whether from crime and insecurity, government interference or even arrest, or other factors that may threaten their well-being—also reduces fieldworkers’ incentives to deviate from survey protocols. Steps that can be taken in high-risk areas include the following:

• • provide fieldworkers with letters of introduction explaining their work, including contact information for supervisors and managers

• • require fieldworkers to introduce themselves to local authorities before beginning their work in a primary sampling unit

• • substitute sample locations that put the personal security of interviewers at high risk

• • plan work hours and locations to avoid more dangerous periods of the day

• • have interviewers work in teams with knowledge of a project security plan

Finally, PIs and sponsors should ensure that the local partner pays fair wages and makes adequate provision for interviewers while they are in the field. Participating in training sessions or observing fieldwork can help the sponsor to evaluate whether fieldworkers are satisfied with their working conditions. High dropout rates indicate that the demands of the work and the compensation being offered are not well balanced.

Recontact

A common method is to recontact a portion of the survey respondents. As part of this process, respondents usually are asked factual questions about the interview, including when and where it was conducted, content of the survey instrument, duration of the interview, and a description of the interviewer. It also is common to re-ask a small portion of the interview questions to validate that recorded responses—including both demographic and attitudinal variables—are consistent with what was entered by the interviewer.^{Footnote 3}

Recontact is especially important to validate PAPI surveys that were not recorded or observed by a team member other than the interviewer. The use of CAPI for real-time monitoring has decreased the importance of recontact as the primary means for detecting data-quality issues. However, recontact can be used as an additional safeguard or verification in cases in which questions remain about the quality or validity of a specific interview.

Paradata

To ensure data quality, it is essential to record extensive paradata—that is, variables relating to how the survey was collected such as interviewer identity, time, date, and geolocation. Implementing surveys using CAPI facilitates the collection of paradata because most survey programs allow for automatic collection. It also is easier to collect data throughout the interview—for example, by placing timers at multiple points in the survey.

Several quality checks can be accomplished using the paradata. Some of the simplest and most important relate to time stamps. Extremely short interview times likely mean that the full instrument was not implemented properly, whereas extremely long interview times may be the result of a break during the interview or another abnormality. Extremely short duration (e.g., five minutes or less) between face-to-face interviews also signals that the sampling plan is not being followed correctly. The time of day also is instructive because valid interviews are unlikely to take place overnight or at other unusual hours.

Additionally, there is a limit to the number of interviews that an interviewer can conduct each day. A sudden surge of interviews for one or more interviewers—especially close to the fieldwork completion deadline—may suggest a problem. Likewise, if interviewers report conducting interviews in distant locations on the same day, it may indicate a departure from the sampling plan.

Data-Analysis Methods

Several checks within the dataset can be used to identify potential data-quality issues. First, it is important to conduct basic logic checks within the survey, some of which relate to sheer impossibilities. For example, a 25-year-old respondent could not have lived in the same city for 35 years. If an interview is fabricated without an actual respondent present, the fabricator may make such errors. Other checks should include rare response combinations across variables. For example, a respondent who identifies as an atheist is unlikely to also attend church regularly and pray daily. These combinations suggest either fabrication or a respondent’s lack of comprehension or attentiveness, which are data-quality issues.

Undifferentiated response patterns also may indicate fabrication or satisficing that affects data quality. In the simplest form, this outcome includes straight-lining, wherein the same response is selected for each variable in a section—or even throughout the entire survey. More sophisticated fabrication methods include an interviewer asking only a single question and then filling out the remaining items in a battery by inferring the respondent’s likely answers. Principal-component analysis (PCA) determines if an interviewer has a lower variance for a set of questions, which could indicate fabrication by this means (Blasius and Thiessen Reference Blasius and Thiessen2015).

Because of the potential for logic checks or correlational analyses across the dataset, convincingly fabricating an interview is a difficult undertaking. For this reason, a common strategy is to generate fraudulent observations by duplicating valid ones. This form of fabrication can be done by an individual interviewer or another member of the data-collection team. However, given that most statistical programs include means to check for exact duplicates, fabricators may alter a small number of variables to evade these checks. Recent advances, including the use of the PercentMatch program, are effective ways to check for near-duplicate observations to prevent this type of fraud.

Additional checks include looking for short paths throughout the survey, particularly if there are extensive sections with questions that are asked only if a respondent provides a given response to a filter question. Similarly, high percentages of “don’t know” or refusals by the respondent, including break-offs, may indicate a problem. A high percentage of missing telephone numbers, meaning that callbacks are less likely to be performed, is another potential red flag.