Sources of error in questionnaires

The research question has been formulated, the theses have been established, and the variables to be measured are clearly defined. Now, just quickly write down the questionnaire before finally getting started with the analysis! Right? In practice, there’s more to it than that.

Sources of Error in Questionnaires

Setting up a questionnaire isn’t as trivial as it’s often believed to be. If you have no prior experience, you should take some time at this stage to become aware of potential sources of error and how to avoid them whenever possible. You’ll find that this time is well spent, as errors in data collection are difficult—or in the worst case, impossible—to correct later in statistical analysis.

In general, three types of sources of error can be distinguished:

• Characteristics of the question as a source of error
• Characteristics of the respondent as a source of error
• Characteristics of the response as a source of error

The key aspects here are the wording and the order of the questions. So-called order effects can have a strong influence on the collected data, as illustrated by the following example. In a 1950 U.S. study by Hyman and Sheatsley, participants were asked about their agreement with the following statements:

‍Question (A): Free reporting for U.S. reporters in communist countries

Question (B): Free reporting for communist reporters in the U.S.

Agreement when order was A, B: (A): 89.8%, (B): 73.1%

Agreement when order was B, A: (A): 65.6%, (B): 36.5%

Thus, respondents’ judgments can be significantly distorted by the position of the question in the questionnaire. The simplest way to counteract this is to implement question rotation when designing the questionnaire. This ensures that the position is randomized and varies from participant to participant. Alternatively, filler questions can be used to separate directly related questions.

Questions as a Source of Error

When wording your questions, remember: Avoid relativizing filler words such as "actually" or "pretty good." Due to their mitigating effect on the statement being made, they artificially increase agreement and thus distort the results. Instead, use clear wording.

Also, be cautious when using antonyms. For example, if you want to determine public support for delivery drones, you could ask whether they should be allowed or whether they should be banned. Since people generally tend to maintain the status quo, you would likely find that more people oppose a ban than actively support their approval. Try to avoid such formulations as well. Additionally, keep your questions short, precise, and neutral. Avoid negations. Use clear language and steer clear of unnecessary technical terms.

Respondents as a Source of Error

Measurement errors often occur simply because respondents lack the necessary knowledge or experience but still feel compelled to provide an answer. Carefully consider when you truly need a response and when you can include an outside option (e.g., "I don’t know"). If you are asking for personal information, such as age or current employment status, always allow respondents the option to skip the question.

Another common issue is the tendency of respondents to adjust their answers based on social desirability. This effect is particularly strong in face-to-face interviews but should not be overlooked in anonymous surveys either. Therefore, a statement at the beginning of the study guaranteeing anonymity and encouraging honest responses is essential. Additionally, "lie scales" can be used to identify dishonest participants (e.g., "Sometimes I feel jealous when others have good luck.").

Responses as a Source of Error

‍What response format should I provide to respondents? Yes/no? A graded scale from 1 to 10? From -3 to 3? There are countless options, and the right choice depends entirely on your questions. It is generally believed that a 7-point scale is often the best solution for graded responses. However, sometimes it makes sense to exclude the neutral middle to force a tendency—in this case, a 6-point scale would be more suitable. From a statistical perspective, 5-, 6-, or 7-point scales can be considered quasi-metric, allowing for various analyses. With 3- or 2-point scales, the data falls into a nominal or ordinal scale level, which can sometimes complicate the analysis.

Of course, you should label the endpoints of your scales. However, avoid using extreme labels, as they may reinforce a central tendency bias. A scale ranging from "Absolutely terrible!" to "Couldn’t be better!" will produce less nuanced results compared to a narrower scale that respondents fully utilize. If you want to number all scale points, consider whether the scale is one-dimensional or two-dimensional and reflect this in the scale values. For example, with a two-dimensional 7-point scale, it is useful not to count from one to seven but to assign zero to the neutral midpoint and count upward in both directions.

Summary

‍In this article, we provided a brief overview of potential sources of error in questionnaires. As you’ve likely noticed, designing a questionnaire is a complex task. The better your questionnaire is designed and administered, the more meaningful the statistical analysis will be. If you need assistance in constructing your questionnaire, our statistics team is happy to help.

References

Strack, F. (1994): Zur Psychologie der standardisierten Befragung, Berlin/Heidelberg/New York