4. Data Gathering

Once the data collection method has been developed and pre-tested, we proceed to the gathering of the information we wish to collect. This is where a number of nonsampling errors can occur. These cover all errors that are not related to the sampling plan or the sample size, and that cannot be calculated. However, some of these can be controlled when procedures are set up properly.

During the data gathering phase, nonsampling errors can derive from errors committed by the fieldworker (in the case of telephone or personal interviews and surveys) or by the respondents themselves. In both cases, errors can be intentional or unintentional.

Errors by fieldworkers are referred to as interviewer error (or bias) and can be the result of cheating (e.g., filling out the survey instead of questioning a respondent or choosing a different respondent than the one designated by the sampling plan); leading the respondent by suggesting answers; fatigue due to stretches of interviewing that are too long and tedious; and mistakes in administering the survey correctly, especially if there are skip patterns. Finally, interviewers can also introduce bias through their appearance, demeanor and even their age or gender. Interviewer training, careful supervision, reasonable compensation and validation of completed surveys are some of the techniques to control for interviewer error. A rule of thumb suggests that at least 10% of completed surveys or interviews should be verified by re-contacting the respondent. Unfortunately, this is not possible in anonymous surveys.

Respondent errors can also be the result of cheating (e.g., pretending to match the screening criteria or passing oneself off as different from reality by inflating educational attainment or income, for instance) or introducing bias (non-response error) by choosing not to respond to the survey or to certain questions. Incentives can help in reducing these types of intentional errors but if too appealing, can also lead to more cheating to qualify for the survey. Refusals and item omissions can be calculated as a percentage and should be reported along with the findings. There are a number of tools available to calculate response rates, such as this one by Answers Research Inc.

Unintentional respondent errors are usually due to incorrect interpretation of a question as well as fatigue and distractions. This is why pre-testing a questionnaire is so important: it tells us not only whether questions are understood correctly but also whether it is too long to maintain the respondent’s attention.