Category Archives: Research Methods

“Survey Research” Krosnick 1999

Citation: Krosnick, J. (1999). “Survey Research”. Annual Review of Psychology, 50, pp. 537-367. doi: 10.1146/annurev.psych.50.1.537

Summary by: Alexandra Pounds

Image Credit: The Blue Diamond Gallery

  • Big Picture: The paper offers many points to explain biases and survey error.
  • High response rates do not necessarily mean you’ll get more representative data. You can get representative data even with low response rates. Response rate does not equal accurate or representative data.
  • People who agree to be interviewed/surveyed are likely to believe it’s their social responsibility to participate. This means they will likely be biased towards altruistic answers.
  • Pretesting is important. Three methods:
    • Conventional Pretesting: Test a small number of surveys, then look at common problems to adjust the survey.
    • Cognitive pretesting: Think aloud or probing interview to identify respondents misunderstandings or difficulties.
    • Behaviour coding: Respondents taking the survey are monitored and compared for differences. This is especially helpful for interviews, where interviewee behaviour may change depending on the respondent.
  • Interviews:
    • Rigid interviewing: Questions should be worded identically to every respondent to avoid errors/bias. If respondents ask for help, the interviewer should say something like, “it means whatever it means to you”.
    • Conversational interviewing: this allows free dialogue between the interviewer and the respondent. It may allow the researcher to get more accurate and detailed responses, but can produce more interviewer error.
  • Questionnaires:
    • Open versus Closed questions – there’s a lot of debate on which is more reliable.
      • Open: allows for qualitative analysis; respondents may choose to answer in different ways than the researcher intended;
      • Closed: allows for quantitative analysis; must have comprehensive answers to allow respondents to answer how they want to; may be better for inarticulate respondents;
    • Rating Scales: Data reliability and respondent satisfaction are improved when each rating is labelled with words instead of numbers.
  • Optimizing versus satisficing
    • Optimizing = giving the most accurate and honest answer.
    • Satisficing = giving the easiest answer that the respondent believes will satisfy the researcher.
      • This will happen more towards the end of the survey or in difficult surveys because respondents get tired and lose motivation.
      • This tendency will push respondents to give neutral / “I don’t know” / “no opinion” answers because they are safer or because they require less decision-making. Offering these answers does not increase the quality of data due to satisficing tendencies.
      • Satisficing is less in face-to-face interviews because the respondents feel more accountable to the interviewer.
      • This tendency will cause people to select the first answer listed because it’s easier.
  • Other biases:
    • Confirmation bias: respondents will emphasize recently presented information because they wish to “please” the researcher and because of their tendency to satisfice (choose the easiest answer that will satisfy the researcher).
    • Social Desirability bias: respondents will overstate admirable qualities and downplay shameful qualities to protect their reputation or because they have inaccurately recalled their past behaviour.
    • Acquiescence: “the tendency to endorse any assertion made in a question, regardless of its content”. Agree/disagree, yes/no, and true/false questions are very prone to this. To reduce this bias, it’s a good idea to present the question twice, once in a positive light and once in a negative light.
      • For example: “I enjoy eating apples” versus “I do not like apples.”
      • The average acquiescence effect is about 10%.
      • When people guess, they choose “agree” and “true” more often than “disagree” or “false”.
      • Satisficing can enhance the effect of acquiescence.
      • Yes/No questions are less affected by acquiescence than agree/disagree or true/false questions.
    • Nondifferentiation: where people answer all the questions the same, usually due to fatigue or lack of motivation leading to satisficing.
  • Ratings versus Rankings:
    • Respondents find ratings easier and more satisfying than rankings; therefore, rankings are more susceptible to acquiescence and satisficing.  Respondents usually make more mistakes with rankings than with ratings.

“Pretesting survey instruments: An overview of cognitive methods” Collins 2003

Citation: Collins, D. (2003). “Pretesting survey instruments: An overview of cognitive methods” Quality of Life Research, 12(3), pp. 229-238. doi: 10.1023/A:1023254226592

Summary by: Alexandra Pounds

Photo Credit: Wikimedia Commons

  • Big Picture: Cognitive testing methods are useful for pretesting questionnaires to help the researcher understand how respondents are interpreting and answering questions. Cognitive testing can be done by interviewing respondents or by asking them to think aloud.
  • Surveys should be tested before they go out to make sure all questions are being interpreted and understood in the same way by all respondents. A good way to do that is through cognitive testing.
  • Respondents have a tendancy to “satisfice” = avoid hard work of coming up with an answer by giving an answer they think will satisfy. This leads to error.
  • The Answering Process:
    • Comprehension: Do the respondents understand the question in the way the researcher indented?
    • Retrieval of Information: Can the respondents recall the information correctly and easily? Could they recall events that didn’t actually happen?
    • Judgment: how does the respondent determine whether the information that they’ve recalled is appropriate to the question?
    • Response: If the question is closed (ie, multiple choice), does the respondent have enough options to answer the way they want to? If the question is open (ie, a text box), will the respondent have the mental energy to answer completely?
  • Cognitive Interviewing: Think-aloud versus Probing.
    • Think-aloud works better for self-completion questionnaires: respondents are asked to think out loud as they go through the questionnaire.
      • Respondent-driven, less involvement by the interviewer, harder on respondent
    • Probing works better for face-to-face interviews: respondents take the questionnaire as part of an interview.
      • Interviewer-driven, risk of error from interview bias, easier on respondent
      • Standard Probes:
        • Ask the respondent to paraphrase the question in their own words;
        • Respondents group items/questions to relate them to each other;
        • Respondents answer questions using vignettes, or scenarios/hypothetical situations;
        • Respondents rate how confident they are in their answer;
        • Interviewer measure response latency, or how long it takes the respondent to answer the question as a way to test for confusion or difficulty in retrieving an answer.
  • Limitations of cognitive interviews:
    • qualitative rather than quantitative
    • relies on verbal responses of the respondents – respondents may choose not to bring up or be unaware of their own thoughts
    • may discriminate against respondents who are less articulate, such as respondents who are introverted, or respondents who are not answering in their native language.
    • non-standardized because this method is new