Some general rules about asking better questions in online surveys

How To ...Posted by Ray Poynter, 29 May 2020

I am conducting a review into surveys and the questionnaires they use to achieve their aims It will be a while before the work is complete. But, along the way, I will share some thoughts and updated. Here are a few thoughts about some general rules for writing questionnaires, and some examples of how these rules apply.

Note, these rules are general. I will offer some suggestions for how to design questions that link to specific objectives in another document.

Rule 1, there will always be exceptions.
One relatively common exception is where a client wants to use an alternative form of a question. Providing the client has been made aware of why you prefer your version AND provided their version is not fundamentally flawed, then they are the client and you should use their preferred version 

Four Basic Rules of Survey Questions

  1. People have to understand what is being asked
  2. People should be able to answer the question
  3. People need to be willing to answer the question
  4. We must be able to interpret the answers.

Other useful rules

  1. The questionnaire should work well on a mobile – never launch a study that you have not tested on your mobile. If you are working for a client, try to insist the client completes an interview on their mobile before launching too.
  2. There should be at least one option for everybody – i.e. are the options exhaustive? For example:
    • Do we need a ‘Don’t Know’?
    • Do we need a ‘Not Applicable’ or ‘None of the above’
    • Do we need an ‘Other’?
  3. If asking a single pick question, there should only be one option for each person.
  4. Avoid leading questions.
  5. As intuitive as possible – avoid long questions and long answers
  6. Only collect sensitive data if it is needed (and in those cases follow all the relevant rules/procedures)
  7. Only collect data where there is a reasonable need for it and which the participant is likely to feel we have a reasonable need for.
  8. Try to use questions that have been previously tested, new scales developed from scratch should be tested before being used.
  9. Only force answers where necessary, and almost never force people to type something into an open-ended question.
  10. Keep the survey as short as possible. 5 minutes is good, 10 minutes is often acceptable, more than 15 minutes is usually too long to ensure quality.

Bonus Tip – ask a closing, open-ended question – Always
For the last twenty-plus years nearly every study I have run as finished with the same type of last question. The typical form for this question is:

‘Q: Is there anything else you would like to tell us about this topic or about the questionnaire? Please type your answer in the box’

This closing question has proved invaluable over the years and it is the first question I read when I am checking a project. It tells me about the frame of mind of the participants, it highlights possible problems with the questionnaire, and often gives me insights into things I will want to explore in the analysis.

Examples of typical problems, and possible solutions

Here are examples of questions that do not meet these rules, along with some suggestions for how they might be improved.

Not understanding the question

‘Q: How often do you encounter a 404 error when browsing? ‘

Problem
Some people may not understand what they are being asked. Many people are not familiar with the term 404 error.

Alternative

‘Q: How often do you encounter a link that does not work when you are browsing?’

Note

This is not the same as a 404 error (there are other reasons the link will not work), but it is easier for people to answer – this is a constant trade-off that needs to be assessed, the balance between getting the information you desire and the ability of people to answer the question.

Not being able to answer the question

Q: How many units of alcohol do you consume in a typical month?

Problem
1) Many people do not know how to convert a pint of beer or a glass of wine into a unit of alcohol.

2) Unless the number of drinks in a typical month is zero, many people will struggle to know how many alcoholic drinks they had in the last month, never mind in the mythical ‘typical month’.

Alternative

Q: In the last week, how many glasses of drinks containing alcohol did you drink?

  1. None
  2. 1 to 3
  3. 4 to 7
  4. 8 to 12
  5. 13 to 20
  6. 21 or more
  7. Not sure

Note

The answers may need adjusting, depending on the objective of the question. For example, if the difference between 1 and 2 drinks is important, then 1 drink and 2 drinks need to be separate answers. In your study you might not need the same number of codes at the higher end, or you might need more. In general, avoid participants typing in numbers, it tends to lead to lots of mistakes.

The alternative question no longer measures units, it measures drinks. We are treating a small glass of wine, a large glass of wine, a glass of beer and a pint of beer as one unit. This is less accurate in terms of measuring units, but reflects the accuracy a typical participant can give. If you actually need to know units (as in some health studies), then you need to supply a lookup system for the participants.

The question does not measure a typical week, it measures the last week. The researcher may want the typical week, but research has shown that humans can’t reliably answer ‘typical’ or ‘average’. If you are particularly worried about whether this measurement is too specific to the last week you could ask a follow-up question. For example, something like “Q: Was the amount of alcohol you consumed in the last week … a) A lot less than normal, b) less than normal, c) about the same as normal, d) more than normal, e) a lot more than normal”

Not being willing to tell the truth

Q: Do you cheat on your expenses?

Problem

Most people who do their expenses honestly are likely to answer this question truthfully. However, many people who overclaim expenses may not tell the truth. Part of the reason for this is social desirability bias, but it is also partly due the question being a leading question (the world ‘cheat’ is heavily loaded).

Alternative

Q: In companies like yours, what percentage of people you think claim more on their expenses than was intended by the company?

  1. 81% to 100%
  2. 61% to 80%
  3. 41% to 60%
  4. 21% to 40%
  5. 1% to 20%
  6. None

Note

This type of projective approach leverages research that shows that people are quite good at predicting what others do, often better than we are at estimating our own behaviour (see Mark Earls, HERD). The long list of percentages of people who overclaim expenses leverages BE thinking to avoid the scale being anchored on the socially desirable ‘None’.

Not being able to interpret the answers

Q: Was the course interesting and useful?

Problem:

If somebody answers No, we do not know if the course was interesting but not useful, useful but not interesting, or neither interesting nor useful. This type of question is referred to as a double-barrelled question.

Alternative

Q: Was the course …

  1. Interesting (Yes/No)
  2. Useful (Yes/No)

Note
If there is a longer list of attributes, consider showing the list and asking the participant to select all that apply – along with a None of these.

Not having exhaustive options

Q: Which of these meats do you prefer?

  1. Beef
  2. Chicken
  3. Pork

Problem

Not everybody eats meat, and even if they do, they do not necessarily eat any of these three. The list does not include an item that every participant can select.

Alternative

Q: Which of these meats do you prefer?

  1. Beef
  2. Chicken
  3. Pork
  4. None of these

Note

None of these is one of the ways we can make a list exhaustive. Other options include Don’t Know, Not Applicable, or Other (specify). The right way to make a question exhaustive will vary from question to question, and project to project.

Conflicting Answers

Q Are you … (Select one)

  1. A qualitative researcher
  2. A quantitative research
  3. Neither
  4. Not sure

Problem

Somebody might want to answer that they are a qualitative researcher AND a quantitative researcher.

Alternative 1 – make it a multi-select

Q: Are you … (Select all that apply)

  1. A qualitative researcher
  2. A quantitative researcher
  3. Neither
  4. Not sure

Alternative 2 – stay with single select, but change the options

Q: Which one of the following best describes you?

  1. A qualitative researcher
  2. A quantitative researcher
  3. Both
  4. Neither
  5. Not sure

Note

As with all the other cases, the choice of the alternative form depends on what the researcher needs to know.

Leading Questions

Q: Do you prefer French wine to German wine?

  1. Yes
  2. No
  3. Neither
  4. Don’t know

Problem

We are leading with a suggestion that French wine ‘should’ be preferred to German wine, because of order and acquiesce biases.

Alternative

Q: Which of these do you prefer?

  1. French wine
  2. German wine
  3. Nether
  4. Don’t know

The order of French wine and German wine would be randomised.

Note

There are other ways of a question being leading, for example by using loaded words. The example earlier of “Q: Do you cheat on your expenses?” was an example of a leading question, because the word ‘cheat’ is negatively loaded.

Your suggestions?

I would love to hear from you, in terms of questions such as (but not limited to):

  1. Which parts do you agree with?
  2. What would you change?
  3. What would you add?
  4. Other comments?