When to randomise an answer list?
One of the questions I get asked fairly often is when should an answer list, in a survey, be randomised and when should it be presented in the same order to everybody.
In my opinion, the key issue is to think about how the respondent is answering the question in terms of:
- Does the respondent ‘know’ the answer? In which case the questionnaire needs to help them find their answer.
- Is the respondent looking at the answer list and picking the most applicable option or options? In which case randomising the list is highly desirable.
Examples of the first category, where people already know the answer are: How old are you (show the answers in ascending age), gender (nobody randomises male/female do they?), and where do you live (organise list alphabetically or regionally). I would also include, in this category, questions like what make of car do you own, what type of phone do you have, and I might include which of these supermarkets is your main one – if the list is short.
Examples of the second category, where the list guides the selection, include, which of these statements best describes your attitude to …., which of the following beers have you seen advertising for in the last month, which supermarkets do you ever use?
However, randomising can require more thought that simply clicking the randomise button in a survey scripting tool. If I want to ask which soft drinks have you drunk at least once in the last month, I want Coke, to appear next to Coke Light, Coke Zero etc. Sometimes, to help the participant, it is necessary to break the list into several lists, each randomised. So, when asking about which magazines has somebody read in the last three months, you might show a list of weekly titles, monthly titles, and online titles, rather than randomising them as one list.
Of course, even in a randomised list, key options, such as: Other, None of these etc, should remain fixed, at the bottom of the list.
One pedantic point is that researchers should avoid saying they have randomised the list to remove order effects. Every randomised list, the list as seen by a specific respondent, has order effects. The items near the top of that version of the list will be more likely to be picked and specific adjacency effects will exist (where seeing one item next to another changes the chance of it being selected). However, randomising flattens the order effects out, distributing them across all the items in the list and across all the participants, instead of focusing them on the items at the top of the list.
Do you have any guiding rules for when to randomise and when not to?