Do you understand device agnostic research?

Most people seem to accept that about one-third of people who take online surveys are attempting to do so using a mobile device, mostly smartphones, tablets, and phablets. Consequently, the market research industry has started talking about the need to be ‘device agnostic’.

The mobile agnostic spectrum
The growth in the demand for device agnostic mobile research, and the growth in the number of people offering ‘device agnostic’ solutions has not resulted in a clear definition of what we mean by device agnostic. If we look at the various things people are doing with mobile market research, we see a spectrum:

• Mobile possible: every question will appear on the mobile device, all the words will be there, but it may look different on different devices, and the user may need to scroll left/right as well us up/down. At the software level, this means not using technologies that do not work on mobile (e.g. Flash). At the design level this term means very little, it includes surveys with long questions, long answer lists, and which last for more than 20 minutes.
• Mobile friendly: every question appears in a relatively convenient format. Images fit on the screen, left/right scrolling is eliminated, and unnecessary screen junk is removed from the mobile version of the survey. At the software level this means being creative in how to render surveys on the mobile, for example a question that might be a slider on the PC becomes a series of text boxes; similarly, a grid question might become a series of questions. At the design level it means the researcher shortens the text for questions and answers, they avoid large grids, they use shorter answer lists, and the survey (on PC and mobile) is made shorter.
• Truly agnostic: every question delivers comparable answers on PC, tablet, and smartphone. At the technology level this means identifying comparable ways of displaying questions on different devices, and it means banning question types that deliver incompatible results. Questions do not have to look the same on each device, provided they deliver comparable results. At the design level it means the questions, answer lists, and instructions all need to work on smartphone, tablet, and PC. It does not mean they have to be the same, for example the instructions for how to view a video could be different for mobile and PC, provided they both work and both and they both result in comparable results.
• Mobile only: refers to research that is only suitable for mobile devices, typically because it uses one or more of geolocation, passive data collection, apps, or ‘in the moment’.

One of the worrying things about this spectrum is that the online access panels suggest the vast majority of online surveys they are sent are either mobile impossible or at best mobile possible – even though about a third of research participants are attempting them from mobile devices.

Achieving device agnostic
The software providers are doing a good job of tackling the technology issues, looking at approaches such as responsive design and providing guidance on what works. There has been some RoR looking at what sorts of questions deliver comparable results on PC and mobile (which turns out to be most types) and which question types tend to give different results (e.g. the multi-select grid).

What is holding device agnostic back is mostly research design. Researchers and research users are trying to hang on to question types and to specific questions that either do not work on mobile or which deliver different results on mobile.

If you conduct online research you either have to prevent people using mobile devices (which alienates research participants and narrows the pool of people being researched still further) or you have to embrace mobile and only use question types and questions that work on PC, tablet, and smartphone.

Researchers and research users need to accept that being mobile agnostic is more than clicking the “Mobile Optimize” button, it is also thinking about the research experience and using a research pallet that is appropriate to the participants.

Will everything be device agnostic in the future?
No! There are some research projects that require a specific channel. For example, the UK’s National Readership Survey, is conducted door-to-door, with interviewers using double-screen laptops, collecting 36,000 interviews a year. This study seeks to measure media consumption and needs to be nationally representative, so it has not been possible (yet) to move it to CATI, online, or mobile. Similarly, there will be some studies that will need to use a PC-sized screen (for a complex choice set for example) and there will be a growing number of projects that will be mobile only, such as ‘in the moment’, geolocational research.

However, my estimate is that 90% or more of online research should be, and needs to be, device agnostic. At present, estimates from the panel companies suggest that 5% to 10% are device agnostic – so there is a long way to go.

What about feature phones?
In most markets in the world, feature phones are still important commercially, socially, and in research. However, feature phones are not a major feature of online research. If you look at the 2014 GRIT 2014 CPR study you will see that globally, with a sample of over 200K, nearly everybody who did an online survey from a mobile device was using a smartphone. If you are currently conducting online research, then the issues that face you are mostly related to the integration of smartphones and tablets. In terms of planning for the future, the Economist this week reports that within 5 years 4 billion people (out of a global total of just over 7 billion) will have a smartphone connected to the internet – so the future is smartphone, even if some people currently need to utilise feature phones.

Want to find out more about mobile?
If you would like to find out more about mobile market research here are three great options:

1. The Handbook of Mobile Market Research, published by Wiley and written by Ray Poynter, Navin Williams, and Sue York. Available from all good book stores and from ESOMAR.
2. The University of Georgia’s MRII Mobile Market Research Course – an online course that leads to a Continuing Education Unit. The course is designed to be completed in about 10 hours of study and has been created by Ray Poynter, Reg Baker, and Navin Williams and will be using The Handbook of Mobile Market Research as a course textbook.
3. Mobile Market Research 101 – a NewMR Webinar, March 26. In this 45 minute webinar (plus Q&A) Ray Poynter will introduce all the key elements of mobile market research, including apps, passive data, geolocation, and survey design. Click here to register – it’s free