Using sampling error as a measure of reliability, not validity

Last week Jeffrey Henning gave a great #NewMR lecture on how to improve the representativeness of online surveys (click here to access the slides and recordings). During the lecture he touched lightly on the topic of calculating sampling error from non-probability samples, pointing out that it did not really do what it was supposed to. In this blog I want to highlight why I recommend using this statistic as a measure of reliability, but not validity. If we calculate the sampling error for a non-probability sample, for example from an online access panel, we are not representing the wider population. The population for this calculation is just those people who might have taken the survey. For example, just those members of the online access panel who met the screening criteria and who were willing (during the survey period) to take the study. The sampling error tells us how good our estimates of this population are (i.e. those members of the panel who met the criteria and who were willing to take a survey at that particular time). If we take a sample of 1000 people from an online access panel and we calculate that the confidence interval is +/-3% at […]

The ITU is 100% wrong on mobile phone penetration, IMHO

The ITU (the International Telecommunication Union, the UN agency that looks after ICT – information and communication technologies) has produced a useful update on ICT facts and figures. The report is well worth reading and shows, amongst other things: As more and more mobile phones are bought, the growth is slowing. In 2005/6 the global growth rate in cellular subscriptions was just under 25%. In 2012/13 it was down to just over 5%. In the developing world the growth has fallen from over 30% in 2005/6 to just over 6% now. None of which is surprising, but it is nice to know the numbers. The internet continues to grow in all regions and globally. With 77% in the developed world having internet access, and 31% in the developing world. Globally just under 3 billion people are using the internet, almost 40% of the population. About 50% of the households with access to the internet are in the developing world (although that is a much lower penetration rate than in the developed world, 28% in the developing world and 78% in the developed world). Fixed-broadband is much cheaper in the developed world than the developing world, although the price has been […]

Significant Digits – a key element of clearer numbers

I am in the process of writing an introductory statistics book for market researchers. This post and some of the following posts are taken from that book, in an attempt to field test the style, approach, and depth I am employing. All comments welcome. My recommendation is that most numbers in presentations and reports should be presented as 2 or 3 significant digits. I feel that the issue of significant digits is more important than the more frequently discussed issue of decimal places. In a number, the significant digits are those that carry the key details. If a bank robber steals $56 million, the 5 and the 6 are the significant digits – and the million gives the scale of the number. If we say that PI is 3.1416 then we are showing it to four decimal places and five significant digits. Table 1 shows the number of internet users in five key, original, members of the EU; showing the raw numbers and the same numbers using two significant digits. Column B shows the estimates in the format they were downloaded from the InternetWorldStats website. These raw numbers contain 7 or 8 digits, and commas are used to help make […]