Are Online Surveys as Accurate as Offline Surveys?

Bruce Paul
Vice President, International Research
ACNielsen

Many major corporations are turning to the Internet to measure consumer and business attitudes, perceptions, and intentions. At the same time, some companies are reluctant to trust online research and replace traditional telephone research tracking and ad hoc projects with online alternatives. Many companies comfortable with their legacy of telephone, door-to-door, or mail research have resisted even considering online research, based on a vague notion that “it just isn’t as accurate” or “online respondents are different from my customers”. However, whether pushed by budget cuts or the fear of falling behind the younger generation of researchers, many research department heads must eventually take a long hard look at online research and ask: will online research work for my company?

Online research has been used in savvy research markets with high Internet penetration for over a decade now. During that time, a lot of lessons have been learnt about when to use online research and when to avoid it. And during that time, the global Internet landscape has changed dramatically, with a steady increase in Internet usage in developed countries as well as huge increases in many developing markets. Today, online research is available in over 200 countries.

Most of the large, successful consumer products firms (as well as those in almost every other field) are using online research. But other companies should not jump into online research simply out of peer pressure. It is important to evaluate the advantages and disadvantages of online research.

The Benefits of Conducting Surveys Online
The primary advantages of online surveys versus phone surveys are:

• Online surveys eliminate interviewer bias: telephone interviewers can have different effects on respondents’ answers if they are male/female or cheerful/droll or fast/slow. Online research eliminates variations.
• They are usually lower cost: internet surveys involve pop-ups on websites or the emailing of invitations, and after that the respondents complete the questionnaire.
• Faster fieldwork: telephone interviewers can only speak to one person at a time while hundreds of online respondents can all answer at the same time.
• Sometimes more accurate responses: online respondents can read the questions and possible responses themselves, resulting in higher quality responses.
• Greater recall: online respondents do not feel pressured into giving quick answers so they are likely to think of more to say, or more brands, etc before moving on to the next question. Telephone respondents feel more pressure to answer or move on because the interviewer is waiting.
• Sometimes more detailed responses: since online respondents have more time to think, their replies to open-ended questions are often more thoughtful and detailed. Telephone respondents feel pressure to begin responding quickly (often before they have had a chance to think much) and they cannot double-check their responses for accuracy.
• Pictures, sound clips and video clips can be included in surveys: this enables online research to do as much as face-to-face research at a lower cost and with a much greater geographical spread.
• Easier to target lower incidence populations: relatively hard to get at respondents (such as doctors or users of a certain brand) can be targeted through specialised panels or simply larger email approaches.

Some of the Disadvantages
The primary disadvantages of online surveys compared to phone surveys are:

• You are less able to double-check the identity of the respondent: a respondent may not be truthful about who they really are. Online respondents can more easily lie about their gender and age than can other respondents.
• Professional survey takers: some online respondents try to go through the survey as quickly as possible without really reading or thinking about their answers. However, ACNielsen has implemented procedures so that such respondents can be identified based upon their responses and eliminated from the results.
• You can’t explain questions: if the respondents do not understand the question, an online survey cannot clarify like a telephone interviewer can. This can be corrected by making your questions clear and concise.
• You can’t ask follow-up questions: an online survey cannot ask a specific respondent to clarify their answer if it does not make sense or if it goes off on an interesting tangent. This can be done via online focus group approaches.

Conduct a Parallel Test and Compare
The only way to find out how an online survey compares to your current survey approach is to try it! This can be done most directly through a parallel test—ie running the survey using both methods simultaneously and then comparing the results. Although the results are rarely identical (for the reasons mentioned previously), it is important to see if the core metrics line up.

The following metrics should be the same if the online research is to replace another type of research:

1. Is the top-of-mind (#1) mention the same using both methods?
2. Is the rating of products or concepts at a similar level, using both methods?
3. Is any market movement equally indicated using both methods?
4. Are stated behaviours (previous brand usage, previous category involvement) similar using both methods?

These metrics may differ between online and offline research, but may actually indicate that online is MORE accurate than offline:

1. Unaided awareness: because online respondents generally take more time to think about their responses, they will generally be able to recall more brands, more ads, more experiences, etc. This can make the results MORE accurate than other approaches.
2. Complex questions: because online respondents can read and re-read questions, they are more likely to respond accurately. In other approaches, respondents rarely ask the interviewer to re-read the question for them. (Note: this can be eliminated in either case if the questions are kept very simple).
3. Detailed questions: because online respondents do not feel the time pressure to give an answer, they may actually investigate the answers before they respond. For example, in a recent survey regarding credit cards, a large number of online respondents actually walked away from their computers to find their recent credit card statements in order to give exact dollar figures—telephone respondents rarely do this.

A Parallel Test Case Study
In a typical example of a parallel test, the world’s largest apple juice producer wanted to test whether they could move their telephone tracking (of usage and attitudes) from telephone to online surveys in the US ACNielsen conducted an online survey of N=425 respondents in 5 cities for the company at no cost in order to check the viability of switching methodologies. The telephone survey was N=800 respondents. Only the core metrics were tested: top-of-mind awareness, unaided awareness, aided awareness, and the results were as follows:

• Top of mind awareness was almost identical between the two samples, at a national and a city level.
• Unaided awareness was higher for each brand in the online survey since respondents took more time to think about what brands they knew. However, the order in which they mentioned the brands was nearly identical between online and offline. Online respondents were generally able to remember some of the smaller brands that telephone respondents did not.
• Total awareness was identical between the two samples. Such an exact match is extremely rare, even if two identical telephone surveys are conducted, so the data had to be doubly verified.

As a result of the test, the manufacturer decided to switch their U&A tracking to online and has saved almost 25% of the annual project cost, while at the same time getting faster results.

ACNielsen has conducted a number of similar parallel tests in different categories and different countries and found that while most of them show that online surveys yield the same results on key metrics as offline surveys, sometimes they do not. Some brief examples of parallel tests that upheld or refuted the case for switching to online:

Digital cameras in four Asian markets and three European markets (online vs telephone): not recommended for online because of potential inherent bias, since online respondents were more likely to own and use digital cameras since they are associated with sending images online.

Sanitary protection in Asia, Customer Satisfaction survey among current users (online vs mail): recommended to use online since satisfaction levels were nearly identical. Online users were more detailed in describing problems.

Household cleaners in Asia, Usage Patterns (online vs telephone): recommended to change to online since attitudes and usage patterns were similar. Claimed usage of brands was higher by telephone, though this was determined to be overstatement (so online was more accurate).

In conclusion, moving research from telephone or mail or face-to-face to the Internet can increase speed, add accuracy and save costs. But each case should be evaluated carefully, preferably with a parallel test, to make sure that it truly merits the switch.