Conducting quantitative consumer research is a big endeavour for any company. It takes time, effort, planning and strategizing, and –unless you are doing it yourself, it also requires choosing the right data provider.
But choosing a data provider alone also gets tricky when it comes to deciding what method will be used to target and survey the consumer group you’re after (other than just going with the devil you know). Online consumer panels are the most common methodology for quantitative research many companies and marketers use to carry out consumer studies today. While they are a great option for certain uses, they present many challenges you should consider before investing time and money into understanding what consumers want.
But first things first: What’s an Online Consumer Panel?
Consumer panels are groups of people that have been recruited to answer surveys online. They’re online communities that get paid to answer questionnaires periodically on many diverse topics. When research companies build these communities, they make an effort to make it representative of a certain population i.e: Panels that are dedicated to the French market, or just survey video gamers. The recruiting information presented to prospective panelists usually emphasizes the opportunity for the panel member to affect new products or policies.
Here are a few key things you should know about online panels when recruiting respondents for your next consumer study:
Missing the mark on targeting
The vast majority of consumer panels out there, use non-probability samples (participants are not selected randomly. Instead they are recruited online via opt-in pages where they join the panel). In order to form these panels, researchers try to replicate the composition of a larger population. I.e: Canadian online panels, will try to be representative of the Canadian population at large. If they see they are lacking in women in their 30’s, they will target them with paid ads in order to recruit them online. The problem with these samples is that respondents are recruited through a website that offers incentives for them to join and complete surveys (more on that on the next point). This may only attract certain types of people that are “self-selecting” into the sample, under-representing others that might be relevant to your study, or that are simply attracted by the monetary compensation
Due to these issues, most online panels have poor representativeness which forces research companies to apply certain techniques to reduce respondent bias, like profiling panel members to match the make-up of the targeted population (age, sex, education level, etc), or weighting the sample so it matches relevant demographic or social variables.
Online consumer panels incentivize respondents with either monetary compensation or other perks like travel miles. There is a large portion of people that register to more than one non-probability panel in order to accumulate rewards. These professional survey takers or PST’s are a real problem for the integrity of a panel. A comScore study concluded that fewer than 1% of panel members in the ten largest market research online survey panels in the United States were responsible for 34% of the completed questionnaires. As you can guess, this is a problem when you are trying to reach diverse, real-world consumers that are actually interested in contributing to shaping your brand.
While incentives are a great way to source respondents fast, it also brings up the problem of the quality of the data obtained. A big concern with online panels is that respondents rush through the questionnaires to get their reward, impacting the quality of their responses (and the overall data).
1% of panel members in the ten largest market research online survey panels in the United States were responsible for 34% of the completed questionnaires.
The needle in the haystack might stay in the haystack
Another challenge for consumer panels is that they struggle to target niche populations and obtain enough respondents that meet detailed criteria i.e: respondents from a specific part of town, or people who have specific interests (like practicing yoga, or parents whose kids have dyslexia). Since their samples are mainly recruited online (and through a self-selection process), these panels have a limited amount of respondents. Granted, some web panels have up to 30M users, but even at those large numbers, they might have more trouble identifying respondents within their community that match very specific requirements. In some cases, even making them unfit to conduct consumer research at all.
Also, there is the problem of fake profiles or multiple memberships: users that register into multiple panels or that have more than one account and pose as individuals from a different demographic group in order to obtain the remuneration the panel is offering, skewing even further the data extracted from the sample. These are known as “super-panelists”.
Size. Sometimes it doesn’t matter
Online panels can be huge, some have hundreds of thousands or even millions of members, but the response rates to any specific survey request are frequently very low. This is because not all panel members are active at the time a survey is deployed, so the advertised number doesn’t usually give you access to the full audience.
Making the right choice
With a better understanding of the challenges online panels have, make sure you ask the questions and explore the options available to your specific project. Consumer research is a drawer full of pencils as there are many methods and tools you can use to understand your consumer base. Make sure you go with the methodology and the research team that better understands your needs. If you’ve already been through a consumer panel and encountered challenges with your study, we invite you to explore how Potloc has created a whole new method that uses social media, geo-targeting technology, and non-incentivized surveys to gather high-quality insights.
5 Questions you should ask your consumer data provider:
- What methodology will be used for my study?
- If it’s an online consumer panel, how will the data be weighted?
- What kind of engagement will I see from your community and what’s the completion rate you predict? The higher the completion rate, the higher the “representativeness” of the sample collected.
- What biases can be expected from the panel and how are they addressing them?
- What’s their reach for niche and hard-to-reach audiences?