We sat down with Marc Di Gaspero, Head of Research at Potloc, for a Q&A session to discuss the topic of data quality and his advice for businesses that are looking to conduct sound consumer research in a market saturated by online consumer panels and other traditional methods.

What’s the current state of consumer research?

Answer: I think consumer research is one of those fields that’s taking some time to evolve. We’ve seen great technological advances that have significantly impacted the way consumers behave nowadays, and consumer research needs to be agile to catch up with those behaviors.

With the advent of social media and the mobile phone, researchers gained unprecedented access to consumers everywhere, on a massive scale. That’s a goldmine of real-world data that calls for new ways to access and harvest it. New methodologies on how to survey consumers who are now almost constantly on social media are emerging, Potloc being one of the pioneers to do so. In the end, what should matter for businesses is the quality of the data they obtain from their studies.

Could you talk a bit more about data quality? What are the factors that businesses should be looking for when conducting their research?

Answer: Data quality, and the research methodology used to collect the data, is the most important thing businesses should be looking for when conducting their research. As it is the data and the insights extracted from the right respondents that will drive business decisions and ultimately impact their results. With bad data comes bad insights. Bad insights lead to bad business decisions, and that leads to bad results.

My main concern today as a researcher is that data quality is often overlooked as many businesses try to get a sample of respondents at the most affordable price, and –unfortunately, many data providers are willing or even forced to play this game.

While price is important and will never be taken out of the equation, my advice to businesses would be that they should be more curious and pay more attention to the research methodology used to collect respondents.

Businesses should be asking their providers that use traditional methods (phone surveys, online consumer panels, intercept interviews, etc.), how respondents were recruited to be part of the panel (randomly by phone vs. self-selection); if and how much respondents are incentivized to answer the survey; what are the processes put in place internally by the data provider to guarantee that the data quality is high (how does the panel provider identify phenomenons such as ‘speeders’, ‘straight-liners’, ‘duplicates’, inconsistencies between answers, etc.); and also what is the expertise of the data provider in terms of questionnaire design regarding the client’s industry.

I will personally always lean towards a smaller sample of respondents made of ‘better’ respondents than the opposite. What is the purpose of having a low (theoretical) margin of error if the data collected is poor? 

We’ve all heard many horror stories about online consumer panels: From professionalized respondents to how they struggle to target niche populations. Could you paint a picture-based on your professional experience, on what’s the quality of the output data coming from panels?

Answer: I would say that there are huge discrepancies in terms of the quality of output data coming from panels. Some of them are extremely careful with how they build their "community", have thorough processes to guarantee that data quality is high, and employ questionnaire design experts that are verticalized by industries. Others don’t. Just like in any other industry, there are good and bad apples.

However, there are in my opinion 2 practices that most of the panels out there share and that unfortunately prevent them from achieving the highest data quality they could get:

The first would be the questionnaire’s length that they accept or tolerate. It is not rare to see questionnaires with a duration north of the 15 minutes mark. The problem with this is that you encourage behaviors such as speeders and straight-liners (people who rush through a survey, not actually reading the questions and taking the time to think and answer, just to get to the end of the survey). Respondents that go through a long questionnaire even if they are willing to get through the end, mean the quality of the responses decreases towards the last questions. There’s also a decrease in the ‘representativeness’ of their sample when they collect respondents that are over-interested/concerned/obsessed with the topic of the survey. Do the exercise: Close your eyes and think about the most exciting topic you can think about. Would you answer questions about it for 15+ minutes straight? The truth is that most people don’t.

The second would be the incentives promised to respondents after survey completion. While incentives are obviously not important enough to have respondents making a decent salary out of filling questionnaires, some respondents will actually go through the survey without really paying attention to it just to get the promised reward, or might actually lie to some questions that they believe would disqualify them had they chosen the "right" answer.

How do you ensure high data quality in your everyday work here at Potloc?

Answer: Related to what we have just discussed, here are the concrete things we do today at Potloc to ensure high data quality:

We limit questionnaire length to 6-7 minutes most of the time, with 10 minutes being the top limit we have ever accepted.

We don’t incentivize respondents. Respondents see our posts on social media and answer because it’s relevant to them, not because a reward is promised upon survey completion.

We recruit survey design experts from leading market research companies to make sure our questionnaires are well-structured, easy to fill out, with minimum bias.

Thanks to these actions, our processes put in place internally to check data quality are always enforced, usually discarding very few respondents from the sample collected. Ever since I joined Potloc, I’ve always been impressed –and surprised, with the depth and richness of the qualitative comments we gather when asking optional open-ended questions. There is always a WOW moment with our clients that other methods seem to overlook.

What are the 3 top pieces of advice you can give to businesses looking to conduct meaningful consumer research?

Answer: Always ask the right questions and be curious about the way your research will be conducted:

Price is important, but don’t make it the sole factor in your decision to work with a data provider. Looking solely at the cost of a survey is a mistake when choosing the right data provider. Data quality, while being harder to grasp when looking at the results and harder to be pointed at when an initiative succeeds or fails, is actually what will enhance the probability of success of your initiatives. Quality is the main metric you should evaluate when conducting consumer research.

Be curious about the research methodology your data provider uses, how they ensure high-quality data, and assess how confidently and transparently your data provider answers these questions. If the answer is not clear or hesitant, keep looking.

Be clear about your objectives and your expectations. Your data provider will never get to understand your environment and your business issues as much as you do. You are also responsible for the quality and relevance of the results and insights that will be presented to you at the very end of the project.

Where do you think we are heading in terms of methodologies to conduct consumer research? Online panels have been around for a while, and before them, we had phone and intercept interviews. What’s the next big thing?

Answer: Social Media, definitely. Just to put in perspective: some of the largest consumer panels out there have up to 30M panelists. There are 3.8 billion social media users around the world and all projections see that number climbing steadily. So, naturally, consumer research has to evolve to tap into that audience in a way that allows it to collect data coming from real people, buying and using products in real-world situations.

Today, at Potloc, we only see social media as a middleman, helping us to get in touch with our target populations (We push our surveys through ad platforms such as Facebook Ads Manager. Then once people click on our ads, they are redirected to a survey platform where they complete the questionnaire). But there are so many other ways to conduct consumer research on social media such as using listening tools to run sentiment analysis on users’ comments and much more.

Talking about social media, how would you compare what we do here at Potloc with what online panels do ?

Answer: The approach is quite different. On top of keeping our surveys to a  6-7 minutes limit and not giving incentives to respondents upon completion, there are 2 other things that come to mind:

The first one is that as we don’t actually own and manage a panel, our “panel” is the sum of all social media users out there (3.8 billion worldwide at the time of this interview), which gives us access to millions, if not billions of potential respondents across the world. So, technically, we have the biggest panel on earth! This gives us the ability to survey niche audiences, whether because they are geographically narrow (i.e. surveying respondents from a city district) or because of the qualifying criteria that respondents must respect or the topic to be covered (i.e. surveying respondents who own an electric bike).

The second difference is how we acquire respondents: As we are getting all our respondents through social media, we have developed an expertise in building ad campaigns and communication strategies that allow us to attract the right respondents at an affordable price, without compromising data quality.

To know more about the Potloc methodology, check our website or talk to one of our experts.