Incentivized Reviews: How Accurate Are They?

Research shows that 98% of customers read product reviews. It’s no secret that the sheer numbers of online reviews are skyrocketing as consumers share more and more data on their product and customer experience. As reported by Forbes, the vast majority of the world’s data was created in the last few years.

With so many online reviews out there, we decided to look at how incentivized consumer reviews fare against regular reviews from consumers that were not incentivized. Is one better than the other? More positive? Are they similar in the topics they cover and level of detail?

Incentivized reviews are written by consumers who received the product for free or got a discount coupon in return for a review.

Consumer insights professionals often read product reviews seeking an unmediated picture of what their customers are thinking. The question is, how accurate are the insights gained from incentivized reviews?

We used the Revuze Explorer platform to analyze data collected from the razors and razor blade industry. Of course, the results would differ for each industry, brand and product, but since this is based on a substantial sample of data (more than 300,000 reviews) it can be considered representative.

Distinguishing Between Incentivized and Non-Incentivized Reviews

Revuze offers an AI-powered product insights platform that can take any type of unstructured data about consumer products and, through automated, deep understanding of consumer intent, use it to understand an entire market — brands, products and features reviewed. Data sources include online reviews, emails, social media, call transcripts, etc.

Revuze automatically identifies if a review is incentivized, whether the review was marked as such by the reviewer or by the ecommerce site (e.g. “This review was taken as part of a promotion”).

Leveraging its deep consumer understanding, Revuze technology detects all possible variations of how consumers specify that this is an incentivized review (“got coupon,” “part of promotion” etc.) and marks these reviews as incentivized.

Now that the incentivized reviews are easily identifiable we can check their volume and notice that in this sector (razors and blades) incentivized reviews make up 28% of all reviews. This varies according to industry, for example in the fabric softener industry incentivized reviews comprised 35% of all reviews in 2018.

Incentivized Reviews Boost Star Ratings and Sentiment

If we isolate the organic reviews from the incentivized reviews, we can see how they affect the star rating and sentiment.

Looking at these images we immediately see that the incentivized reviewers were more positive towards the product:

• Average star rating of incentivized reviews is 3.5% higher than non-incentivized reviews (4.48 vs. 4.33 stars) 
• Average sentiment of incentivized reviews is 8% higher than non-incentivized reviews (91% vs. 85% positive sentiment) 

From a marketing perspective, this boost to star rating and sentiment can be a boon to sales, as customers reading the positive reviews may be more compelled to purchase the product. However, for consumer insights professionals, this increase in positive sentiment can mask important issues, which we will explore below.

Incentivized Reviewers Focus on Shorter-Term Product Aspects 

What makes the sentiment more positive on the incentivized reviews side? We can dive deeper into the data using the Revuze System’s capabilities to understand what causes these ratings and sentiment differences. 

Let’s examine the top 10 aspects/topics that affect the star rating. The images below show the top 10 aspects (by volume) for reviews with high score (4-5 stars) vs. reviews with low score (1-3 stars).

Now we can see clear differences between the two groups:

• Incentivized reviewers mention that they received this product as part of a promotion
• Price features strongly in the non-incentivized reviews, however the incentivized reviews do not tend to evaluate the price of the product. In the non-incentivized reviews, price is the second most evaluated topic while in the incentivized positive reviews it is not in the top 10 mentioned topics
• Incentivized reviews appear to be delivering reviews a short time after initial use of the product, so they focus on overall experience and less on long term usage feedback. Non-incentivized reviews include long term usage feedback such as quality or product design. In this case the razor handle is mentioned frequently by the non-incentivized reviewers and is not mentioned at all in the top 10 by the incentivized reviewers.

Let’s look at all discussion topics of incentivized reviews vs. non-incentivized reviews. The following charts were taken directly from the Revuze Dashboard. The Revuze system identified 56 areas of interest (or aspects) by analyzing actual consumer reviews. The color of each aspect represents the overall sentiment for the aspect, with green representing positive sentiment, and red negative sentiment. This is an easy way to discern at a glance what consumers like and dislike about a product or brand. 

We can immediately see that the importance of the topics (their order from left to right) is different as well as the sentiment (color) for each. If we compare the most negative aspects mentioned in the non-incentivized reviews, we can identify aspects like durability, blade sharpness and materials that are mentioned negatively. 

These aspects seem to be related to long term evaluation of the product. Incentivized reviewers tend to evaluate the product very soon after they receive it, so they will be less likely to run into problems that only arise after long-term use. 

Conclusion: For Accurate Insights, Take Incentives Into Account

Comparing over 300,000 incentivized and non-incentivized reviews leads us to the following conclusions:

• Incentivized reviewers are more positive towards the product by an average star rating of an additional 3.5% and average sentiment that is 8.33% higher.
• Incentivized reviewers mention that they received this product as part of promotion
• Overall positive incentivized reviews do not tend to evaluate the price of the product. In the non-incentivized reviews price is one of the most mentioned topics, while in the incentivized positive reviews it is not even in the top 10 mentioned topics
• Incentivized reviews appear to be delivering reviews a short time after initial use of the product, so they focus on overall experience and less on long-term usage feedback
• Non-incentivized reviews include long term usage feedback such as quality or product design, which can be crucial feedback for consumer insights professionals

So, should incentivized reviews be ignored by consumer insights professionals looking for honest product feedback? Not necessarily. However, based on our data and analysis, we highly recommend that when analyzing consumer feedback, that you consider whether reviews are incentivized or not, as the two behave differently.