April 2-3, 2019

Register Now Convince the Boss

Using Social Media Data to Predict New Product Success

Almost all of our clients have social listening programs. Those thinking “downstream” are already considering whether the social data is good enough to replace tracking data. Can it tell us, for instance, whether customers are satisfied with a product? Can it tell us whether we have a healthy brand? Can it tell us about consumer innovations in a way that can guide new product development? Inevitably it has fallen short. As Forrester encapsulates: “Listening to data gleaned from social media channels, ratings and reviews and forums has existed for a decade, yet most CI pros still cannot demonstrate the value of monitoring social data.”

Together with our partner Converseon, we have taken several steps to combine their expertise in semi-supervised machine learning for text analysis with our understanding of CPG product data to do two things: (1) Understand the requirements that must be set in semi-supervised learning to identify both the product attribute and the motivational conditions that the attribute satisfies; and (2) understand the statistical criteria for discovering a trend or level shift in either text-derived or sales data and, separately, the criteria for establishing a causal relationship (not merely correlational) between text-derived and sales data.

Using a case study in vegan food, we will demonstrate how we first extract attributes from the text chunks that are the descriptive label in syndicated data. We then use these attributes to create search filters for posts in a vast “foodie” corpus. We then refine the search procedures using the ability to detect emotion and motivation in the post. Finally, we use time series methods, including Granger causality tests, to demonstrate that the time series for an attribute derived from filtered social data can predict sales trends up to 90 days before a product attribute appears in market.

Presentation type:

  • Case Study Presentation

Subjects covered:

  • Predictive analytics


  1. High-quality, properly processed social data can predict inflection in sales trends up to 90 days before they happen.
  2. There is an important feedback loop between social and sales: in the beginning social predicts sales but after a while the sales activity itself contributes to the social trend.
  3. Special filters applied to the social data (e.g. need states, combined emotions) increase the likelihood of predicting sales.


More sessions

See all sessions

Making a Difference for the Business: Key Drivers of Success

The research organization exists to inform the business and to enable enhanced decision-making that will ultimately drive growth. At the same time, researchers often struggle with how best to make [...]


Picking Consumers’ Brains for Better Advertising Effectiveness

Bringing art and science together to produce positive ROI, drive sales and penetration uplifts and, of course, winning ads. *This session is provided by ESOMAR.


How IBM Incorporates ‘Win/Loss’ into Continuous Customer Insight

When IBM’s new CMO joined in late 2016, there was an increased emphasis on customer advocacy. One of the vehicles introduced to capture client feedback was the Win-Loss program. Pilot work [...]