Key Learnings from Sawtooth’s 2023 Analytics & Insights Summit
Credit: Unsplash
By Dan Yavorsky, Ph.D., Senior Vice President, Analytics
I was fortunate to recently attend Sawtooth's Analytics & Insights (A&I) Summit, where leading analytics professionals discussed new research and shared some of the latest methods and applications of consumer insights and marketing science.
This year’s conference did not disappoint! Below I share some of the key takeaways and themes from the event, which demonstrate how marketers can apply advanced analytics techniques to better measure consumer preferences and behavior, thereby improving business decision-making and increasing return on marketing investments.
The Evolution of Conjoint Analysis
Conjoint analysis, a core part of every marketing researcher’s toolkit, has been in use for over half a century. Literally, the first paper on the use of conjoint in marketing was Green and Rao, 1971.
In his book titled Applied Conjoint Analysis, Vithala Rao, a Professor Emeritus of Marketing and Quantitative Methods at Cornell University, says “Conjoint analysis is probably the most significant development in marketing research over the last 30 years or so.” Despite the technique being mature, the field of preference measurement via conjoint remains active, important, and growing.
Today companies across a wide range of industries use conjoint analysis to hone product design, product features, and pricing, thereby predicting and optimizing success within the market. Its popularity stems from how it can be effectively used to model real-world tradeoff decisions that customers make through a set of simple hypothetical choice tasks.
A recent innovation in the field has come from my colleague Professor Eric Bradlow, GBK Co-Founder, Professor of Marketing, Statistics and Data Science, Economics and Education and Vice Dean of Analytics at the Wharton School. Prof. Bradlow along with Wharton Marketing Professor John McCoy and Ph.D. Student Rachele Ciulli, recently proposed a new conjoint methodology. The approach allows brands to apply learnings from two (or more) similar product categories (like smartphones and tablet PCs) to more accurately predict customer behavior and inform decisions.
Chris Chapman, Ph.D., Principal UX Researcher at Amazon, and co-author of Quantitative User Experience Research, also shared his 'wish list' for the future development of conjoint analysis during GBK’s recent webinar. Dr. Chapman highlighted three areas:
Integration of experiential features, product designs or form factors into conjoint analysis. For instance, capturing how important the physical interaction with the product is to customers.
Improving segmentation with conjoint analysis. For example, clustering, machine learning, and other modern approaches with conjoint analysis have the potential to refine segmentation and preference estimation.
The development of a design and statistical model that takes into account distributed decision-making. This is particularly crucial in the B2B space, where purchasing decisions are not made by individuals but by teams.
Improving Consumer Preference Measurement by Enhancing Conjoint Analysis with Additional Data
Despite its power and wide use, there are several factors that can lead to inaccuracies in a conjoint study. These may stem from poor design, such as an incomplete set of attributes and levels, inadequate sample size preventing valuable subgroup analyses, result misinterpretation, cognitive biases, or interaction effects, among others. For example, a preference for simpler product configurations may reflect ease of comprehension rather than genuine preference, potentially skewing study outcomes. Even large companies with significant resources can experience these challenges. For instance, GBK Collective worked with Microsoft to help the company establish best practices to generate more effective and predictive results.
“I've always described conjoint as the end of the golden road of investigation, but it also marks the beginning of the next phase of exploration,” notes Prof. Bradlow. “Conjoint studies provide a starting point, not the entirety of consumer behavior or factors in their decision-making, often necessitating further qualitative research for hypothesis development. Furthermore, any conjoint study should be validated out-of-sample, so that forecasts and predictions from the output can be utilized with confidence.”
At the A&I Summit, several experts presented new research aimed at improving consumer preference measurement by supplementing conjoint analysis with data collected through survey exercises. A number of sessions also discussed overcoming common challenges or limitations.
“I've always described conjoint as the end of the golden road of investigation, but it also marks the beginning of the next phase of exploration. Conjoint studies provide a starting point, not the entirety of consumer behavior or factors in their decision-making.” - Professor Eric Bradlow
Streamlining Part-Worth Interpretation and Menu Pricing: Enhancing Conjoint Analysis with MaxDiff Exercise and Likert Scales
Dr. Greg Allenby, Professor of Marketing and Statistics at Ohio State University, and Miriam Liu, an Assistant Professor at Towson University, discussed their development of a single statistical model for evaluating consumer preferences that integrates data from a Conjoint exercise, a MaxDiff exercise over the attributes used in the Conjoint, and an additional set of Likert rating scale questions.
Dr. Allenby and Dr. Liu showed how their integrative model can be used to estimate the baseline level of product attributes, which enables across-attribute comparisons and is useful for assessing the drivers of demand and menu pricing. This enables product managers with direct paths to measure how much a new feature influences consumer preferences and to assess how to price a new feature. (Note: a manuscript of their work is forthcoming and will certainly be impactful.)
Improving Pricing Decisions for Big-Ticket Products by Incorporating Respondents' Budget Constraints
A common use of Conjoint in market research is to assess consumer price sensitivity to optimally price a new product. However, that approach can sometimes fall short in accurately estimating consumers’ willingness to pay, particularly for high-priced products where consumers may overestimate their budget or display less price sensitivity on the survey compared with the actual purchase decision.
Dr. Thomas Otter, Professor of Quantitative Marketing at Goethe University Frankfurt, presented his latest co-authored research on an innovative enhancement to the standard choice-base Conjoint model that enables more-accurate measurement of consumer price sensitivity by incorporating consumers' stated budget constraints. Prof. Otter demonstrated that accounting for budget constraints substantially improved the accuracy of competitive pricing and better predicted consumer behavior when compared to a standard conjoint analysis. I anticipate this method will become the de facto method for consumer preference measurement of high-priced durable goods, with rapid adoption across automotive, consumer electronics, and other industries.
“A common use of Conjoint is to assess consumer price sensitivity... However, that approach can sometimes fall short in accurately estimating consumers’ willingness to pay.”
Demonstrating Novel Applications of Conjoint Analysis
Conjoint analysis was originally developed by transportation researchers and has been successfully applied across disciplines ranging from agriculture to zoology. At the A&I Summit, several researchers demonstrated novel or clever uses of Conjoint Analysis, reminding me of the versatility of this method and the myriad problems it can address if properly applied.
Measuring Consumer Preferences over Time
Dr. Raphael Thomadsen, Professor of Marketing at Washington University, presented co-authored research measuring the importance of political agenda items over time and by respondent political ideology. Their study examined the impact of information about the U.S. Supreme Court decision regarding abortion laws (specifically, the decision in Dobbs v. Jackson Women’s Health Organization which overturned Roe v. Wade) on Americans' preferences for political candidates.
Prof. Thomadsen and his co-authors used conjoint surveys conducted at different time points in time to reveal substantial shifts in preferences toward political agenda items following the leaked draft and the official decision. This research highlights the dynamic nature of political preferences and the value of a methodological approach that assesses a time series of attribute importance scores to understanding voter behavior. I see many use cases of such analysis with brand-tracking studies that offer a time series of measured consumer preferences.
Assess Non-Discrete Preferences from Discrete-Choice Experiments
In a separate study, Dr. Mitch Lovett and Dr. Gretchen Helmke, professors at the University of Rochester, conducted a conjoint survey experiment to analyze split-ticket voting behavior. While their focus was on political candidates, marketers can draw insights from the research for studying consumer switching behavior. The findings of the study underscored the importance of considering the “discreteness” assumption in the discrete choice model underlying a standard conjoint analysis. Voters in Professors Lovett’s and Helmke’s study did not make all-Republican or all-Democrat choices on their voting tickets, and we may see consumers make similar “split-brand” purchase decisions.
Improving Preference Measurement by Improving Respondent Enjoyment of the Choice Tasks
Another standout session was presented by Dr. Jeff Dotson, Professor of Marketing at Brigham Young University. Inspired by Tinder, Prof. Dotson and his co-authors developed a novel approach to simplify choice-based conjoint questionnaires on mobile devices where respondents evaluate only one alternative at a time, swiping right to choose and left to not choose.
Early research suggests the approach of using choice sets of size one provides sufficient data to estimate a model of consumer preferences and that respondents found the “gamified” task easier and more enjoyable and remain engaged longer than with traditional multi-profile conjoint choice tasks. This approach may offer marketers a streamlined and (dare I say) fun way to gather valuable insights while maintaining data quality and reducing respondent burden.
“Inspired by Tinder, Prof. Dotson and his co-authors developed a novel approach to simplify choice-based conjoint questionnaires on mobile devices where respondents evaluate only one alternative at a time.”
Simulations and Practical Guidance for Marketing and Analytics Leaders
A few other standout sessions from the A&I Summit involved simulation studies to provide practical guidance or rules of thumb for marketing researchers involved in segmentation, conjoint design, Bayesian estimation, or optimization of post-conjoint market simulations.
Finding Contrastive Market Segments with Archetypal Analysis
Traditional market segmentation relies on cluster analysis techniques to identify homogeneous groups. However, this approach may overlook valuable insights hidden in segment differences. Jacob Nelson, Senior Data Scientist with Harris Poll, presented an alternative method called archetypal analysis. Rather than identifying groups with shared characteristics, this method focuses on differences among segments, offering novel insights. For instance, a financial services company might unearth distinctive behaviors of niche investors, informing more targeted service offerings.
Anticipating the Impact of Multiple Cluster Structures on Variable Selection in Segmentation
Joseph White, Senior Director of Marketing & Data Sciences at Kynetec, presented on the influence of multiple cluster structures on variable selection in segmentation. Joseph demonstrated the conditions and degree to which certain variable selection-based segmentation techniques succeed in recovering a known segmentation structure. His findings provide guidance on when to expect an unsupervised clustering algorithm to succeed in identifying consumer segments.
Use of Utility-balanced Designs for Improving Consumer Preference Measurement with Conjoint
Megan Peitz, Founder of Numerious, contrasted the effectiveness of various choice task design strategies to enable conjoint analysis to recover individual heterogeneity. While most designs used in choice research are tailored for aggregate insights by ensuring one-way and two-way level balance in the design, Megan presented research on utility-balanced designs and demonstrated how they can improve estimation of individual-level preferences in choice-based conjoint analysis. This can make a significant impact for brands, improving decision-making and enabling them to tailor offerings more effectively.
Ensuring Accurate Preference Measurement Via Sufficient MCMC Sampling (Guidelines for hierarchical Bayesian multinomial logit estimation)
Peter Kurz and Maximilian Rausch with BMS market research & strategy presented guidelines for the number of burn-in and retained draws when estimating a hierarchical Bayesian multinomial logit model with a Markov Chain Monte Carlo (MCMC) routine. MCMC is a method for estimating the posterior distribution of parameters in a Bayesian statistical model; it’s the Bayesian analog to finding estimates and confidence intervals via traditional non-Bayesian statistical methods.
Kurz and Rausch assessed convergence and goodness-of-fit metrics across simulations that varied the number of choice tasks, the number of parameters to be estimated, and the degree of heterogeneity in the data. They provided clear guidelines for practitioners on the influence of each tested feature for the minimum number of required MCMC draws. By carefully following these parameters, a digital retailer, for instance, could enhance the precision of their predictive models, refining their product assortment and personalization algorithms.
Algorithm choice for optimizing product or product-line offering via post-conjoint market simulations
Dr. Daniel Baier, Chair of Innovation and Marketing at the University of Bayreuth, delivered a dense but insightful presentation of co-authored research comparing optimization algorithms for post-conjoint market simulations. Their research aimed to identify the most effective algorithms for optimizing product-line decisions based on the insights derived from conjoint analysis. Through rigorous analysis and simulations, they evaluated a seemingly exhaustive list of approaches and highlighted their strengths and limitations in the context of post-conjoint market simulations.
The conference wrapped with Sawtooth Software CEO Bryan Orme's touching tribute to Rich Johnson, a distinguished marketing researcher and co-founder of Sawtooth.
I thank Bryan, the Sawtooth team, and the presenters and attendees for a fantastic event. It validated many of the best practices we utilize at GBK and provided insight into cutting-edge marketing research developments that we’re excited to adopt. See you at the next one!