Top 10 mistakes when using conjoint analysis
Conjoint analysis counts as one of the more sophisticated, and powerful, techniques of market research. It's aim is to estimate what drives customer value, and to say what customers would buy when faced with different products at different prices. But it gets the power and insight due to careful design and appropriate analysis. Inexperienced users can create poor designs without realising it, and research users can have poor experiences with conjoint analysis because the agency or consultants do not really understand what is involved.
1. Poorly defined attributes and levels
The primary problem with conjoint analysis is poorly defined attributes and levels. An attribute is the feature or category, and the level describes the performance. So an attribute might be 'Weight' and levels might be 50g, 75g, 100g for instance.
The aim of conjoint analysis is to simulate a real purchase choice. If level definitions are unclear or vague for a customer or buyer it is very difficult to make a purchase-like decision. This is particularly true for those from a survey-design background who might use 'very good' or 'good' in their definitions, making them extremely ambiguous. What is a 'good' washing machine and how does it differ from a 'very good' one?
The best rule-of-thumb for each and every level is that the words or phrase should be exactly what you would put on the packet to describe the product. You should be able to take the level texts and put them directly on packaging or an advert. That means short, clear and distinct and in words that grab the interest of the buyer. This obviously takes some work to get the descriptions right. But when presented as choices, the products need to look and feel like realistic product options. If not, you're not really investigating a realistic choice.
2. Overlapping attributes
Attribute and level definition is really fundamental to good conjoint design. A common secondary problem is that the attributes themselves overlap - technically they aren't fully independent. Brand-names create halo effects that can touch on other attributes - Apple 'User friendly' might be perceived differently to Huawei 'User friendly' for instance, or Apple at 'low price' may simply seem unrealistic. Keeping attributes clearly separate requires clear wording and care in the design.
A secondary problem is that for new products, often features are still being fleshed out. For product managers, the process of creating attributes and levels can be part of the strategic thinking of how to bundle or separate core features. For instance, does speed mean throughput or initial start up time? What does it mean for a delivery to be on time - would the right day do, or is it to the hour?
3. Long feature lists and doing too much
Software and technology products often have long lists of features and options that can be combined in multiple ways. In conjoint analysis each feature becomes an attribute, rather than a level, with 'Yes' and 'No' as the levels. This can end up creating very large numbers of attributes, and make the choice tasks very complex for respondents to pick through.
Similarly, where the number of attributes is very large, the choices can become overwhelming because there is such a lot going on. Techniques like partial profile designs can help, as can improvements to layout and look - in real life we look at a lot of features, but keeping a tight attribute set is still the best option.
In these situations, doing a separate project ahead of the conjoint analysis, using things like MaxDiff and then grouping or bundling features into the conjoint exercise may be more appropriate.
4. Missing attributes or not enough blue sky
If doing too much is a problem, so is not doing enough. To simulate market conditions, the products on offer should reflect the range of products available, now and into the near future. If a feature or level isn't included in the conjoint design, it can't be included in later market modelling or estimations of market share.
Similarly, if a key feature is missing, the end result will show relative preferences - which genuinely can be useful - but might not reflect real purchasing intent.
5. Unrealistic price ranges
Understanding price and willingness to pay is a common use of conjoint analysis. For products with similar price levels, or where the main question is about brand, format and price (eg shop-shelf displays), price is relatively easy to handle, so long as it looks realistic, and isn't overly dominated by low prices.
Where it gets complicated is where features or attributes imply different price buckets. For instance adding storage memory to a device often has an implicit naturally higher price - eg 64 is less than 128 is less than 256. This adds complexity to the prices that need to be shown in order to keep them realistic. Otherwise the conjoint can end up dominated by high features and low prices creating 'no-brainer' type choices.
6. Lack of realism
A common complaint that we would make about conjoint designs is that they often lack realism and they look boring. The aim is to attempt to simulate real purchase decisions. If someone is buying insurance, then the conjoint design should reflect the style and design of a typical insurance purchase. Similarly, if products are supposed to reflect e-commerce choices, or choices from a supermarket shelf, then keeping the context helps with the realism.
Conjoint designers should take the time to make the layout and look-and-feel mimic realistic scenarios as much as they are able, including working on UX and CSS to match realistic styling.
7. Not using an appropriate statistical design
Underpinning the analysis behind a conjoint analysis design, is the need for an appropriate statistical design plan - the set of choices and attributes and levels that are shown are not random, but rely on an appropriate experimental design plan to ensure the maximum data comes from the smallest number of choice tasks.
Fortunately, much conjoint analysis is carried out using off-the-shelf specialist software which makes it easy to create good quality statistical designs. However, for those looking to cut corners, they may attempt a DIY approach putting options up at random, or attempting a 'full-factorial' (ie every combination) approach.
Statistical designs have criteria like 'balance' and 'orthogonality' and should limit the appearance of dominant combinations (eg a high-feature-low-price option that would clearly be chosen rationally). If the design is wrong or does not work, it is almost impossible to rectify in analysis. Designers should be checking the design for statistical issues before it goes out into field.
8. Not understanding analysis or using the model
Conjoint analysis can use a number of different forms of analysis, which are fundamentally related to the way the choice tasks are designed. In off-the-shelf packages analysis can be as simple as a touch of a button. However, non-conjoint agencies often struggle to understand the outputs from conjoint analysis and how to interpret them.
For instance, a common mistake is just to show utility scores, and possibly important scores as if these are the final output. Utility scores do provide powerful and useful information about underlying preferences. However, the real strength of conjoint analysis is in modelling, where different products with different feature combinations can be tested against each other too look at which generates most market share, and what level of potential revenue would accrue. Add in costs and you have a profit model.
9. Issues with sample
Conjoint analysis is a research technique that relies on statistics and sampling - you need enough responses to be able to generate estimates of what drives choice. And the amount of sample available affects the number of choice-tasks and attributes you can include (in choice-based conjoint, the choices are 'spread' across the available sample and analysed as a group).
In some situations, particularly in B2B markets or healthcare markets, sample is difficult to get hold of. Realistically, this should mean keeping the conjoint design simpler, or using more adaptive type approaches to gather more information from each individual.
The second issue with sample, is that if you only use customer lists, then you will underestimate the choices and decision making behaviour of those buying from competitors. The data can still be useful, but care needs to be made about extrapolating findings.
10. Not building price curves
For any conjoint analysis that includes price, a fundamental output should involve building demand curves and the estimation of revenue potential with a view to identifying maximising points. If costs can be allocated, this can also drive profit curves and estimates. Understanding price sensitivity helps identify which features drive price and where customers most seek out value.
