Market Intelligence describes the set of activities that provide a company with a view of a market using existing sources of information to understand what is happening in a market place, what the issues are, what competitors are doing, what customers or consumers are doing (eg social media) and what the likely market potential is for new products or services based on previous activities and responses.
Broadly speaking Market Intelligence can be divided into two areas depending on the source of the data:
- Market Intelligence based on external data
- Social media monitoring
- Market Intelligence based on internal data
These last two categories are increasingly being redefined as Big Data. However, Market Intelligence is also used to refer to the collection and monitoring of external data such as analysts reports, competitor financial data, press monitoring or social media monitoring. Increasing attention is on Market Intelligence from internal information that provides an insight into markets and customer behaviour, from sources such as databases, prospect lists, website activities, transaction histories, loyalty cards and so on. See some examples of our work.
Market intelligence from external data is normally gathered by looking at secondary information sources, usually through desk research or carried out through a continuous or semi-continuous monitoring process. Often this means sourcing and analysing published information to build a picture of a market and to try and answer some specific commercial questions such as what is the market potential, what are competitors future plans likely to be, what prices might customers be willing to pay, what's the best means of entering a market.
Central to successful desk research is the ability to track down sources of information and to provide a skilled analysis to read the data and identify not just the data, but the story behind the data. For example identifying who your competitors are and analysing their market position against yours to find strengths and weaknesses and indications of new developments, or identifying potential channel partners or locations to set up new offices. Market Intelligence specialists usually have a nose for what they don't see and what is missing in the the data that they find. These can often translate into hidden market gaps and opportunities. They also have a sharp sense of how to use partial data such as financial reports, in order to make assessments about general performance - for example whether revenue growth is being led by prices or by sales growth.
To help internal market intelligence or market analysts, many companies use external resources like analyst houses who provide not just data, but also a commentary and advice on the current market picture based on their contact with suppliers and other companies. Many of the analyst houses act as brokers between suppliers, collating information and sharing it more widely. In some industries the analyst house can have very specific and large sets of data such as retail audits.
A specific form of Market Intelligence is competitive intelligence. This is typically undertaken on an on-going basis and involves the collection of news, materials and other information about competitors from a wide variety of sources. This may involve collecting information about market positioning and market messages, core clients or contracts, size and structure of the business and issues like pricing or typical deal structures. Examples might include collecting price-check information, or details of promotional and advertising campaigns, or monitoring news channels for information about new products or new technologies (eg patents). Although competitor intelligence can be carried out as a one off project, in reality, because of its on-going nature, competitive intelligence is often more about putting structures in place to enable information about competitor behaviour to be fed-back and monitored, than specifically finding one-off pieces of data. One key point is that for legal and ethical reasons, competitor research should not be carried out in any underhand way (eg misrepresentation) and so should rely only on openly available information sources.
Competitive intelligence can also use primary sources of data, such as feedback from sales teams, suppliers or distribution channels or feedback based on direct win-loss research that are often used to track bid performance.
Increasingly, Market Intelligence can involve collecting data from posts, tweets and other social media. This type of 'market intelligence' overlaps with some forms of market research and with PR monitoring. For some companies, the volume of comment (Big Data) together with the need to manage and monitor across multiple languages and multiple domains mean that large scale software is used to capture and then text-analyse the data to produce what is called sentiment analysis to gauge the general tone of the online commentary. Social media can provide companies with good insights into the mood of a market about, for instance, a new product that has been launched and it can provide very specific detail and suggestions for product or service improvements. However, much of this monitoring is also about reputation monitoring as part of a communication or PR campaign to allow companies to be alert to negative comment or problems that are publicised via the social networks.
From a data point of view, obviously there are potentially very large sources of data and IT tools for social media monitoring need to be large scale in order to scrape (ie capture) data from a wide range of sites in a similar way to a search engine robot, and then mine this data with text analytics to find useful information. The scraping of data also brings in some ethical issues as some conversations may be deemed to be private particularly in juristictions like Europe, where companies are only allowed to collect information to a legal limit. Obviously this differs from the US where all information in the public sphere can be freely used by businesses.
There is also some caution necessary over how representative views are. Not all sites can be scraped (some like Facebook do not allow social media listening) and those that can be scraped might not be representative and commentators themselves tend to be more active and enthusiastic (positively and negatively) and may not properly represent the views of the whole market. The structure of discussions online also needs to be taken into consideration. For instance more opinions are voiced where there is disagreement than where there is agreement. Although online conversations might not be representative it is quite possible that they include or influence important opinion leaders. For this reason it is advisable to cross-check and validate how sentiments online translate into more representative forms of market research.
While much marketing intelligence is associated with collecting information externally, a great deal of insight can come from making better use of existing information such as customer databases, web-analytics and test-marketing - an area that is increasing being known as 'Big Data' analysis. For instance by carrying out database analysis on orders taken it may be possible to understand where you have cross-sale and up-sale opportunities, or to understand what type of customers are your most profitable. Common database analysis include tracking recency, frequency and value of purchases. Looking for pareto segments. Augmenting database lists with external data to identify purchasing patterns.
Database information is not the only source of market data. Your website may also include a high degree of valuable information about who is looking for your products and services. Web site traffic analysis can help you understand what customers are looking for and why, and can be used in conjunction with test advertising and variations in site and page delivery (eg varying landing pages) to provide direct measurement and enhancement of marketing effectiveness.
One challenge is the increasing volume of data that is potentially available (Big Data). Large sources of data have the potential for showing hidden patterns and correlations within the data. Finding these patterns from a large dataset is a challenging statistical task both to handle the data size and to be able to extract meaning from the potentially infinite number of cross-relationships, without getting lost among false-correlations.
At the moment there is great hope for statistical analysis of Big Data, but it may be more effective to use the flow of information as a means of carrying out more experiments. That is testing something in the market and then seeing how the market reacts. Experiments online can be carried out small scale and very rapidly to test for improvements and refinements - eg A-B testing to see which advertising message works best, or experimental designs to look at how landing page combinations deliver sales conversions. These experimental approaches also translate directly into algorithmic marketing. That is, automated rule generation that adjusts marketing communications and messages based on observed responses in a similar way to Amazon's recommendation list, or Google's personalised search.
Finally, don't overlook knowledge about customers, markets and competitors that comes from your staff. Often this is a poorly tapped source of information. Collecting and disseminating such information falls into the realms of customer knowledge management and making better use of this customer knowledge can help businesses focus far more on what the customer wants and says.
For help and advice on carrying out any market intelligence projects on-line or off-line contact firstname.lastname@example.org