Market research is a core skill for product managers. Artifacts like product requirements, competitive landscapes, and positioning all require some type of research. Regrettably most product managers only leverage their intuition and secondary research sources like websites. Product managers should conduct more primary research to support their work with actual market facts instead of assumptions based solely on their personal experiences.
Market Research Landscape
There are three broad categories of product manager research techniques: Intuition, Primary Research, and Secondary Research. Product Manager Intuition is the combination of domain knowledge, skills, and experiences that a product manager brings to the table. Individual intuition is very valuable and is often the main reason why a product manager is hired. Unfortunately it is hard to justify a particular opinion aside from the statement ‘based on my experience’. Intuition can’t be independently tested to see if it is valid. Primary research is focused on gathering data from external sources, such as customers and prospects.
There are two main branches of primary research: quantitative and qualitative research. Quantitative research concentrates on gathering and analyzing large amounts of data via surveys or questionnaires. Analytics data such as user recordings, heat maps, and website analytics are often the most common quantitative analysis techniques product managers use. Qualitative data focuses on gathering subjective data like transcripts from in depth interviews, focus groups or video recordings. Qualitative research provides more insight into the research subject’s thinking or behavior, but typically lacks the statistical rigor that quantitative research offers.
History of Market Research
Market research can be traced back to the 1920s. In A Brief History of Market Research by Kuba Kierlanczyk he noted:
The first true instances of market research came about in the 1920s when a man by the name of Daniel Starch developed a theory that advertising had to be seen, read, believed, remembered, and most importantly, acted upon, in order to be considered effective..
A man named George Gallup, a contemporary of Starch, further developed the practice with his rival theory of aided recall, which prompted people interviewed to recall an ad seen in a publication without actually showing it to them. This system would later be adapted and used to measure the effectiveness of radio and television advertising.
It is interesting to note that the discipline of product management first emerged around the same time. Neil McElroy’s famous one page memo on the need for ‘brand managers’ at Procter & Gamble laid out the foundation for today’s technology product management.
Ernest Dichter pioneered a new form of consumer research in the late 1940s, called Motivational Research. Based on Freudian psychoanalytic concepts, Dichter believed that consumer’s held within their minds hidden realm of desires, taboos, repressions, and secrets. While interesting many users began to consider Motivational Research a bit too mystic and the pendulum swung toward more quantitative techniques.
Technological advances like telephone polling, computers, and the Internet transformed market research. Companies leveraged the scale these technologies could bring to research projects. Quantitative research became the dominant force in market research. In today’s market tools like HotJar or LuckyOrange allow product managers to inspect user behavior at unparalleled depth.
Why Do Product Managers Prefer Quantitative over Qualitative Market Research?
Product Managers have a strong preference for quantitative research. A recent study of over 1,700 product managers found that only 25% performed Win-Loss Analysis. Win-Loss Analysis uses structured interviews with customers and prospects to learn about opinions on messaging, positioning, value equations, pricing, and decision criteria.
Product managers prefer quantitative research techniques for a couple of reasons. First, quantitative research offers results that usually are statistically significant. You can use standard statistical analysis techniques to analyze data:
Secondly, quantitative market research are easier to execute. For example you can send a NPS survey after every customer service interaction to gauge customer satisfaction. Or you can use tools like HotJar to anonymously observe user interactions with your applications.
Qualitative research requires long form, direct interaction with the subject of the research. Recruiting customers, prospects, or even lost prospects/customers is hard. Product managers often use CRM or Sales Force Automation contact databases to solicit customers and prospects to participate in interviews. Email solicitation yields results like most email campaigns – a 20% open rate and a 2% to 5% conversion rate. It can take a lot of contacts to generate a pool of 20 to 40 interviews.
While Qualitative Research does not offer the ability to use statistical techniques like Student’s T-Test to determine the statistical significance or margin of error in a survey, there is a relevant branch of statistical analysis known as Grounded Theory. Grounded Theory is a systematic methodology in the social sciences involving the construction of theories through methodical gathering and analysis of data. One aspect of Grounded Theory is the concept of saturation. Saturation of data means that researchers reach a point in their analysis of data that sampling more data will not lead to more information related to their research questions. In conducting Win-Loss Analysis interviews, saturation is often achieved after 15 to 20 interviews. If 15 prospects say “I chose you competitor because their pricing was 40% less than yours” a product manager can reasonably infer that there is a problem with their pricing strategy.
How Can Product Managers Better Leverage Qualitative Research?
Qualitative research is one of the tools a product manager should have in their market research toolbox. Just as there is a role for quantitative research like Google Analytics, there is a role for qualitative research. Qualitative research, like in depth interviews, are best suited for answering questions that begin with ‘Why?’ Why does this positioning resonate so well with this market sub-segment? Why do users consider this feature set to be innovative while others consider it to be irrelevant? Why do customers not want to buy an upgrade that includes these 10 new features? These are the types of answers that you cannot get from an online survey or a website analytics tool. An interview-based research program is required.
An interview-based qualitative research program consists of give basic steps:
1. Planning
The first step of the process is to do all of the preparatory work required to ensure the success of the project. An important first step is setting research objectives. Setting clearly defined Research Objectives will help you to both target your win-loss research as well as set expectations for the success of the program. Setting research objectives is best done as a team project. And be sure to align your Research Objectives with the strategic goals of your organization. There is no sense chasing information from buyers that your organization has no interest in anyways. Research objectives may include:
- Your Product-Market-Price fit
- New market problems that your organization can solve
- Your service levels
- Persona refinement
- Buyer purchase-decision process
- Marketing channel effectiveness
- Sales process
- Communication style
- Better understanding of your place in the competitive mix
In addition to establishing your Research Objectives, you should also define the questions that will be asked during the interviews. The questions should be open ended and designed to encourage a free flowing discussion.
Finally, you should design the process you will use to recruit potential interviewees, conduct the interviews, analyze and report the results you have learned. The plan should layout the roles and responsibilities of each participant in the project. You should also develop a schedule and a simple status reporting schedule so everyone can stay up to date. Create a project charter that contains the research objectives, interview questions, roles, responsibilities, and schedule. Conduct a review meeting with the participants to approve and commit to the plan.
2. Recruitment
The next step in the process is to recruit people for the interviews. Most companies target a mix of net new customers and existing customers who have upgraded or expanded their use of products. You are looking for a mix of won deals and lost deals. Most projects with a single research objective try to get 15 to 20 interviews.
A portfolio approach solicit the interviews works best. The tactic that has the highest success rate is when a salesperson makes a personal outreach to a potential interviewee. Generally 25% to 40% of these contacts will convert to an interview. The second most common approach is email. Organizations pull a list of candidates from their CRM or sales force automation system and then email them asking for participation in the project. This approach performs like most email campaigns – a 20% open rate and 2% to 5% click thru rate. 50% of those who click thru convert to interviews. The final tactic is to do phone follow-up with contacts that opened the email but did not click thru. This tactics performs like telemarketing campaigns – a 1% to 2% success rate.
Most interview programs use incentives to encourage targets to commit to doing the interview. Companies offer a $25 or $50 gift card. This provides the interviewee with something tangible in exchange for their valuable time.
3. Interviews
The core activity of the project is conducting interviews. Most interviews take 20 to 30 minutes to complete. Companies use internet meeting services that facilitate the interview scheduling process and allow the discussion to be recorded. This enables the interviewer to focus on the discussion instead of trying to listen and take notes at the same time. There are services that will transcribe the recordings for you into a Word document for about $1 minute.
Organizations often outsource the entire process to third parties. Experience has shown that interviewees are more comfortable talking to an independent third party instead of a representative from the company. This results in a free and easy discussion. It also avoids making embarrassing or disparaging comments about their experiences or opinions.
A critical aspect of the interview process is asking why a customer believes certain things they say. Surveys and check lists are one way to get customer feedback, but they lack the ability to follow up on interesting statements. The real value of using experienced interviewers is that they can follow up and explore why a customer believes specific things. Often this is the most valuable outcome from the interviews.
4. Analysis
After the interviews are completed, the recordings are transcribed. Next the team reviews the transcripts to identify common themes. These themes are analyzed and documented into a final report. The report contains a summary of the interviewees – company size, interviewee title, etc. For each theme or finding, specific quotes from the interviews are included. This lets the report’s reviewers hear, from the customer’s perspective and in their voice, the exact point they were trying to make. A meeting is held with all of the interested internal organizations to review the report’s findings and conclusions.
5. Action
The final step is to take action on the recommendations. Effective interview programs are really part of a cycle to drive improvements in the business. Win-Loss Analysis is a variant of the Six Sigma DMAIC (Define, Measure, Analyze, Improve and Control) methodology. If action is not taken based on the recommendations from the interview analysis then an opportunity to fundamentally improve your business will be missed.
Product Managers Should Do More Qualitative Research
Product managers tend to shy away from doing qualitative research for a number of reasons – it is hard to recruit candidates, they may not have experience in conducting structured interviews, and they may prefer the anonymity that surveys or analytics can provide. There are some research topics, however, that do not lend themselves to Internet surveys or in-app analytics.
Product managers can overcome the greatest frustration with interview-based programs by incorporating ‘asks’ for study participation into their company’s standard business processes. For example the standard sales process should include a request to participate in a post-decision interview 60 days after a final decision is made. This can provide a pool of pre-qualified candidates that can be tapped for a formal study.
Want to know more? Here’s what we have to offer.
