Balancing Qualitative and Quantitative Insights

Getting information from your customers/users is critical to quickly test and improve your product. However, this is often easier said than done. In the early stages of a product it can be hard to determine what impact (if any) your features are having. Further, if sufficient time is not put into thinking about what will determine success before product development begins, there is a big risk of spending too many resources on the wrong things. This is where a disciplined approach to collecting insights helps. Insights are something that gives an accurate and deeper understanding of how your product can better deliver value.

To start, you can collect data from your application. Quantitative data is comforting because, if done well, it can provide a strong foundation to understand your product and how users behave. As they say, “numbers don’t lie.” In most small companies though there often isn’t enough data to attribute strong statistical significance to anything. Margins or error are enormous and collecting enough data can take weeks. Further, the big concern about trying to find trends in data is that you have a high likelihood of coming to a conclusion that isn’t connected to the real world. Errors in data analysis are some of the things that keep all product people up at night. Numbers don’t lie but we can certainly lie to ourselves as to what they mean.

The flip side of collecting insights is qualitative data. This is what your customers/potential customers are actually saying about your product or service. Talking to customers is a great way to get rich, direct feedback into what features customers find or would find valuable. Further, qualitative data collection has more flexibility for serendipitous learning. There is a high likelihood that your customer is going to tell you something you never even considered when talking with them. However, the downside of qualitative feedback is that it is susceptible to bias. You might be asking leading questions which give you misleading information. Talking to customers is time intensive and each conversation may be divergent enough that you will not feel confident to make a conclusion after dozens of conversations.

Here is a quick list of things I have learned about quantitative and qualitative feedback: 

  1. Track things in your application and know what questions you want to answer before you ship a new feature. 
  2. Learn how to conduct user interviews that remove your own biases and conclusions from the questions. Read The Mom Test
  3. Empower other stakeholders in the organization to collect and share feedback with you. Sales and customer support feedback is crucial to take into consideration with other data.
  4. Find people who can challenge your conclusions and keep you from going on a fishing expedition to find data and evidence to support your assumptions.
  5. Do the best you can with what you got. You may not have 3000 NPS responses every month but 40-50 starts to paint a picture. 6 user interviews that are all pointing in the same direction is sometimes all you need to prioritize work.

Build in collecting and analyzing both qualitative and quantitative feedback in your process. Realize that you may make the wrong call from looking at your insights. However, that is why it is crucial to collaborate early and often when analyzing product insights and never assume your conclusions are static. The simple fact is that for as long time you will never have enough data to feel like you are truly making progress. What matters though is that you are doing the best you can with what you have at your disposal.