Notes from an Interactive Discussion I led at ProductCamp Boston 2011

At ProductCamp Boston on April 2, 2011, I led an interactive discussion on “How are we in PM, UXD, Engineering gaining insight into our customers’ use of our products?”   I didn’t have slides.  The idea was to share, learn, and brainstorm together.  It was an interesting discussion, just the kind of thing I have come to love about Unconferences.  I wish we had videoed it.  Here are my notes from what the roughly 25 of us discussed. 

1) What questions about customers' use of your product are you trying to answer?  

Overall Value

  • What is the true value of my product?  What is its ROI?
  • Do my customers understand and agree with the value proposition
  • What is my users’ emotional view of my product?
  • What are their switching costs to leave us?

UXD type questions.  Patterns of Use

  • Who are the personas who use my product and what is their pattern of interaction with it?
  • What are their patterns of use?  Time of day? frequency, path through the application
  • What is their intended goal in doing this action?  What is their actual experience?
  • What is the rest of their world like?  
    • At work: What other apps do they use?
    • At work: What are their offline processes and how do they tie into our app
    • Outside work:  What expectations are set for them by their consumer apps


  • Are we building the right features?  The right amount?  Should we build more or less
  • In which features should I invest and in which should I stop investing?
  • What should I end of life?
  • What weird, unexpected uses do customers’ have for my product? 


  • What errors do uses encounter?  How frequently?  Through what usage pattern?
  • Where do they experience performance problems?
  • Where are users stuck?  Where do they go for help?
  • What do they jump through hoops about, not knowing we could make things easier or better?
  • What questions do my customers have about the product?

2) What techniques do you use to answer these questions?


  • They are important and useful and we all use them
  • But our discussion ended up focusing on their limitations

I use surveys but

  • I’m not sure how to handle survey fatigue
  • I can’t survey for everything?  How do I get insight on the smaller features?
  • How can I find out how customers’ use my product while avoiding sample bias?
  • How can I get a large enough sample size when I try to understand my customers’ use of my product?

Usability Testing

  • Make it an integrated part of your process.  ABT.... always be testing.
  • Share usability testing with the entire team, regularly
  • Steve Krug “Rocket Testing Made Easy”

Ways to do usability testing and get feedback

  • Use demo’s at a conference for 
  • Bring people in
  • Even better, go to them and immerse yourself at their site
  • Create a mock up or wireframe and get feedback on that before building

Leverage application usage analytics

  • Put up a public web demo and measure what people do
  • Put up a free version or freemium and analyze usage patterns and behavior
  • Put up a link to an unbuilt feature in the product and see how many people click on that... and what user segments they are in ... then build it.
  • A/B testing
  • Put up a search log and see what they search for... then build that.

Go where your customers are talking and listen

  • Discussion Forums, User groups
  • Twitter, Social Media

Learn from your competitors

  • Study their products
  • Do usability testing on the competitor’s product using
  • Watch your competitor’s forums

Ask others in your company

  • Connect deeply with customer support
    • Ask them what customers are telling them
    • Sit with them
    • Answer the phones
    • Look at bugs and enhancement requests
  • Ask Sales and SEs

Ask the right questions

  • If you could wave a magic wand, over my solution what would you have it do?
  • If you could wave a magic wand in your work setting, what would happen?
  • Net promoter.  How likely are you to recommend this to a colleague or friend? To whom, why?
  • If we gave this to you for free, how widely would you deploy it and when?

3) What are the biggest insights you have gained 

  • “I have gained a ton from lots of usability testing.  Many small changes resulted in big wins”
  • “I recommend getting customer feedback early, often, constantly.  Get going on it.  Don’t wait”
  • “I thought noone was running on this platform but surveyed and found a few were.  Tied this back to revenue and discovered it was some of our biggest customers.”
  • “I had a large 10 year old product with many modules and old thick client GUI.  Wanted to replace GUI with a web GUI but it was a 20 personyear effort to get all the features.  Surveyed customers and realized that we could give them what they actually use and make it much easier to use in 2 personyears.” 

4) What are the biggest mistakes you have made

  • “I built product for exactly 1 customer”
  • “I had what looked on paper like the perfect Beta.  Regular 30 minute phone check-ins each week.  Feedback on the product.  But I didn’t measure the customers’ actual use during the Beta.  They had barely used it.  They weren’t actually that motivated”
  • “Trying to convince the engineering teams and higher level managers of the insight gained from customer conversation and usability testing without having hard evidence to convince them.”
  • “Giving too much detail on an early product concept to the whole sales team and having them talk too broadly about it.”

I enjoy learning from my peers questions, techniques, successes, and failures.  Keep the discussion going by commenting below.

Karl Wirth is the CEO, co-Founder, and Product Manager for Apptegic.