Internally, Microsoft runs every aspect of the company using data. We refer to this as our “Data Culture”. As the leader of business intelligence and analytics for James Phillips, the Corporate Vice President who leads the Business Applications Group, I am often asked why and how we manage user feedback. With this post, I would like to share our practices around collecting and using data to accelerate development of business applications like Power BI, Power Apps, Microsoft Flow and the range of Dynamics 365 products.
The primary metrics for Microsoft products in the Business Applications Group are the MAU (Monthly Active Users) and the NPS (Net Promoter Score). These metrics drive our organizational behavior to ensure that people use our applications and that they are satisfied. We view this combination of metrics as a leading indicator of product success and our progress on the mission of empowering every individual to achieve more.
What is the Net Promoter Score?
NPS is an industry wide customer loyalty metric and an approach to manage user perceived product quality. At Microsoft Business Application Group, we determine NPS by asking two questions: “How likely are you to recommend <product name> to a friend or colleague?”, followed up by the free-form question, “Why?”.
Users answer the first question on a scale of 0-10 pts. Those who answer 9 or 10 are called promoters, 7 or 8 answers are the passives, and those scoring between 0 and 6 are called detractors. Then the Net Promoter Score is calculated by subtracting the share of detractors from the share of promoters in your population, and can reach values from -100 to +100. The above calculation makes it a very sensitive and useful metric, since shifting user attitudes are immediately visible through a changing score.
Even with that number, we still get the most useful insights from analyzing the comments provided by users as they explain why they would (or wouldn’t) recommend a product. These comments provide valuable direction for Product Managers on what they need to prioritize to ensure the success of their users.
Every user has a right to share their feedback
In most of our modern SaaS products we try to capture NPS effortlessly, with a dialogue shown to end users. Below is an example of this survey:
We have put a lot of effort into ensuring this dialogue is as optimized as possible. Users have the options of providing numeric feedback in two clicks, sharing a comprehensive opinion if desired, or simply dismissing the dialogue. We schedule such survey to appear not more often than every 90 days, and only to users that have a history of meaningful activity. Thanks to all these considerations, more than 25% of users who see the dialogue provide us with useful feedback. This is a large share, particularly when compared to the low single digit response rates to the email surveys we used previously.
Even though it is rare for any particular user to see an NPS prompt, we capture a lot of useful feedback this way due to our large user base. All feedback is read daily by the respective Product Managers, and it influences decisions on the agile development of our products. Since many of the Microsoft’s business application products are now releasing monthly or even weekly updates, the opinions of users can quickly turn into new or improved features, or optimized performance.
One of Microsoft’s core values as a company is customer obsession, and this influences even how we process anonymous feedback. We intentionally sort through all feedback manually to stay close to our users, instead of automating it with data science.
The user is our ultimate boss
It’s common wisdom in business that “the user is the boss”, but for large products it is often difficult to establish an efficient connection from that “boss” to the product decision-makers. In the Business Applications Group, we ensure that all product leaders are motivated to increase NPS, as it is one of the two top KPIs they are held responsible for, as earlier explained in the post on our executive dashboards. Metrics have the power to orient behaviors of organizations, and hence every product team is motivated to read and immediately act on user comments passed via the NPS feedback dialogue. We have seen this on-going feedback process drive rapid development and product strategy choices, and balance investments between product performance and new features. Our users have, in turn, rewarded us with growing NPS scores.
One of my wise managers taught me that no matter how positive or negative it is, “feedback is a gift” that you should be thankful for. So, I would also like to thank all of you — our users — for sharing your feedback with us and helping us continuously improve Microsoft’s business applications.
Slawomir leads business intelligence, data science and advanced analytics for the Microsoft Business Applications Engineering group. Until 2015 he served as a Director of Consumer Intelligence at Microsoft after 11 years with Procter and Gamble, where among other assignments he was a CIO in Central Europe, spearheaded use of Business Analytics and led all digital marketing and CRM operations in 100+ countries of EMEA. He enjoys speaking and writing about enterprise data transformation, data science and NPS.