Published on March 5, 2021

Executive Summary

Product analytics is an integrated set of data about your users’ behaviors that allows you to analyze these behaviors both at the level of the individual user and in aggregate to gain insights about larger patterns and trends.

The use of product analytics in MedTech is still fairly nascent but holds tremendous promise to help turbocharge an evolution towards the frequent release of medical devices. In the context of Software as a Medical Device (SaMD) and other types of connected medical devices, product analytics add value in 3 different areas:

  • Device design and development
  • Device operations and end-user support
  • Helping users better use their device and improve their health outcomes

Introduction

In our prior blogs, we made the case for why frequent releases can help improve the quality and user experience of medical device software.  As with many areas of innovation and quality improvement, the better the data that’s available to the product design and development team, the more the team can accelerate the product development process resulting in a better product.

After writing about reducing surprises during HF Validation tests and tools to implement frequent releases for SAMD, we looked at techniques to amp up frequent release cycles and then had experts weigh in on challenges to frequent releases in design.

In this blog, we’ll dive into a class of data that is heavily used in other industries to great success; product analytics.  The use of product analytics in MedTech is still fairly nascent but holds tremendous promise to help turbocharge an evolution towards the frequent release of medical devices.

What is Product Analytics? 

Product analytics is an integrated set of data about your users’ behaviors that allows you to analyze these behaviors both at the level of the individual user and in aggregate to gain insights about larger patterns and trends.

The value of product analytics is broadly recognized and captured in a number of business domains such as subscription services and e-commerce, and is key to keeping company apps and solutions continually relevant, always improving, and to use a slang expression: addictive.

In the case of connected medical devices, product analytics starts with “instrumenting” the graphical user interfaces of your connected medical device (e..g, smartphone app, web browser, on-device user interface) to gain real-time insights into user interactions with your application.  In other words, you add invisible triggers in specific places around your application that produce detailed logs of specific user behaviors.  In addition, you capture properties of those users such as browser type and country. Generally, firms execute this with a third-party software solution such as Mixpanel. 

This is an anonymized example of one row of product analytics data created by a user’s visit to Orthogonal’s website.  As you’ll see below, they arrived at our website via Google, and this is the fourth page they visited on a specific day. The record details their journey through our website, what kind of device they were using, and where they are located.

Often, product analytics data captured through these invisible tripwires in your application can be more valuable if you link data from other systems where you already have more detailed or complementary person-level data about those same users. When combined, this data becomes very rich. This linked, person-level data from multiple sources becomes even more valuable when some of those data points are used to segment your users by demographics, health data, or the type of smartphone used.

Product analytics as a building block for the rapid product improvement of connected medical devices

In the medical device space, the application of product analytics is still in its infancy. However, to get to frequent releases in the design and development of Software as a Medical Device (SaMD) and other connected medical devices, product analytics will need to become a standard tool in the toolboxes of R&D, software engineering, human factors, and user experience (UX) professionals.

Product analytics helps answer key questions that would be difficult or impossible to otherwise answer such as:

  • How many account sign-ups were successful of those attempted?
  • Is a new feature being used?
  • If some users are unable to see a particular screen, who is this happening to and why?
  • How does product usage vary by country or demographics?
  • What paths do users most frequently take in the software?
  • Where do users abandon the software most?
  • How often do users view key indicators for health decisions?
  • How often do users view alerts in the software?
  • How and how often are help and instruction features used?

To be clear, we see product analytics as a compliment, but not a complete replacement for other types of qualitative and quantitative methods that we deploy for research, design, and validation. 

We have been working with more rudimentary versions of product analytics products going back 12 years.  As early as 2009, we saw the potential value of the product analytics concept and have worked with it intermittently, even if the early products were not very effective in their implementations.  We are very methodical and technique-driven in our work and are constantly identifying new or alternative approaches; we work off of fundamental principles of good design that allow us to borrow, adapt, combine, fuse, and generally re-mix methods based on experience and our understanding of the research challenge.  Product analytics is a powerful tool in the toolbox that provides broad and deep data about real-world product use.

Product analytics can work together with other methods, especially qualitative methods, by uncovering insights into how users are navigating through the product. This information can then be used to design traditional human factors tests, which strive to further uncover the “why” behind what users are actually doing. 

How product analytics fits into the medical device lifecycle

The data made available with product analytics has value across the complete medical device product lifecycle.  At different stages of the life cycle, the data can be used in different ways (i.e., different use cases) to create (novel) value.  At a high level, we divide these use cases into three categories.  Product analytics can be used to help with 1) device design and development, 2) device operations and end user support, and 3) helping users better use their device and improve their health outcomes.

A few quick things to keep in mind before reading this list of examples.  

First, product analytics’ value is in helping you track what is actually happening between your user and their device.  Once you identify an interesting what (i.e. pattern) in this data, you still need to follow up with qualitative research to understand why that pattern is emerging in the data.  You’ll need to determine the significance of the pattern and then decide what, if anything, you want to do in response to that insight.

Second, all use of product analytics must be thoughtfully integrated into the same mechanisms you use to ensure the effectiveness of your device and protect the safety and privacy of the user/patient.  This is where fast feedback loop techniques such as Three User Thursdays become a very effective complement to product analytics.

Use Case #1) Device design and development

Anytime that a person (i.e., user) is interacting with a connected medical device, the data that provides context for those interactions can be captured as the raw data for product analytics.  There are numerous points in the device design and development lifecycle where this kind of data is being created.

To make an obvious point, the greater the number of users of the device, the more that data can be generated to create a richer input for insights.  So while product analytics could be used as part of an initial feasibility study for a new device, the study’s relatively small sample size will enrich the data available to the product team for those study participants, but expectations about the value of the data should be tempered.  This is why R&D teams instead rely on simulated use activities such as usability studies, which is an imperfect but very useful way to try to predict real-world usage of a device.

The following are all examples of ways that this type of analytics can be used to improve and accelerate the early design and development and subsequent enhancement of medical devices.

  • During early product feasibility studies, this data can provide real time insights into if and how the study participants actually interact with the prototype of the device.  This type of automated data collection can be far more powerful and accurate than older methods to track usage such as diary logs.
  • Later in the product development lifecycle, this data can be used as an additional data source to ensure the consistent and proper execution of verification and validation protocols.
  • During clinical trials, product analytics provides real-time insights that are an excellent complement to other data end points used to evaluate the device in a controlled setting.

But the greatest value of product analytics comes to bear once a connected medical device is approved and begins to be used as a “real” medical device.  At that point, the data collection hooks that have been built into the device provide a tremendous tool for understanding how the device is actually being used (i.e. real-world data).  This is a fundamentally different (and better) calibur of data than the “artificial” data that was generated pre-launch. For example, product analytics can tell you how often and where a user is activating contextual help, which can identify gaps in design or labeling usability.   This data can then be used to guide the future support and evolution of the product (i.e., real-world evidence).

In addition to better understanding actual usage of a medical device, product analytics can also be used to support the rapid testing of new versions of the product where the control group (i.e., Group A) continues to use the current device and the test group (Group B) uses a modified version of the product (either in development or as a new feature release to a limited group of users).  With this data, their experiences and outcomes can then be compared to evaluate the new product idea on a regular basis, and the product development team can quickly react to these learnings and generate a new “B” test to quickly deploy and evaluate.

Obviously, A/B testing and product iteration for a connected medical device will never happen at the same speed as other types of connected solutions such as social media tools. Any change you make to a medical device needs to be evaluated in terms of patient risk. Many changes do not involve high patient risk and you can use things like well-architected software segregation (a. la. IEC 62304) to guide when you might need to involve an IRB or do additional regulatory filings.

For low-risk changes, you have more flexibility in terms of rapidly performing A/B tests. Similarly, you can use “canary” tests where you do a phased rollout of a new software change to a subset of the entire user base and see how well it works in production. Each phase of the rollout then becomes a separate cohort to measure in the test. The FDA Pre-Cert model also recommends this approach, although for somewhat different reasons.

Product analytics also makes it easier to do more sophisticated segmentation of a devices’ user base in order to do more targeted analysis of current device usage and A/B testing of potential changes in the product.  

Once a new version of a product is ready for a clinical trial, the built-in product analytics can enable a more ambitious and continuous clinical trial operation that speeds the time to approval of a device for new features and also for the use of the device for new indications or features. While continuous and adaptive clinical trials are still emerging tools for primary data collection, but we believe that product analysis has tremendous potential value to amplify the benefits of these newer clinical research methods in terms of speed to demonstrating outcomes, and facilitating better outcomes.

Put together, this all means that you can launch a good medical device quickly and then use product analytics to enable you to iteratively and rapidly evolve your product from good to even better to great.

Use Case #2) Device Operations and End User Support

Anyone who has ever made a call to a technical support line for product assistance understands that it can be quite difficult for a remote technical support team to efficiently and effectively help you diagnose and resolve the issue you are reaching out to resolve.  This is, in large part, because they can’t be physically with you and the device to observe first hand what is going on.  So real-time product analytics data post-launch can be especially useful for an operations team that’s supporting remote users by giving them real-time visibility into how exactly a user is interacting with a device.  (Think of the difference between describing to someone what you are seeing in an application on our laptop and having them be able to log into your laptop, see what you are doing and remotely take control of your laptop to diagnose and fix an issue.)

Since all user data is stored in a single database, it also becomes much easier to develop a broader situational awareness of what is happening across all of the devices currently in use.  It can help you quickly differentiate between isolated issues and widespread ones. If, for example, a smartphone device maker releases a security update for a specific smartphone model that unintentionally breaks the connected device app, looking at aggregate data across all users it can be much easier to pinpoint the common factors (e.g., smartphone model, OS version, language, geographic location, phone carrier, utilization of a common subset of application features) that could be the cause of the emerging issue. Given the highly dynamic nature of changes to smartphone hardware, OS versions, applications, carrier-specific customizations, browser versions, and so on, product analytics can help to identify, diagnose and pinpoint emerging operational issues.

As you read the previous paragraphs, you may have noticed that there is a fine line between end user support and adverse event reporting and management.  As product analytics gains more traction in the connected medical device space, it may end up raising the bar for our industry on what is considered possible or necessary for good quality management processes for reporting and addressing adverse events.

Use Case #3) Helping users better use their device and improve their health outcomes

The outcomes impact of medical devices, like other kinds of therapies, are often a “you get out of them what you put into them” value proposition.  In other words, just because a patient has a medical device, it doesn’t mean that they are guaranteed to take all of the steps to maximize the diagnostic, monitoring, and/or therapeutic value they can get from the device.   It’s the difference between device compliance or adherence and active engagement with a device.

Indeed, an entire speciality has sprung up around fusing the fields of user-centered design (e.g., design thinking), behavioral psychology and economics that looks at how we can borrow from the same methods used by social media companies to make their devices addictive (in the figurative, if not the literal sense of the word) or by consumer products firms compelling enough to buy, and use those to nudge patients towards healthier behaviors.  

This is where product analytics comes into focus.  Commercial firms with products and services to sell have been the bedrock of the user base for product analytics, growing it from an impressive $6.9B market in 2019 to a twice-the-size projected $13.9B market  by 2024.  So if medical device manufacturers can piggyback on this sophisticated, well funded and rapidly growing toolset, they can use product analytics data as a part of the patient care feedback loop that nudges and enables patients to make better use of their medical devices and achieve better healthcare outcomes.  

A few brass tacks examples to illustrate this somewhat abstract point:

Example #1) Predicting and averting patient decline

It is possible that patterns in this data could be correlated with future, high-risk/high-impact declines in a patient’s health.  For example, what if a medical device maker could:

  • Look at the longitudinal health of a diabetes patient,
  • Identify patients who have experienced a rapid decline in their health with irreversible harm to their body,
  • Correlate that to the product data to identify distinctive usage patterns of the device that are predictive of future declines, and
  • Use that as an early warning detection system to prompt any range of interventions to prevent that (avoidable) decline from even occurring

Example #2) Identifying where patients are repeating known mistakes and helping prompt them to avoid future repeats… at a personalized level

At a more micro-level, a medical device could:

  • Add product usage data to the mix of other patient data they have at their disposal,
  • Feed it into an artificial intelligence/machine learning (AI/ML) algorithm that identifies specific days when a patient’s chronic condition (e.g, asthma, dry eye disease, or chronic back pain) is likely to be worse,
  • Take that information to helpfully suggest/nudge them to plan their day prophylactically to minimize the impact of the chronic disease on that day.

Example #3) The Observer Effect and the gamification of health management

Scientists of many disciplines are aware of the observer effect where “the act of observing will influence the phenomenon being observed.”  Sometimes simply sharing the data captured about a person’s use of a device and sharing it with them is enough to change behaviors for the better.  This was the unanticipated lesson from a research study of asthmatics that led to the now-famous Propeller Health case study in digital therapeutics, where researchers learned that simply by showing patients how they used (and don’t’ use their inhalers) they could be induced to use their inhalers better.  

Others have taken these ideas to the next level through gamification, where the human instinct to compete is leveraged to induce people to improve their health statistics relative to the benchmarks set by groups and the activities of others in their social circle.  Witness how “step counting” became a group activity and a sort of cultural moment when Fitbit and others allowed the real-time measurement of human movement to become a digitally-sharable score.

As an industry, we’ve probably just scratched the surface of what we will be able to achieve by playing product data back to the person in an attempt to change their future health behaviors and outcomes.  

Product Analytics Dashboards and Reporting

Product analytics information is consumable without the need for highly specialized skills. This allows a range of professionals to consume the data, such as product management, design, and marketing. It can be used during initial design/development, clinical trials, and in production once the product is formally launched. 

We showed you an example row of raw data for product analytics. The following are examples of real-time reports and dashboards that show the power of this data viewed at aggregate levels for individual users and/or at cross-user levels: 

This following is an example of a product analytics dashboard implemented with Mixpanel. Custom dashboards allow various groups such as R&D and marketing to see exactly how users are (or are not!) using the product in real-time:

The next example (below) is of a product insights report (implemented with Mixpanel) that allows the device manufacturer to track how many people by country are using the software in a given time frame.  Product analytics tools usually come with a variety of out-of-the-box reporting options based on industry best practices, as well as customizable reports. Reports can also typically be exported so that it can be used as part of a cross-system data warehouse or analytics tool. These reports can often be generated and modified by analysts who do not have any programming skills.

The Product Analytics Vendor Marketplace

There is a rich marketplace of product analytics tools to select from including:

At Orthogonal, our go-to tool for product analytics with connected medical devices is Mixpanel. After doing an assessment of a number of tools in the marketplace over 2019 and early 2020, we found Mixpanel to be a good fit for connected medical devices because:

  • Mixpanel is built around events and user properties which are the building blocks of data that will be used for analysis and insights. It offers the ability to build custom reports and dashboards. This allows the creation of dashboards for different audiences in the organization such as product management, UX, development, and customer service. 
  • At the same time, Mixpanel makes it very easy to share the data it collects with other data analytics tools (e.g, external data warehouses) which could often contain sensitive patient data that you need to keep under strict controls.
  • Unlike most of its direct competitors, It can be configured to be HIPAA and GDPR compliant.
  • It features an easy-to-use interface and is easy to get up and running with basic queries on the data it collects.
  • The technical integration of Mixpanel’s APIs with the connected device’s code base is fairly seamless.
  • The pricing model requires a low initial investment and continues to be cost-effective as your usage grows, allowing for easy experimentation on new products and with new customers.
  • Finally, Mixpanel has been a collaborative partner.  They have been very supportive of the application of their tool to the full lifecycle of connected medical devices as a product category. 

Alternatives to Product Analytics, and Why They are Not as Effective

You might ask what are the alternatives to implementing off-the-shelf, product analytics using a tool such as Mixpanel.  We’ll be blunt in our answer so that you can spend your time making new and interesting mistakes and not the same old and proven ones that our team has already (painfully) learned with experience:

  • If you are going to implement product analytics, please for the love of your patients, their providers, your colleagues, and your family’s sanity, do not attempt to build a homegrown product analytics tool from scratch. It is possible to build your own product analytics data collection and analytics tools with custom software.  However, this is going to be expensive to build, expensive to maintain, as you’ll be bypassing the large economies of scale that these vendors have used to develop a rich suite of product analytics offerings. 
  • Another product analytics alternative we are frequently asked about is the use of Google Analytics (GA).  This is a very reasonable question, especially since GA is available at no licensing or usage cost.  However, while we have found GA to be a very useful tool for a variety of use cases, product analytics is not one of them. The biggest problem with using GA for product analytics is that GA does not allow you to access and analyze the full suite of raw data collected about your user’s past behaviors.  In other words, GA does not offer true product analytics since it does not give you access to the full person-level data!  So, in using GA you have already sacrificed a large part of the potential value of product analytics before you have even started your first analysis. 

What’s next?  How do I start my journey with product analytics?

So if this blog has piqued your interest in this topic and you are wondering what would be involved in adding product analytics to your medical device toolkit, we recommend the following initial steps:

  1. Define your requirements for product analytics. Start by working backward.  Identify the product analytics data you want to visualize and how it can help you. Figure out what you will actually do with the data. Identify areas to work on, and flesh out possible leads on data insights/trends with qualitative work to understand “why”.  It can even be helpful to prototype the analytic tools that you would want to create from your data as static, low-fidelity reports or even something more interactive like pivot tables based on mocked-up data.
  2. Address the privacy questions. Next, ponder the question, “Just because we can collect a ton of data about users, can we and should we?”  As with everything else we do as medical device professionals, privacy, ethics, rights in data, and respect for our patients need to be front and center.   Choosing a privacy-minded, regulation-compliant product analytics tool is an important first step. But in the end, your product analytics tool can’t prevent you from doing the wrong thing with what data you collect or how you steward that data.
  3. Evaluate the products against the requirements defined in the two above steps. You’ll want to look at other key factors such as HIPAA and GDPR compliance, ease of use, cost at various points of scale, ease of implementation, and support of key features such as cohorts, flows, and funnels. Once you have determined these, it is time to select a tool.
  4. Get a product analytics expert on board. It can be fairly quick and painless to get a tool like Mixpanel installed so that you are quickly collecting data and querying that data. However, there is a learning curve in terms of knowing which are the best data elements to capture, the right questions to ask of the data, and how to efficiently and effectively iterate through your analysis to rapidly arrive at the most useful insights. A product analytics expert has been there and done that before. So even if they know nothing about medical devices, they can apply their personal knowledge of healthcare and devices to work with you to help you accelerate your time to actionable insights and your overall ROI on your efforts.
  5. Instrument your app & launch. Note where privacy and confidentiality are needed and anonymize data where necessary.
  6. Design your dashboards to support your identified goals (from #1). Once an app is instrumented, information is real-time and needs to be monitored. With a product analytics tool, information is easily consumed via reports and dashboards. Each audience (UX, Product, Marketing) can have its own dashboard.
  7. Prioritize work for new features. Implement the new features and test in the field.

By integrating the rich data streams of product analytics about your users into your post-launch product enhancement processes, you should be able to accelerate continuous learning cycles that enable ongoing, safe, and compliant improvement of your connected device.

User Research -> Design and Test Intervention -> User Analytics

Conclusion

We’ll close with two final takeaways on this topic:

  1. Be faster. Product analytics is a powerful tool for gaining insight into how customers are actually using your product. It keeps the cycle of innovation going because the data provided can lead to rapid insights and faster release cycles that make your product less prone to user errors and more desirable to users. 
  1. Go Off-the-Shelf. Leveraging an off-the-shelf product analytics can help you implement product analytics at scale and receive feedback in real-time that can be customized for the various audiences in your MedTech organization including R&D, HF/UX, software engineering, quality and regulatory, and marketing. 

If you do go down this path, we’d love to hear from you directly (or through an anonymous email account!) so that we can share ideas and best practices.  In our estimation, product analytics is far too important to the health of each of our family, friends, neighbors, and co-workers to not help each other spread this value across every connected medical device.

If you would like to talk to our product analytics experts about how Orthogonal can help your team accelerate their journey to success with product analytics, please contact us.

If you think that the idea of using product analytics is terrible, or you have an experience that proves it’s a terrible idea, we’d also love to hear from you!

Put it all together and hopefully, you agree with us that this kind of real-time data provided can enable insights that lead to more rapid conceptualization and turnaround of new features. 

About The Authors

Bob Moll is the principal UX Architect at Orthogonal. You can email him at bob@orthogonal.io

Ke Li Yew is an analyst at Orthogonal. You can email her at kyew@orthogonal.io

About Orthogonal

Orthogonal is a software developer for Software as a Medical Device (SaMD). We work with change agents who are responsible for digital transformation at medical device and diagnostics manufacturers. These leaders and pioneers need to accelerate their pipeline of product innovation to modernize patient care and gain competitive advantage.

Orthogonal applies deep experience in SaMD and the power of fast feedback loops to rapidly develop, successfully launch, and continuously improve connected, compliant products—and we collaborate with our clients to build their own rapid SAMD development engines. Over the last eight years, we’ve helped a wide variety of firms develop and bring their regulated/connected devices to market.