Planning for Feedback: Three User Thursdays

There are many excuses for not getting feedback from users:

• They’ll steal our intellectual property.
• They’ll reject it if it’s not perfect.
• It takes too much effort and is too expensive.
• If we talk to them, we may have to change what we’re building, and it’s too late for that.
• We don’t have time.
• We’re afraid of being proved wrong.

For Orthogonal, the value of getting feedback from users far outweighs the inconveniences. Getting early feedback on our Software as a Medical Device (SaMD), Digital Therapeutics (DTx) and connected medical device systems increases product effectiveness and reduces risk. Furthermore, changes to the product can be made when it is relatively inexpensive to do so, rather than waiting until the end of development or even after release. It’s imperative for medical device software developers to get this right, as we want to avoid use errors and maximize patient engagement. 

In Orthogonal’s Agile in an FDA Regulated Environment eBook, we reference the following statistic: According to the annual Standish Group CHAOS Report, 45% of software features that are built don’t get used. It may be because they’re not important to users, or because of inadequate design. At the same time, important features are often missed or discovered late in testing. Fixing mistakes so late in the process puts undue strain on the development team, increases costs and extends the project timeline. 

When it comes to software development, you seldom get it right the first time. That’s why frequent feedback from users is so important. By starting with prototypes and moving through development, you are able to iteratively improve the product. User feedback prevents wasted effort on unnecessary features, makes the features you do build usable and effective and reduces human factors risks. Given the long development cycles for medical devices, this is an especially useful technique for accelerating the product development lifecycle.

User feedback needs to be planned for and made part of the development process. We borrow the concept of “Three User Thursdays” from Lean User Experience. In this model, you define a testing cadence and line up users even in advance of knowing what you are going to test. You then maintain a backlog of things to work with them on, from prioritization of features to task flows to testing alternative designs. There is always something to test with users, particularly early in the project timeline when the design is evolving at a rapid pace. 

While Three User Thursdays testing is informal, it is still executed with a simple moderator’s guide as an input, and you record what was tested and the questions asked. As an output, we create a PowerPoint deck that summarizes key observations and recommendations for updating the design or tested material. If mock-ups or a prototype are involved, screenshots are included so there is a clear record of what was tested. It’s important to note that testing can easily be done with remote users. Figma and Zoom make it possible to have users drive a clickable prototype while sharing their screens. As a bonus, multiple observers can participate without them getting in the way of the test. 

The Three User Thursdays process involves integrating the disciplines of User Experience (UX) Design and Human Factors. The UX designer and Human Factors researcher work closely to come up with both the stimuli for testing and the moderator’s guide. Other team members are needed to recruit users and make sure they’re available for testing. Three User Thursdays testing can be informal (though documented), with formal testing at less frequent intervals. Integrated design and Human Factors, as well as a steady testing cadence, will give your design process a discipline and rate of effectiveness not otherwise possible.  

The insights obtained from Three User Thursdays can be surprising. For example, in testing an onboarding process for an app, we discovered that users did not mind a lengthy process as long as it provided valuable information at each step. The desire to get fully up and running outweighed the greater time investment. In another example, users were asked to create an account as one of three user types. During Three User Thursday testing, we discovered that some users were not clear on which of the three types applied to them. Because this was discovered early in the product design process, we were able to improve the account creation flow to make choosing a user type path unambiguous. 

We recommend the following best practices for implementing Three User Thursdays:

• Test early.
• Use Figma for design and quick-to-create interactive prototypes (also enables easy remote testing).
• Always write a moderator’s guide.
• Write a results report with screenshots and recommendations.

Although Three User Thursdays is a qualitative discipline, you can use quantitative data to inspire topics to test. For example, by integrating Product Analytics into your product, you will receive data after release of how people are actually using your software. This will reveal the “what” of behavioral patterns, such as account sign-ups or drop-offs, but not the “why” behind the data. Three User Thursdays can be used post-release to answer those “why’s” and solve issues that were not evident pre-release. 

A well-planned and well-executed Three User Thursday testing process will deliver returns that more than make up for the time investment. The benefits you’ll gain in terms of product effectiveness and risk reduction cannot be overstated. It not only helps you create a better product, but also sets you up for success for when you need to invest in larger and more formal formative or summative tests. You’ll end up with SaMD, DTx or a connected medical device system that meets users needs and improves health outcomes.

About the Authors

Bernhard Kappe is a software development leader, technologist, and pioneer who consistently provides his clients with an unfair advantage by introducing them to important new trends and techniques at the leading edge of the adoption bell curve (e.g., Agile software development, Lean User Experience, Software Product Management, DevOps, Lean Startup and open-source software).

For the last eight years, as founder and CEO of Orthogonal, Kappe has exclusively focused on adapting these approaches to the medical device and HealthTech industry.

He has worked with clients including Novo Nordisk, Google, Tandem Diabetes, and Nanowear to connect medical devices to mobile apps and the cloud while shortening product development life cycles, improving delivery predictability, and creating engaging user experiences. By developing new diagnostic and treatment devices and business models, his clients have seen software-enabled market growth and improved patient outcomes and have contributed to bending the healthcare cost curve.

Bob Moll is Principal User Experience Architect at Orthogonal and has over 20 years of experience on a range of design projects, many in the healthcare space.  Bob has delivered presentations on the value of and best practices for Product Analytics and Three User Thursdays at the Human Factors and Ergonomics Society’s (HFES) International Symposiums on Human Factors and Ergonomics in Health Care.