Case Study: Capturing Quality Images for ML Algorithm

Scope

Our client, a MedTech startup, engaged Orthogonal to build a Software as a Medical Device (SaMD) in the form of a mobile app with a cloud backend for their Machine Learning (ML) diagnostic algorithm. The algorithm identifies signs of strep throat by analyzing images of patients’ throats, making it a Clinical Diagnostic on Consumer Electronics (CD-CE). The client had trained their algorithm with data accumulated over the last five years, and were ready to invest in converting its core logic from Python into a native mobile ML model in Swift that would be embedded in the mobile app part of their device. The client asked Orthogonal to build an iOS app for the iPhone SE for use in a clinical trial of their algorithm.

Challenge

To be used in a clinical trial, the app needed to guide non-medical professional users in using their smartphone camera to take images of their (or another’s) throat. Captured images would be analyzed by a combination of three ML algorithms, two that reside in the smartphone app and one in the cloud, which delivered a confidence level for a strep throat diagnosis.

For the algorithm to work, users needed to take ten high-quality images, with good lighting, visibility of areas of the throat like the uvula and no teeth or tongue blocking the view of the throat. One of the key challenges for Orthogonal and our client was figuring out the best method to capture these images with the iPhone. The method needed to satisfy two different requirements. First, photos captured by the app needed to be of sufficiently high quality to meet the needs of the device’s algorithms. Second, the process had to be easy enough that the average user could accomplish the task and come away having had a positive experience.

Approach

At the outset of the project, the app’s approach to capturing images was to guide users through the process of taking a running video of their throats. Individual frames would then be selected from the video and fed into the diagnostic algorithms. This method had worked well during the client’s initial device R&D, including performing successfully in controlled experiments.

However, once Orthogonal began developing the smartphone application and testing it with representative users, it was discovered that this method regularly yielded images of too low-quality to be used for diagnosis. The chief issue was the variance in the built-in camera across different iPhone SE models. For example, one model could have three camera lenses, whereas another could have just one lens. iOS also contains proprietary algorithms for ensuring image quality that interface differently with different camera setups, making it challenging to standardize the capture process.

With the device on an aggressive timeline, both sides needed to lean into Agile’s ability to reprioritize tasks based on changing circumstances to determine a better method of collecting images.

Result

Thanks to Orthogonal and the client coming together to dynamically shift the deliverable priorities, within a matter of weeks the development team pivoted from capturing images of the throat through video to taking multiple individual photos consecutively. When tested, this approach yielded photos that were of the necessary quality to be analyzed by the device’s algorithms.

From this experience, the development team took away the following lessons:

1. Address areas of high technical risk first: Our team was reminded of the importance of prioritizing areas of high technical risk at the outset of a project. Even under a tight deadline, uncovering big risks early and iterating to find a working solution is worth the time and investment.

2. Flexibility in development: Agile methodology was instrumental in allowing both developers and the client to make necessary changes to the project in progress, continuously reprioritizing our work to deliver what the client needed based on information uncovered in the development process.

3. Communication is key: As a trusted business partner, Orthogonal was able to maintain clear, honey and transparent lines of communication with the client. Our good working relationship allowed us to communicate effectively to arrive at the right outcome.

This case study underscores the value of adaptability in the development of SaMD. Agile methodology, thorough experimentation and strong client-developer communication helped to overcome unexpected challenges and achieve a successful outcome. Lessons learned were discussed amongst Orthogonal’s delivery managers and documented in a shared Confluence space for reference on future client projects.

– – –

Need help developing a medical device that integrates a native iOS or Android app and Machine Learning algorithms? Orthogonal’s team of Agile developers will work with you to create a user-centered, safe and effective mobile medical app. Contact us for more information on our mobile medical app development services.