Move Faster & Break Nothing Transcript: Lilly’s SaMD Best Practices

Randy Horton

Published on: January 3rd, 2022

Randy Horton:

We’re going to ask our first two speakers to introduce themselves. They’ll be giving us an overview of their best practices for accelerating SaMD development. We’re pleased to have Don Jennings and Carl Washburn joining us from Eli Lilly.

Don Jennings:

I’m Don Jennings and I’m in the Digital Health Unit at Eli Lilly and Company. We’re basically the group of people that build Software as a Medical Device (SaMD). Elsewhere across Lilly, we have peer organizations that build connected devices and, of course, many different peer organizations that actually create the drugs for our combination device products.

Carl Washburn:

I’m Carl Washburn. I work with Don and I used to work with Michael Iglesias, first at Johnson & Johnson and then here at Eli Lilly. I also worked at Siemens Medical. I have a background in software engineering and in developing Software as a Medical Device.

I tip my hat and thank Orthogonal for putting this opportunity together. We are all challenged to move faster by implementing Agile development methodologies within our software development lifecycle processes. When we say “break nothing”, we mean that not only must we be compliant with regulatory requirements, but also that our medical devices, including SaMD, are safe and effective.

Eli Lilly’s SaMD Development Best Practices

Don Jennings:

Carl and I are going to talk about five different areas around Software as a Medical Device development.

We say “best practices”, and we hope to outline best practices for you wherever we can. But I would also say we’re going to outline the areas that we have found most challenging. The flip side of something being challenging is that same something being very invigorating and very important. Every time I say something is challenging, I wouldn’t have in any other way.

These are the important problems we need to solve as an industry in order to bring digital into the therapeutic space, and to create therapeutic solutions for very complex diseases that make the lives of our patients better. Challenging is not bad. Challenging is the place where we all have to get better as an industry.

System of Systems

The first area I want to discuss is a “system of systems.” At Lilly, very rarely do we create a connected device, a piece of software or Software as a Medical Device that is standalone. What we usually create are SaMD working with a connected device, or a connected device delivering some type of drug. The three of those products – connected devices, software and SaMD – come together to create the therapy that we want to push into the hands of patients.

When you are working with people across multiple domains, such as mechanical engineers, electrical engineers, biologists, chemists, software developers and physicians, it’s very difficult to stitch all these different products together into a system of systems even when the teams making the products are in your own company.

But more often than not, your company doesn’t make every single system that you want to bring together to create one new system of systems. I think that the Dexcom G7 coming out is an excellent example of that. Dexcom makes a best-in-class continuous glucose monitor (CGM), which is why many different companies are integrating the Dexcom CGM into their product offerings. It’s a system that’s being integrated with other systems to create a higher value therapeutic product. You have to be able to develop with these different systems being created in parallel and oftentimes not even being created in your own backyard.

Among the things you have to take into account when you’re building a system of systems are the different levels of requirements. Every system has its own design history file and its own risk management file. It may be complex, but you’re in control of the entire stack as you develop it, test it and put it into production.

But when you’re stitching together all of these different system of systems into a larger product, you have to have requirements that go above what you’d have for an in-house product. You also need a risk management strategy and risk analysis that goes across these different systems.

Because you want to know which of these systems are going to create the potential sources of greatest risk for you, a top-down methodology for looking at your risk becomes even more important. You’re going to want to spend your risk analysis bandwidth on the most challenging areas, not the areas that are probably not going to give you a lot of issues.

You’ll also want to find mitigations across these different systems. System Three might have an excellent mitigator for a risk that System One creates in the system of systems. You want to know about that upfront, because you want credit for that mitigator system coming from System Three before you spend a lot of time trying to mitigate a risk coming out of System One in your system of systems.

Risk Management

The second area to take into account is risk management. This is a very challenging area in Software as a Medical Device, and in probably no matter what type of Software as a Medical Device that you create, for reasons of interoperability and system of systems. When you’re bringing different systems together, interoperability is at the forefront. Every system has to be interoperable with every other system that you’re combining to create a product.

The FDA certainly cares about interoperability. They have guidance about interoperability, and they’re going to ask you how your different systems are interoperable when you submit a combination product to them. You have to be able to characterize the risk of every individual system at the interface. So for example, you have a software development toolkit that’s going to be used by other companies as they use your product in their system of systems therapeutic offerings. You have to be able to characterize the risk at that interface, so they know where your risk analysis left off and where to pick up their risk analysis.

Talking from a failure mode and effects analysis perspective, you have a sequence of events to cause and chain together that could potentially lead to a hazardous situation. But you only understand the sequence of events and the probabilities up to the point where it’s going to interface with another medical device. You can only characterize that partial probability of a hazardous situation being triggered, and you have to characterize it in a way that another user of your medical device or Software as a Medical Device is going to be able to incorporate into their risk analysis.

We found that to be a very challenging yet invigorating topic in the last year at Eli Lilly, as we start to integrate other people’s products into our Software as Medical Device, and as we let other companies use our products in their Software as a Medical Device offerings.

Developing SaMD Using Agile

The third thing I’ll mention, before I hand it over to Carl, is the use of Agile in the software development process. Carl likes to say that design control is fundamentally a waterfall methodology. You have five phases, Phase One through Phase Five. Phase Two, Design Outputs, is where all of the software development occurs: your architecture, your design, your coding, your unit testing. We found Phase Two to be the easiest area to start to adopt Agile methods within the design control process. But there are a few tricks to being able to do Agile effectively.

Something that Carl taught me, when I first met him several years ago, is that you have to code from approved requirements. Your requirements have to be locked up front before the developers start to develop. You can add to your requirements as you go in rolling approvals – you don’t have to know them all before you start.

But in an Agile methodology, where you’re working sprint to sprint, you’re not always sure what you’re going to tackle in that two week sprint, let alone in the multiple sprint increment. You need the flexibility to say, “We’re not going to work on this requirement this sprint, we’re going to work on a different one.”

How do you do that within the Agile framework? We found that you can have your product owners move user stories into a sprint. If you can capture that in an unambiguous way, then you can say that the moving of the user story into your sprint is the approval of a system requirement or equivalent thereof. Then you can have the development team work on features out of that user story during the sprint.

At the same time, the team can be creating the software specifications. You can do a finish-to-finish: At the very end of the sprint, you can have the software done, you can have the software specifications written that go with that system requirement and that user story, and the product owner can approve it at the very end. So you’re preserving the spirit of only developing against requirements that are approved, but you’re allowing the development teams the flexibility to do it in an Agile sprint-based way. Carl, I will hand it over to you for the last two topics.

Content and UI Verification

Carl Washburn:

Thank you, Don. I wanted to tell you what a pleasure it is working in close partnership with you at Lilly, and how important it is for the development team to be joined at the hip with a quality organization. It allows us to ensure that we look at the shared objectives we have as an organization of moving faster and breaking nothing. It also ensures that we maintain regulatory compliance and that our medical devices are safe and effective.

I’m going to talk about two processes within design control that required us to think originally; where we needed to update our approval and verification as well as review processes. I’m going to start with content and user interface verification. If you think about everything that’s within a mobile medical application, everything that a user sees, all the words, all the sounds and all the icons, all of that is content and user interface.

When we first started, Lilly had begun transitioning from a pharmaceutical manufacturing organization to a software development organization. We have a lot of stakeholders within Lilly who care about what the user sees, including legal, privacy, global patient safety, cybersecurity, business quality and others. Back then, every one of those stakeholders had their own little approval process. They said, “Hey, if you want this screen to say this, you need to go through our approval process. Here’s the steps, here’s the tool, here’s the checkpoints and here’s the governance.”

Imagine every one of those groups wanting to have the final word. At every turn, we heard, “Make sure you come to us last and let us have the final say.” But they all can’t have the final say. How do we take so much input from stakeholders that may have conflicts in what the user should see and turn it into something that we can all agree upon internally, while looking at all of our review and approval requirements?

Our solution was to create a formalized governance committee. It’s made of the many teams mentioned earlier, such as legal and cybersecurity. What we do now is a formal process where we have pre-reads, formal meetings, action items and meeting minutes, and all of those are overseen by the quality processes. All in all, it’s probably a six page procedure.

You may think, “Well, that sounds easy.” No, it’s not. The difficulty was aligning the organization to follow this common review and approval process. It took almost a year to get agreement from all the stakeholders, and to get them to understand it’s in all of our interests to make sure that we meet these requirements. It’s not majority rule. Any team member can press the emergency stop button and say, “No, this does not meet privacy requirements, we need to resolve it across the other organizations.”

We’ve had this in place for a year and it’s worked remarkably well. It shows that we can take difficult processes, such as Agile methodology that Don was talking about, or something seemingly as simple as agreeing upon what the content should say, and create the supporting processes necessary for internal and external auditors – as well as a common voice to our customers.

Technical Validation

That’s one example of thinking originally. The other example is what we call technical validation. When you think of validation, you typically think along the lines of human factors: We need a plan, we need protocols, we need a report and we need to move forward.

But if you look at physical medical devices, such as an injector or a blood glucose meter, there’s a lot of testing that falls under the heading of technical validation. Some of those tests are drop tests, some are exposure to high heat, high humidity or to low heat, low humidity. They look at the normal expected use of that physical medical device and then, through validation and technical testing, they push it to the limits. Will it fail if the patient puts this device on the dashboard of their car and lets it bake in 130 degrees, for example.

How do you translate those kinds of tests into software? At first glance you think, “Well, software probably would work fine in a hot car, software would probably work fine if you’re driving over a bumpy road, and none of that really applies.” That may be, but the concept still does. You just have to think about it differently.

What are the common causes of failure for software? What are some of the typical root causes of software recalls? Maybe the patient is on an airplane crossing over time zones. Maybe a patient goes into a low bandwidth area and has only one bar. What happens if we’re trying to push a lot of data back and forth to the app? What happens if we have high server loads? So we came up with a template of common causes of software failures. That template is included in our requirements and our testing, and we point to it as part of our technical validation to compliment the human factors’ traditional validation.

That’s the creativity that we have to exercise. I thank Bernhard and Orthogonal for helping us understand how to take Agile methodology and put it within the context of what would seem like a highly disciplined, highly structured development model. They’ve shown us where we can benefit from modern software development practices. We are on a never-ending journey, and that’s why these forums are so valuable. We listen to you and we learn. And the faster we learn, the better for all of us.

Randy Horton:

Thank you. That was terrific. You packed 10 hours into your 15 minutes. We appreciate the insights and your time. It’s a great context setter for the next several speakers.

Below is a list of the other sections of our Move Faster & Break Nothing Webinar: