Talk
Accelerate Updates to AI Algorithms with PCCP: Webinar
Talk
Operating Medical Devices in Cloud: Experts build consensus on safety
Orthogonal’s Bernhard Kappe and Randy Horton were thrilled to participate in a webinar on December 16, 2020 place that was hosted by our partners at MedTech Intelligence titled Connected Medical Devices, Software as a Medical Device (SaMD) & the Cloud: Delivering Validated States, Safety & Compliance on Continually-Evolving 3rd Party Platforms
This event was a direct outgrowth of our work with a talented group of industry professionals who came together in 2020 to create the first-ever written industry guidance about best and worst practices for responsibly operating functions of medical devices in the cloud.
If you missed the talk and want to check it out, or you attended catch the talk and want more or want to share it, you have a few options:
- You can watch the video of the talk on our Youtube channel.
- Scroll down this page to read the transcript of the talk and see the companion slides.
You are also welcome to:
- Provide input to the Consensus Report’s Working Group by completing this survey.
- Read a companion article in Biomedical Instrumentation & Technology (BI&T) titled Hey You, Get On the Cloud: Safe and Compliant Use of Cloud Computing with Medical Devices.
Event Summary
The tremendous growth of public cloud computing platforms such as Amazon Web Services (AWS), Microsoft Azure, Salesforce Platform and Google Cloud has been driven by a wide variety of benefits related to reliability, performance, scalability, security, convenience, cost, and functionality. Seeking to capture these benefits, more and more medical device companies are placing key components of medical device software and functionality on public cloud computing platforms.
While the cloud offers high value for medical device manufacturers and device users, it also creates new challenges. Use of the cloud entails a tradeoff between realizing the benefits and losing some degree of control over the underlying technology. Although regular cloud component updates from Amazon, Microsoft, Salesforce, Google, and others can help keep their platforms safe and reliable, constantly changing computing environments pose a new class of challenge to traditional approaches to the idea of a validated state that ensure safety and compliance.
In this webinar, a group of industry leaders and pioneers in the use of medical devices in the cloud outlined these challenges in more detail, and share an emerging set of best practices that address these challenges so that medical devices can harness the full benefits of cloud computing. They will be sharing insights from their experiences in developing and launching cloud-based medical device functionality, as well as interviews they have conducted with professionals with experience in this area.
The content they presented is a “sneak preview” of recommendations they are in the process of creating for that will provide the first-ever written industry guidance on this important topic. This working group has been approved by the Association for the Advancement of Medical Instrumentation® (AAMI) to create and (after AAMI’s review and approval) publish the final guidance as AAMI CR510, A Consensus Report on the Compliant Use of Cloud Computing for Quality Systems and Medical Devices.
Event Speakers and Panelists
The working group for this Consensus Report who will be presenting at this event are:
- Clay Anselmo, Principal Quality & Regulatory Consultant
Shriner & Associates - Mike Attili, President, Amaxo
- Pat Baird, Head of Global Software Standards, Philips
- Randy Horton, VP Solutions & Partnerships, Orthogonal
- Bernhard Kappe, CEO, Orthogonal
- Artiom Kisselev, Master’s student, UC Berkeley/UCSF Program in Translational Medicine
- Joe Lewelling, VP of Emerging Technologies & Health IT, AAMI
- Josh Schulman, SVP of Clinical Innovation, MaxQ AI
- Nicola Zaccheddu, Sr. Software Quality Engineer, Philips Healthcare
The event was moderated by Thomas Maeder from MedTech Intelligence. Also joining the panel discussion was Tom Brennan from Rootstock Software, sponsor of the event.
Who Will Find Value in This Content
The webinar (and the in-process consensus report) are designed to inspire discussion among cross-functional teams across medical device manufacturer’s organization and in the broader MedTech industry. We encouraged attendees to take time to debrief with their internal teams and compare notes after the webinar. Stakeholders in this content include:
- R&D, Product Development and Engineering Teams – Hear software integration and product development case studies for cloud application
- IT/IS, Software and Data Teams – See successful software development, security and data strategy implementations
- QA, RA Teams – Discover strategies navigate compliance, safety and security concerns
- Executive Leadership, Market Access, Digital Health and Connected Device Product teams – Learn from the success of others already succeeding with Cloud Strategies.
Webinar Presentation
Tom Maeder
Hello, I’m Tom Maeder, the conference director at MedTech Intelligence, which provides education and information for the medical device and diagnostics industries through e-publications, conferences, workshops, and webinars. Welcome to today’s program on connected medical devices and the cloud.
Tom Maeder
I’m pleased to present an extraordinary group of industry experts for today’s webinar, and want to express my appreciation to Rootstock, which kindly sponsored today’s program, so that we could offer it to a broad audience at no cost. A few housekeeping items before we begin. By default, you were listening through your computer speaker system, if you prefer to listen by phone, select telephone in the audio panel, and the dial-in information will be displayed. If you have audio issues, log out and reconnect. And, if the issues persist, contact Veronica Allen via the chat box, and she will assist you. You can download a certificate of participation, and a PDF of the presentation from the handouts panel on the screen. Please download these materials during the webinar, as we cannot send them out after the webinar ends. You can ask questions during today’s event, by typing them into the question’s area of the control panel. You can submit questions at any time, and we will collect them and ask them at appropriate times during the program.
Tom Maeder
So, public cloud computing platforms, such as Amazon Web Services, Microsoft Azure and Google Cloud offer tremendous undeniable benefits in reliability, performance, scalability, security, convenience and costs. Amazon Web Services was introduced in 2006. Medical device companies were not early adopters, but by 2020, they’re increasingly placing key components of device software and functionality on public cloud computing platforms. In a field governed by compliance requirements for safety and validation, this poses certain challenges. We all want medical products and services to be safe for ourselves and our families. There’ve been many discussions, and there’re many misconceptions about using the cloud for QMS and medical devices, but it has not been clear what best practices should be in this rapidly expanding area. Pat Baird of Philips, who will moderate the panel discussion today, initiated conversations with today’s group of speakers and panelists.
They approached AAMI, the Association for the Advancement of Medical Instrumentation, with a proposal to create a consensus report that would lead to a guidance on the safe and effective use of cloud computing as part of the QMS system or the functioning of a medical device. Amy agreed and granted the charter to develop Amy consensus report number 510. Today, you will have the opportunity to hear from the group about their work in progress and to ask questions or contribute your comments. And with that, let me turn the program over to our first speaker, Randy Horton was vice president of Solutions and Partnerships at Orthogonal.
Randy Horton
Thanks Tom, and big thanks to the MedTech Intelligence team for putting on this event, and to all of you for taking time out of your busy workdays, and also your pre-holiday schedules now to attend this event. To add a little bit on one point that Tom made before, we’re going to be presenting today on work, we’ve been developing towards this AAMI consensus report on using medical devices in QMS and the cloud. However, we have not yet submitted this report and gotten it revised, and approved and published by AAMI. So, what you’re hearing today is the opinions of the presenters who have been chartered by AAMI to submit their ideas. And hopefully, sometime around quarter one, some version of this will be published as official AAMI guidance.
So, what do we mean when we say cloud computing? Before we get started, it’s probably good to make sure we’ve got a common definition without getting too deep.
NIST has a great, concise definition, a little technical, as you would expect from NIST. Cloud computing is a model for enabling ubiquitous, convenient, and on-demand network access to a shared pool of configurable resources that can be rapidly provisioned, released with minimal management effort or service provider interaction. So, basically, server-based computing used to be a series of products like network servers, storage, and applications you had to procure, install, and manage. Now, it all becomes an on-demand service. Or in financial terms, your fixed capital expenses to buy and manage infrastructure have become flexible operational expenses, you can scale up and down as you need from third-party resources.
Randy Horton
This goes on to define some key characteristics that we’ll cover briefly. If you’re going to call something cloud computing. Cloud needs to be on-demand. You basically, a consumer should be able to log on, provision what they need online without having to call a salesperson or a support rep, such as server time, network storage or more disc space, and scale that up or down as they want. Cloud computers need to be networked. Really, what does that mean? They’re in a data center, that’s on the internet, and you’ve got good network access that can support mobile phones, tablets, workstations, connected medical devices, connected cars, anything. And to do that, you don’t have to physically touch the computer to control it. You can control it remotely. Resource pooling, basically you’re buying access to computing power, and storage and bandwidth with the cloud, but you’re not buying the actual computer.
And somebody on the backend, the cloud providers figuring out how to allocate the resources efficiently across all their computers. Your computing, processing, and data might be sitting right next to your biggest competitors. You wouldn’t know. And, the truth is you really don’t have to care. It’s elastic. Like we said, “Cloud should be able to scale up and down as needed. So for a retailer, for Christmas, natural disasters, you scale up. For an employer, they might scale up during the workday, when everybody comes in and is on the systems. Or for Netflix, you might scale up around July 4th at 5:00 PM, when you drop an entire season of Stranger Things, and a rather large number of people stay up all night to watch the entire season.
Finally, it’s measured. Basically, all usage in the cloud is instrumented, so that the provider, and you as the customer know exactly when and how you’re using the computer. And, by really instrumenting and getting that detailed, granular view of what’s going on, you create some value-based pricing models that are really much more effective than having to make a large upfront investment, and hoping that you’ll then get a good utilization out of that fixed investment you’ve made. And, this also talks about variation in the service models for cloud computing in two ways. The first dimension is Software as a Service, Platform as a Service, or Infrastructure as a Service. Software as a Service, you could think of as like Office 365 or Gmail. You’re getting an application online, that’s all you need, you don’t care what happens below it.
Platform as a Service means you’re building a solution online and you’re deploying it, but you don’t want to worry about the actual computers, you just want to know that they’re there, and you can get as much of them as you need.
Infrastructure as a Service, the third one is really closest to the more traditional model, where you are literally renting computers and they’re your computers, and you’re getting a certain amount of computing power to do whatever you want. The second dimension, I really don’t want to go into other than they say that, “We are talking today about public cloud, not private cloud or hybrid clouds.” And, for the purposes of today’s talk, if you know what that means, great, and if not, I really wouldn’t worry about it.
Randy Horton
So, if you haven’t noticed already, we’re pretty pro-cloud and we do think the cloud is awesome. Some of us on this team have been using the cloud computing really since right after Amazon introduced AWS back in 2006. And, there’s been great growth in the cloud for some really good reasons. And the biggest one, we would really tie back to economies of scale. It’s really about the ability of the cloud to share infrastructure across organizations, standardized things on cost-effective utility computing, and scale up and down on-demand. It’s a very, very profitable business model for the providers, if you’ve looked at the stock prices and financial reports of some of the major cloud providers. And, all that profit creates an amazing capacity on their part to continually reinvest billions of dollars in profit, and making their services even better. So as a cloud customer, what you’re getting is constantly getting richer and better functionality at all points in the technology stack, better security, privacy, and organizational confidentiality, reliable systems, processing, speed, network access, and just an ease of connecting information systems to each other.
Randy Horton
We’re also very pro-cloud in the operation of the QMS and medical devices. We’re already seeing value for operating QMS and medical devices in the cloud. And, we’ll talk about some of those use cases today. We think this trend will continue. To be clear, we’re not saying that all computing for QMS and medical devices belongs in the cloud, but we do think it has a very important role to play in our industry. And next, Bernhard Kappe is going to give some few examples of how cloud is already being used in medical devices and QMS.
Bernhard Kappe
Thanks Randy. So, there’re a lot of different ways that the cloud is being used, and can be used in and around medical devices. I’ve listed a couple of examples that cover a few common scenarios. Some of these are ones that we are going to be addressing as case studies in the consensus report, and addressing the major issues around those. But, we’re always looking for more interesting examples. So, if any of our viewers have interesting examples, we have some contact information near the end of the presentation, would love to get ones that you have gone through and worked through, or that you’re aware of. We’re always looking for more examples. One of the areas where the cloud has been used a lot and will continue to be used a lot is in software that’s used to support a medical device life cycle processes. Everything from to eQMS to ALM, PLM, to configuration management, to manufacturing control.
An example here that we have of this is a software defect tracking application for the QMS, where the storage and processing for the defect tracking is happening in the cloud. Then, there are other examples where either the software is, what you would call, an MDDS medical device data system. That is not part of the functioning of the medical device, but is used to capture, and store data and information, and report on that for a medical device. There are a lot of these, and this is one of the most common cases for the cloud, but it’s also a relatively low risk area.
The ones that we think are very interesting and where the thorny problems are, are those where it’s actually medical device software that is in the cloud, software as a medical device, or that as part of medical device functioning. So for example, A skin cancer detection software system, where photos are captured in a smartphone app, the images are analyzed by an AI algorithm in the cloud, and the results returned via a smartphone app, or a software for instantly insulin therapy adjustment recommendations, again, where a lot of the processing of the medical device functionality as in the cloud. Or a computer aided triage and notification system for early recognition of intercranial hemorrhage, basically a lot of these imaging and MRI algorithms, basically, that are used by radiologists. This is one of the big areas where there is an awful lot of SaMD work that is either in the cloud, or potentially local or both, depending on the circumstances.
Randy Horton
Thanks Bernhard. So, every new technology creates new opportunities, but also new issues. You can call them the unintended consequences. And, that’s especially true with new technologies that are enabling something that’s intricate, regulated, or important like a medical device used for the diagnosis monitoring and therapeutics of the human body, our bodies. I love this little comic of a kid saying, “Daddy, what are clouds made of?” And dad replies, “Linux servers, mostly.” For those of us in the medical device world, looking at how to use cloud in the operation of devices, there’s certainly a wisdom in this comic. Using the public cloud in the operation medical devices and QMS raises some really interesting and tricky questions. What does it mean to have a validated state where your device is using remote computers under someone else’s control? And by design, we don’t necessarily understand or have access to information, on exactly how those computers are operating. Or even what’s happening with them on a minute-by-minute basis.
And in fact, really that’s by design, the whole cloud computing model is built around the idea, you don’t have to worry, the computer will be there and will take care of it. And, while those organizations take their massive profits and have built them doing a really good job of keeping those things running smoothly more or less, and frankly, generally, are often more smooth than we’re able to on our own in our own data centers. It begs the question. If you don’t know what’s going on with your medical device, how do you know that it’s still in a validated state, and it’s operating as designed and approved? So that’s the crux of today’s topic. And, that’s what Mike’s going to start covering in the next section here.
Michael Attili
Thanks Randy. So a question that is, why are we working to develop a consensus report? The members of this group have been working to develop guidance for the use of cloud technologies for medical devices, for almost three years now. And during that time, we’ve heard from many people in the industry pleading for information. It reminds me of the days before AAMI published the TIR on the use of Agile practices. Everyone wanted to use it, wanted to get in on the fun, but they weren’t sure whether it was appropriate, whether it was going to be safe enough, what things they needed to do? The need for some recommendations, for guidance on the use of cloud services has only accelerated over the last nine months with all kinds of digital health platforms moving into the cloud to respond to the pandemic. We’ve wrestled with how to best collect and present what, we on this team and others in the industry have learned.
And as usual, there’s a wide disparity of experience among manufacturers and their familiarity with cloud services. We were concerned that the approval process for a TIR or for a full fledged standard would take too long. And, we’d be either forced with trying to force something out too soon, or that by the time the information got out there, would have been outpaced by developments in the industry. And, we wanted to be able to publish something that would still be useful. So, a consensus report is intended to be able to respond to urgent needs for guidance, while information is still developing in emerging areas. And, that’s really appropriate for medical device use of clouds.
The CR will allow us to establish a common language, and a common set of concerns to frame how to think about using cloud services in this context, and also to collect and to document the best practices and some of the worst practices, the things that you want to avoid. Most importantly, we want to be able to level the playing field, so that manufacturers can reduce the overall risk by ensuring that the developers are aware of the benefits that you accrue from using cloud services, but also the issues that you need to be aware of. And crucially, to be able to ask the right questions when considering how to take advantage of cloud services in the products and services that you are providing.
Michael Attili
So, using cloud services for medical devices, it’s really about trade-offs. It’s recognizing the value of what new things this technology can provide, but also understanding what we’re losing by abandoning the old ways of doing it. And, identifying where the real issues are, versus the perceived concerns, or the fear of adopting something new. The trade-off is gaining a tremendous amount of flexibility, functionality and return on investment from the economies of scale that these cloud providers. And, not just the big service providers, but also those providers that offer niche services that are based in the cloud. That is balanced against, that losing some level of control, and some level of visibility about how the computing that you’re using is actually working. Overall, we believe that the trade-offs are excellent, and that the new wrinkles, if you will, of the cloud for medical devices and the quality management systems, it’s a very good trade-off. But, that certainly doesn’t mean that it’s appropriate to use the cloud in every situation, or without certain special considerations related to the safety and efficacy of how you’re using it.
Michael Attili
Approaching the integration of cloud services into a medical device, it’s similar to the approaches that we’ve taken in the past for other new IT advances. Cloud services share a lot in common with the adoption of multifunction devices, distributed network systems and AI. With multifunction devices like distributing your app or your service on a smartphone, you’re giving up some control over the hardware and the operating system that the device runs on, but you’re still ensuring that your product is working the way it’s intended. With distributed systems, you’re giving up control over the network, over the connectivity, and in certain instances, over the security of the information that you’re transferring. And with AI, you’re giving up control over, or at least some control over the algorithms in how decisions are made or being reported. Just like with these other technologies, when you’re adopting cloud services, your quality system needs to adapt in order to make sure that our devices stay safe and effective.
The change is driven by cloud adoption fall into five areas within your QMS, risk management, security management. And, they go together really defining what it is that you need to be concerned about. Validation, and what your validated state looks like. And then, change management and purchasing controls associated with your service providers. In each of these areas, we’re developing a set of questions that need to be considered and addressed. Things like, what type of changes do you need to have prior notification of and what types don’t you? And along with those questions, a set of suggestions or recommended approaches, where we believe that there’s a consensus among the experts in the field. Our assumption is that cloud technologies have less risk in some areas like security and scalability. But more than others, like transparency of the changes and maintaining a validated state. But, that you can create an appropriate risk management approach for your cloud-based medical devices.
Your security risk management extends into considering the risks from cloud computing. And then, you can create appropriate validation approaches, and change and purchasing controls that are compliant with 21 CFR 820. And, we want this consensus report to show you how to do this.
Michael Attili
So, in risk management area of your quality system, many of the risks that you need to address are similar to those that you’ve already addressed, or likely already addressed for multifunction devices, or connected systems with internal IT services. For example, there are concerns about service availability, security, data integrity and change control.
But cloud services for medical devices have additional risks to consider in each of those categories. The public cloud services make the option more accessible to manufacturers, particularly ones that don’t have deep experience with distributed IT services. And, the essential issue that device manufacturers face is the limited visibility into the changes in the computing environment, and control over when those changes are being deployed. And so, those two issues are really the crux of where you need to adapt your QMS, and where you need to consider changes that you’re going to use to maintain your safety and the effectiveness of your devices.
As Randy mentioned earlier, and Bernard somewhat too, “There’s a significant difference in patient risk related to the computing environment for quality management systems, or for tools that you do production control software that you’re moving into the cloud, versus ones that have a direct clinical impact on the patient.” Those device outputs have a direct effect on patient risk, whereas the QMS and production controls have secondary types of effects. And, there’s often redundant detection controls that reduce the risk even further. So, that distinction needs to be reflected in your risk management and your validation approaches.
The types of changes that you need to consider are additions to services, or improvements, or fixes that the cloud provider offers. There’s security vulnerabilities, and you need to address those specifically in a security risk assessment, and things related to scalability and load balancing. Your service providers need to be able to accommodate a wide variety of customers, and their needs may spike at different times. And so, your service might be subject to changes in scalability, or changes in the location, or the location or the performance of the computing that’s being done, you need to make sure that you consider that as part of your risk assessment.
And then finally, you need to be aware of, just like for many of your other suppliers, what to do in the event that there’s a disaster and that they can no longer offer that service. Or if they choose to go out of business or stop providing that product, how will you address that with your products that are already in the field?
Michael Attili
With security and compliance, the key differences there are that you may not know what vulnerabilities are present in the software or the systems that are underlying your service and you may not be notified when those services are under attack or when they’ve been attacked and there’s been some type of exposure. You also have a whole new set of people that may have access to sensitive information that’s being stored or transmitted as part of your service. And then finally, you may have restrictions on “where”, what legal jurisdiction or what geographic area your data is processed in or stored, because you need to identify all of those issues and track them and develop mitigations for them just like any other risk or security concern in your system.
Unique probably to cloud platforms is that you’re likely running on a shared service with other customers, some of which may have less security restrictions or concerns than you do. And so there’s a possibility of lateral spread that an attack on one of those other customers that breaches their internal security could leak into your service. And so those shared resources, whether it’s the computer or whether it’s the network or whether it’s the storage systems, may be compromised and that may impact your service. So it’s just important to be aware of those, to consider those as part of your assessment and to develop mitigation strategies for them and then finally, to also develop mitigation and recovery strategies in the event that there is an exposure or there is an incident that causes you to either lose the ability to provide service or expose data that should not have been publicly exposed.
Michael Attili
So the crux of it then becomes, given that information that you’ve developed in your assessment of the risks of moving some services or providing some services through a cloud system, is how do you validate it initially when you may not have full visibility into how the service is being conducted, and then also how you maintain that validated state. In a cloud environment, it helps to think about validation as a continuous process. In practice, it’s not, but the interval at which you confirm that your system remains in a validated state depends both on your risks and the types of actions that it can occur. And it’s likely not to be one single interval but for some types of aspects of your system, you’re going to want to do continuous monitoring. For others, you may do it on a hourly basis or a daily basis, and then others, you may just routinely specify a period of a week or a month or a year where you do a full revalidation just to ensure that nothing’s escaped, that you were covering in those other activities.
There’s a trade-off to be made in designing your system and designing your validation strategy that requires a balance between designing and resiliency, so anticipating what the types of failures are or the types of risks you may have in a cloud system and designing your system to be tolerant of that so that in the event that there are changes in network connectivity or latency or the underlying operating system, that your service isn’t affected. Balance that against the need for more frequent or real-time monitoring and verification at your system. It’s important to frame what it is, what does your validated state look like. And then that allows you to identify and build the testing and monitoring structures that you need and to figure out how frequently you are going to need to reestablish that.
Michael Attili
Given that you’ve now know what your validated state looks like and how to maintain it, then your change management practices need to figure out, okay, what types of modifications, what types of changes might happen at your vendor that you will need to consider as it changes on your quality system? There are changes that the cloud vendor might make that are completely invisible, not only invisible to you, but have no impact on the ability of you to provide a particular service. And if you can segregate those and say, “Okay. These set of changes are not going to have an impact and we don’t need to worry about them. We don’t need…” Not that we don’t need to worry about them, but we don’t need to have special controls to consider them, whereas other types will. And then for those types, you need to determine what level of notification you need in order to make sure that you have confidence that your service is still functioning properly.
So establishing with your service providers what types of notification you’ll get, how much advance notice and whether you have any influence or control to be able to say, “Yes, we’re willing to accept that change,” or “No, we want to keep our portion of the system the way it was running before.” And sometimes you may not have that option. Many times, you may not have that option. So you need to plan how to handle changes that your provider might make that fail verification or they’re not anticipated or approved by you, and then how to roll back or suspend or in the worst case, terminate the service, either switch to another mode or switch to another provider.
Michael Attili
Lastly, in your purchasing controls and supplier management, you need to make sure that first there exists suppliers and that they’re willing and able to meet your needs, that the functionality that they provide, the services that they provide, the internal change controls that they use and can commit to and can provide evidence to you that they follow, are satisfactory, that their level of availability is sufficient. No one is a hundred percent available, but what has been their history availability? What has been their history of security maintenance and disruptions? And establish that all in a service-level agreement that the supplier commits to, that will codify the changes that they can make without notice and what notice they will give you for other changes and then whether you have the option to accept or reject those changes. And with that, I think that summarizes what our intentions and our hopes are for this consensus report. Thank you.
Artiom Kisselev
Thank you, Mike. So as you can see here, this comic was created by the company CA Technologies, and there is this smiling guy about to drop his laptop out of the office tower window saying to the other guy, “Oh, come on, sir. I’m pretty sure there’s no need to monitor the transition of our workload to the cloud.” And I liked a lot this picture, which raises a question, “What could go wrong when we add the cloud to the mix for QMS and medical devices?”, or as Mike mentioned earlier, “What are the worst practices today?”. We wanted to see if we could get some answers to that by looking to the Adverse Event Reporting data bases from the US and a few different European countries.
Artiom Kisselev
And what we found is that first of all, the cloud is not involved in that much reports. When the cloud is involved, the adverse event is usually related to four kinds of events, synchronization issues, interrupted connection, maintenance mistakes and human errors. But the cloud-specific nature of these events isn’t really find-able or reported in the reports. Does that mean that nothing is happening? Certainly not, but there might be issues floating under the latter. But for now all these adverse events reported could have happened with a local server.
Tom Maeder
In about 30 seconds, we will begin a panel discussion moderated by Pat Baird from Philips. After that, there’ll be a Question-and-Answer session and remember to submit any questions that you might have through the question function on the Control Panel.
And don’t forget to download the Certificate of Attendance and a PDF of the presentations. And as I said, we will update, put the correct Rootstock presentation in as soon as possible.
Pat Baird
My name is Pat Baird. I work at Philips in lots of standards and regulatory sorts of things around software. And I’ve, over the past year or so, had the pleasure of working with these other panelists in developing some articles for publication, as well as the consensus report that we had mentioned, et cetera. So as we were putting together the agenda for today’s presentations, we thought it would be really nice to have just an open panel discussion about a couple of particular topics. And we can see if that would get attendees’ brain going and come up with questions for us. We’re going to have a big Q&A at the end of all of this. And we know there are some commonly asked questions that we’ve come across and we wanted to try to address it with the panel, as well as some other things, just to get people thinking.
Question #1: Are we really at the point of best practices?
Pat Baird
So I think I’d like to kick it off in talking about best practices I’m used when I’m writing a guidance document, when I’m writing internal procedure or working on a standard. It’s all about best practices. It’s all about how to be really good at what you’re doing. And what I want to ask the panel is because cloud still seem to be evolving a bit, are we there yet? Are we really at the point of best practices? Or are there things that people should be doing as compared to things that people really must be doing, shall be doing? Where is it? What do the panelists feel the current state of the art is? Is it mature and stable or are we going to have some more changes?
Bernhard Kappe
That’s a good question. I’ll take the first look at it.
So, I think one of the fundamental issues here is, in general, general computing is evolving really quickly and changing and continuing to mature and get more complex and more sophisticated and more reliable, but we’re getting less and less control. And that’s sort of the concentrate off that we’ve had, even before the Cloud came on board. It’s just accelerated over the last few years. So, it has gotten more mature, and it’ll continue to get more mature and more sophisticated and better over time. The challenge here is whatever best practices we have today in terms of specific mitigations and specific ways of analyzing it are probably going to be out of date in fairly rapid fashion.
Anyone else?
Clay Anselmo
I think from my perspective, I come at this from a quality and regulatory perspective, from compliance and submissions. And thinking about where we, where we land today, I do think there’s quite a bit of education and growth that needs to happen in the, let’s call it, regulatory science or quality science part of this. Right. And how do we take the models that we’ve been developing as computing technology has become sort of more separate from the device, in many cases, and SAMD has become a more prevalent part of what we do. How do we really extend what we’ve learned about risks that this kind of separation creates, and apply that to the Cloud in a more sort of robust way? Because again, as Bernard said, what we get here is great scalability.
We get great reliability, but we get a more opaque box, right? We don’t get to see inside of it very well. And that box changes a bit, depending upon the provider and your agreement. It can change a lot on a daily basis, or it can change very infrequently. And this brings a different paradigm to sort of designing devices for Cloud deployment, ensuring that you understand their external dependencies and how they affect the device, doing mitigations early. And then, how do we maintain a validated state once something is deployed out there, doing real-time probes and testing of the environment to give you confidence that it still behaves in a way that you expect. And that’s a little bit different. Not a lot different, but it’s a little bit different.
It’s an extension of our thought processes from before, but a level of opacity on that computing platform that you don’t … That we traditionally haven’t thought about it when everything was in one package. So, I think we’ve got a ways to go.
And I think … I’ll just be honest. I think the regulators are going to get more and more educated about some of the risks here as we make mistakes as an industry. And it pays to be a little ahead of the game and be thinking about if this is a way that your company is going to move, thinking about it ahead of time.
Bernhard Kappe
Yeah. Maybe let me give a sort of a quick illustration of kind of the tradeoffs that one might need to continue to make. Right? A number of years ago when people were looking at … As we are scaling up services, scaling down services, orchestrating services that we might use on the Cloud, as a way to mitigate those services malfunctioning, having some initial testing of the service, monitoring of the service, and a way to gracefully retire a malfunctioning service and replace that was a mitigation that you might need to do yourself.
And then eventually, we ended up getting some external software and tools that we could use for that, for what’s called basically self-healing services. But what’s happened now is that self-healing that I described as now part of the underlying infrastructure, it’s better than the mitigations that you could do yourself. So you can say, well, we can do these mitigations ourselves, but it’s not going to be as good as these other ones. So, do you do that and maintain control, or do you take these new and better, more integrated things and include them? That’s kind of the trade-off that I think we’re going to continue to have more and more of as the infrastructure gets more sophisticated.
Tom Brennan
If I could add, yesterday we were talking, it’s kind of changed where when I first got into cloud computing and it was actually not even called Cloud computing, but it was sort of this hosted idea. And there was sort of still all these layers you were dealing with, and you still sort of saw the separation of the layers, the technology stack that was there. But increasingly, like my example I showed you, it’s all kind of gone in the background. And now, it’s just the app that you see in a lot of cases. Not all cases, but in some cases that you see. So, it’s more like water coming through your house or electricity. It just is. And there are standards on what voltage it needs to be, and that there’s quality set up. The water isn’t tainted.
So, it’s gotten to be … It’s moving that way. And I can understand that people would be nervous for that, but I would say I used to show a diagram of a guy that was doing some do it yourself work at his house. And he rented a backhoe and he took it and he dropped it into the ground and he hit his gas line and blew up his house. And I think there’s some of this stuff now. You’re not going to be able to do it yourself as well as somebody like Salesforce or Amazon.
You’re just not, where you have to think about what business are you in, where you need to concentrate. And certainly, I understand you have to have the risk and the mitigation stuff. I understand that completely. But, they’re able to invest in a lot of that stuff far more in a one to many model and shared across many customers than you’ll be able to do yourself. And so, hopefully it gets more and more, and I’ve seen it in our space. People got more and more comfortable with it to the point where I don’t really question my water unless it tastes bad. I guess maybe that’s too late, but that’s the difference.
Question #2: Can we trust the cloud?
Pat Baird
One question that I had is can we trust the cloud? I know that people trust my spreadsheets, that they have high confidence that the calculations inside of Excel are working correctly. Are we at that point when it comes to cloud quality?
Tom Brennan
Well, I wouldn’t have picked spreadsheet for a trust to be perfectly honest with you. There’s a lot of … I think in the UK, there was a big thing on county COVID cases with spreadsheet errors. So, auditors love spreadsheets. When they see them, they dig in and find out what’s going on. So, I wouldn’t have used that maybe as an example, but we definitely have to earn our stripes. There’s no doubt about it as an industry, I would say. I can understand. And some people like spreadsheets because they can do whatever they want with them. That’s one of the things they like about them, and that is a strength of a spreadsheet, and they’re sort of ubiquitous. Anybody could use them. So, that’s great. But, we think a lot of things that are in spreadsheets should be on a platform that’s secured, that’s backed up, that has change control on it, that can be reported with other data, and seen, and has got tracking on field change, and all that stuff that real enterprise level apps have. So, I’m a little nervous about spreadsheets, frankly.
Bernhard Kappe
Actually, I think there’s actually a really good spreadsheet example. And this goes into the kind of … In some cases, because something is familiar, we make assumptions on it, and when really some of those we should validate to make sure that they’re correct, and that it’s fit for purpose. So, one of the things that was a bug in Excel was on certain types of mortgage type calculations, right? There’s a particular type of calculations that was incorrect in Excel over a number of years. And since Excel was used in a lot of mortgage processing kind of work, it turned out there were significant, very, very expensive errors that crept in that industry as a result of that. And it was only discovered after years. So just because something is familiar does not mean that it shouldn’t be validated. On the other hand, if you look at … There was a point in time when Excel was a brand new thing.
There was a time when mobile apps were a brand new thing, and certainly regularators for some early applications, as they were submitted, would ask questions, and return back and say, “Have you validated that someone can actually download this application off of the App Store?” And companies actually needed to go validate that. Those aren’t questions that are being asked anymore. So, some of this is because in this industry, the Cloud is relatively new. And so, there is going to be great … There are going to be greater questions about the details than there are with things that people are already more familiar with. You need to take the same level of rigor as you have with other things and not make the assumption that it’s somehow more dangerous, but also not that it’s less dangerous.
Nicola Zaccheddu
I think it’s really about indeed really understanding how we’re using that specific Cloud computing platform in your product. What is the role of that platform product in your process? What are the potential risks? Right?
So, it really starts with understanding the implication of using that platform on a Cloud, in a Cloud environment, and without the potential things that can happen that may be different from a non-Cloud software. And then, understanding what is the worst case scenario obviously, and the risks really related to that, and trying to find ways of, yes, validating the software as much as you can, but also have backup mechanisms that you can have in place in case something goes wrong, and you have no immediate correction that you can put in place. So, you need to move to a backup plan or to a different tool or to a safe mode for your device.
Josh Schulman
To add to what Bernhard and Nicola are was are saying, in general, the approach needs to account for situations where sometimes things that can go wrong, but it depends on what the impact of that going wrong is. And even though cloud is a new technology for medical devices, this is a technology, it’s still that principle, the underlying principle, of what can we predict and mitigate the impact of particular failures that we think of or don’t think of is certainly what’s driving the way we need to approach these types of things.
Question #3: Is the Cloud really different than some of the trends that are going on with other kinds of computing platforms?
Pat Baird
All right. Thank you all for that. There was a comment talking about some other technologies. So I’m curious, is the Cloud really different than some of the trends that are going on with other kinds of computing platforms? Cloud is new, but how different is it from some of the other things that are going on?
Bernhard Kappe
I would say it’s an extension of trends that have been going on for a very long time of more power and less control, right? At some point in the past, we had a sort of single hard drive that we were storing data in, and you maybe didn’t fully know where exactly things were being stored, but you knew it was going to be going to be on a particular hard drive. Right? Then at some point, well, there could be hard drive failure and failure to write, so why don’t we come up with something called array where the software is now controlling what data is being written, right?
And you actually don’t know where it is and how it’s striped across these arrays. We’ve gotten something that is actually better, but we have less control, and we have less knowledge of what’s happening inside of that. The same thing has happened with all of these other platforms. Mobile apps, you have far less control over when their operating system updates, all of those things. Those are just extensions of the same kind of more power, less control. The Cloud, and really this connected world I think that we are in with the Cloud, is an integral part of the ecosystem. Just accelerates that process. So, the question isn’t how do we validate something and how do we keep everything from changing. How do we deal with the change that is inherent?
That’s a problem that we could sort of avoid facing because it wasn’t happening that often, but in the connected world, it’s happening faster and faster and more in real time, frankly. So, that’s kind of the fundamental shift, I would say. It’s made this a front and center problem, as opposed to a problem that you could solve with inefficient processes
Josh Schulman
To add to that, one of the ways that this manifests itself is in testing, if something was less transparent or it’s just a different paradigm than we’re used to. So, it requires us to take a step back and say we’ve had this particular approach array, but maybe we really need to approach it a little bit differently, think a little bit outside of the box when it comes to these new technologies. So, I think that’s where the testing and validation discussions that we’ve been seeing and working through the issues around this project.
Tom Brennan
I think Bernard was kind of heading into it, Cloud’s sort of part and parcel a lot of things that are happening. Someone gets folded into this service. But, it’s become one of the links in the chain. When you look at mobile computing, what’s going on with that. So many Cloud services connected to it. And the reduction in size on computing and devices, things talking back to home, the Cloud’s become one of those links in there. And it just all seems to … They each seem to help each other, and rising tide floats all boats, and it seems to be doing that.
Bernhard Kappe
Yeah. And I think this is something that … It’s not just in medical devices, right? It’s really in individual’s lifestyle. It’s in how the world works, right? And so especially with a lot of the potential value and innovation that is happening in the medical device space, things that are either implanted, worn, or in-patient hands or connected to patients, these things need to be part of their lifestyles. And those lifestyles are mobile connected systems. There is no way of not addressing it and not dealing with it unless you’re in a very niche somewhere in the medical devices where you can stay unconnected.
Tom Brennan
Yeah. I think the bandwidth side has played a big role in this. When I first got into … Before Cloud, the big issue still was internet connectivity for people. And now it’s so much better than it was in the nineties, as an example. They used that as a timeline, and now we’re getting 5G and other things. So hopefully, that’s more pervasive and I’m sure some countries are still struggling in parts, some rural areas, even in developed countries. So, I think that’s another thing that’s making this whole thing pop. It’s not just the computing world. It’s also communications.
Bernhard Kappe
Yeah. I think that Nicola had mentioned kind of the scenarios and what your backups are. And I think that’s an important one to think about here. And I’d say almost abstracting a couple of things, right? One of them is there are risks inherent to distributed systems that you need to manage, whether the Cloud is part of a distributed system or not. And so then, one of the first questions is, could this be a distributed system, or if it’s a distributed system, what are our backups in case some aspect of that fails? A lot of those are things that are not specific to the Cloud.
They’re more specific to well connectivity using the internet, using cellular, et cetera, to different systems and subsystems that are distributed. And that’s the first set of questions that should be answered. And then, you take the Cloud as probably the most useful way of doing backend systems to have the sort of core central processing or communications hub. And then, you look at the specific Cloud related risks, which aren’t the same as the general distributed system ones. They’re really more on the lack of visibility in terms of what changes are happening and what’s going on underneath the hood as it were.
Question #4: Are there any differences between the major suppliers, or are they all the same?
Pat Baird
Now, I notice that we’ve been talking about the Cloud generically and not about any specific provider, and I don’t like to pick on or focus any one particular supplier, but I’m just curious what the panel thinks. Are there any differences between the major suppliers, or are they all the same?
Tom Brennan
I should probably stay out of that one, but go ahead. And I have a different angle to that probably, but I’d rather hear what the other folks say.
Bernhard Kappe
So, I haven’t done a thorough analysis of all the central Cloud vendors. Primarily, I’ve looked at and worked with three of them that you can imagine who they are. And I think this is one of the things where the Cloud has matured a lot. There were big differences, I would say, five, six, seven, eight years ago, but they’ve become less and less. A lot of the services that people end up using are more commoditized and available in a very similar fashion with different Cloud vendors. They’re all investing a lot of money in that, but there are areas of differentiation on new services or wrinkles to those services, all of which are also designed to lock you in to a particular Cloud vendor.
But even in those cases, just because someone has maybe done the certain things in AI that others haven’t, six months later they’ll have caught up in that. So it is very difficult to sort of treat it as a categorical, “These guys are good at this, these guys are good at this, these guys are good at that.” They’re all investing tons of money and sort of that changes over time.
Tom Brennan
What I was going to say, I think that there’s some excellent providers out there now and I do think the competition between them is really high. The differences are getting less and less, but one of the things that gets lost sometimes is to focus and this is going to sound incredibly self-serving. But you can label me that, okay. But what applications do you need to run the business? What is it? The app that’s going to talk to your device? Or what is the essential app that you really need to make your service happen? And to look at those applications, because I think there’s less and less differentiation at the lower levels of the stack as we call it and how it’s provided. There’s certainly some differences, don’t get me wrong, but I don’t know that they’re as important as the app that’s going to run what you’re trying to do.
And sometimes I think people kind of get caught up in the data center side of it a little more than what’s the app do and what are the controls around that. And all the questions that were brought up earlier about change management control and all that. At the app level that’s so, so important. And that’s what’s going to separate one business from another, is the apps. If you think about it we’re here talking about med devices, but you know what, we’re all really software companies. And I look at things and say, “What should these platforms that can allow you to be a better software company for what you do?”
And there may be an app on it, or it might be a set of tools that are there. And that which is what all the choices we made that we got with Salesforce may not be the right thing for your company, but there’s a set of tools that made sense for what we do there. That’s one of the things I’ve been stressed people to take a look at is those things. It’s the app and what is the essential thing of your business that you need to do well.
Nicola Zaccheddu
Yeah, absolutely. I also think indeed that besides the new, the functionality, the differentiators across different vendors. When you go to the medical world, it’s also extremely critical that you do your, of course, your own work on the supplier going to select. To have a look, what their history is in terms of quality of what they provide. And how transparent their processes are to you. How much you can trust the vendors that is going to be able to update, for example the tools you’re using without your consent. It’s always a good idea in general for suppliers. But especially when you’re dealing with cloud software that can change and the vendor is able to apply those changes without [inaudible 01:12:26] signs, you need to really be able to trust that vendor. So that’s kind of a very important thing when you’re selecting the supplier there.
Tom Maeder
I think this is a great discussion, but we’re running over time and we do need to have time for questions and answers and both fortunately, and unfortunately, we’ve got far more questions than we have time for. So let me run through some of them and ask you to be as succinct as possible in answering them so we can get to as many.
Question #5 Why wouldn’t we be more skeptical about the issues with validating the cloud?
Tom Maeder
The first one is sort of a big one as reticent as you all are with the validation of spreadsheets, which have been around since the 1980s, why wouldn’t we be more skeptical about the issues with validating the cloud? I’m extremely nervous about placing my software in the cloud.
Clay Anselmo
I guess from my perspective, I would probably say healthy skepticism is reasonable. This is a prove it, verify it industry, right? But what I think the panel would probably say is that in many cases, the benefits that we get, especially for the vast majority of our devices that are not high risk and do not have a lot of sort of DGL technical failure modes that relate to these computing platform dependencies. So there’s the 80 20 rule. There’s a lot of these things that this platform, the benefits far outweigh the risks.
But that doesn’t mean that there aren’t situations where either because of the architecture of the product, the real time or intended use is critical. And its susceptibility to some of these external dependencies might make it inappropriate. And I think a reasonable approach at the time you’re conceptualizing whether to deploy a device on the cloud is to think about those things and analyze them through risk analysis and see if you can mitigate them. So, it isn’t a one size fits all. And remember, if it’s a regulated device, you’re going to be asked to prove it, right. You’re going to be asked to prove that this thing is operating in a state of control in a validated state and the changes aren’t going to cause failure modes that are going to affect the safety or performance of the device. So I’m [inaudible 01:14:47] I just think that for the vast majority of devices, most of those risks can be mitigated.
Question #6: Is IEC 62304 a valid process to assess possible failures of cloud components?
Tom Maeder
Is IEC 62304 a valid process to assess possible failures of cloud components?
Bernhard Kappe
I think it’s a tool that you can use. So in particular how you handle kind of risk analysis with 62304 or how that applies software safety classification and software segregation. I think those are valid tools that you can use as part of that process. But anytime you get a system that gets more complex there’s a lot of devil in the details around that stuff. It’s not necessarily easy when you have a very complex system.
[At this point in the talk, there were two questions ERP in the cloud (not medical devices in the cloud) that were answered by Rootstock’s Tom Brennan.]
Question #7: A general question are cloud service providers more attractive to cyber attackers? And how do you address security hacking attempts targeted to cloud activities?
Tom Maeder
A general question are cloud service providers more attractive to cyber attackers? And how do you address security hacking attempts targeted to cloud activities?
Bernhard Kappe
I can take that one. I think the answer is yes. That has both advantages and disadvantages. One of them is because they are so big and clear targets. They see way more things than you would if you’re just posting your things in a data center. They see everything. They see the patterns. They also invest incredible amounts of money and effort to combat those things. So if you’re the biggest economy and the biggest military power, you are going to see more of those things than a small country is. You’re also going to be much better able to combat those things.
So I think again, this is another trade-off kind of thing. But they’re spending far more than any medical device company code to protect themselves and their customers because that’s their business. If they get hacked and if there are lateral types of issues that hurts their business, whether it’s a medical device company that’s using that or a financial services company or a school.
Question #8: What is your advice to a smaller mid-sized cloud service provider that wants to expand to the medical device market, but does not have the resources to go to a full blown ISO 1345 certification?
Tom Maeder
What is your advice to a smaller mid-sized cloud service provider that wants to expand to the medical device market, but does not have the resources to go to a full blown ISO 1345 certification? What are the top three things to implement that would be appealing to the medical device manufacturers?
Nicola Zaccheddu
So definitely I’ll say change management is one of the hot topic when we talk about cloud computing platform. Then I’ll say definitely a strong process around change control that you can show your users, how you control changes and how you verify and validate the changes before you just deploy them to your users. Definitely a problem resolution process. Like if your customer has an urgent issue that relates to you to the tool you’re providing them with, they need to be confident that they can reach out to you fast and you’re going to address their needs fast. Especially if they’re in a medical environment, then maybe safety related issues.
So you may need to give them priority and have a system there that can reassure them. You’re going to take seriously the things they’re going to report to you. And yeah, I’ll say maybe another thing you need to be open to share with them, those processes and make sure they have the gain, the confidence that they need. That you are a supplier that can be trust in their medical products. So I think being transparent and open to damage and letting them examine those things is also very important in my opinion.
Question #9: How often are updates pushed out? How long does revalidation take for those updates and how much work is involved?
Tom Maeder
How often are updates pushed out? How long does revalidation take for those updates and how much work is involved?
Bernhard Kappe
It is all over the place, frankly. And in a lot of cases, we really don’t have good visibility into that. So a lot of this is actually more dependent on individual services and how mature the surfaces are. Rather than it’s more service specific than vendor specific. So as with anything that is out there, that’s maintained by a large company. When you have something that’s been running very stably for a long time, and it’s very mature you probably have fewer updates to that than when it’s a brand new beta service let’s say.
Chances are you’re probably not going to be running your applications on using beta services that aren’t really fully mature yet. But that is one of the fundamental issues. In many cases, we don’t know what’s going on under the hood, in terms of those changes. Very often, we don’t have even the level of visibility that we have say with mobile operating systems where patches… We may not have control over the patches, but we do know when the patch happened.
Question #10: What should be the role of the service provider regarding client software validation required?
Tom Maeder
What should be the role of the service provider regarding client software validation required?
Clay Anselmo
I think it might be a really interesting service to provide as a third party to provide software validation within a cloud platform. But I think generally speaking, it’s not the service provider’s responsibility to conduct that validation. I think really from my perspective, it’s to support it. I think Nicola said it perfectly, this idea of transparency and change controls so that you can not only tell your customer if you’re a service provider, tell your customer what’s going to change and when. But that they can self-serve that. They can go out and see what’s going on. And in the ability to sort of categorize that, there are certain kinds of things that a particular product might care about. Maybe it’s libraries, maybe it’s some other kinds of processing availability, whatever. It can depend on the device, but that transparency, the ability to self-service monitor I think goes a long way. I wouldn’t expect the general service provider to provide validation services.
Tom Maeder:
Well, we have quite a few more questions, but I’m afraid we’re running out of time. And I would like to get to the wrap up by Joe Lewelling, who’s vice president of Emerging Technologies and Health IT at AAMI. And actually one of the questions, which Joe maybe you’ll address, somebody had asked whether there are any details on the consensus report available. So maybe you can give people an idea of where things go from here.
Joe Lewelling:
Sure, I would be glad to. First of all, before I start giving the details, I wanted to thank Veronica and everyone at MedTech Intelligence for putting on this webinar and allowing us to provide information to people, but also allowing you guys to know about the work we’re doing and what we’re aiming for. And also of course, I want to thank Pat and the entire pantheon of panelists and the task group members. And finally, I want to thank Tom and Rootstock for sponsoring it.
The AAMI Consensus Report is an initial consensus document that we are putting together to try to put some starting consensus around where we are on cloud computing and compliance issues. It is not the end of the road. It’s not really the start of the road either. The start is probably the article that as you heard will be published. The peer reviewed article that will be published next month in the AAMI Biomedical Instrumentation and Technology Journal. We do hope to have this consensus report, which involves the work of the task force to draft it. And then a review panel made up of experts from our quality systems activities. We hope to have that done by the first quarter of this year. About the end of the first quarter this year.
That’s always contingent upon there being consensus. We can’t produce a consensus report until we actually get consensus. But I do think we’re heading in the right direction. I’m very positive about that target and that will last at least 12 months, 12 to 24 months, and we’ll be constantly reviewing it.
And it’s very possible that it could lead to a more developed consensus document and AAMI technical information report. An American national standard or ultimately even an ISO IEC standard providing guidance or requirements around this issue.
If you’d like to follow the work and find out how you can be involved in the current work and future work, I would be happy to speak with you at [email protected].
And I would be happy to talk with you about this work and exactly what the process is. It’s a little more detailed than we have time for here. And also talk to you about the future plans on this, as well as other related issues like health software and IT systems and artificial intelligence and big data and more. So great. Thank you very much.
Tom Maeder
Well, thank you. And thank everyone we’re right at the end of the program. So again, I’m sorry that we didn’t get to all of the questions. If you want to communicate with this team or AAMI or learn more about these topics, there is contact information and more details on this slide.
Tom Maeder
So if there are questions that weren’t answered, I apologize that we didn’t have time. But we did get such a strong turnout of people for this program that we don’t want to keep you all day. So I do want to thank all of our speakers and panelists for today’s discussion. And I want to thank Rootstock again for sponsoring the program.
And to do a bit of self-promotion I should mention that we at MedTech Intelligence have a number of other virtual events during the coming months. We have a two-day virtual conference on computer modeling and simulation in MedTech product development and submissions on January 27th and 28th. And for those of you whose companies also manufacture diagnostics, we have scheduled a comprehensive EU IVDR implementation conference on February 24th and 26th. And details on these and other programs can be found on our website: https://medtechintelligence.com/
And I’m always interested in hearing about new topics for programs. So if there are other issues you would like to see discussed, please let me know. And I’m very happy to develop programs to address them.
We’re exactly at 2:30. So with that again, thank you everybody.
Related Posts
Talk
Roadmap for Generative AI Inside Medical Devices: Webinar
Talk
Evolving Your SaMD with Ongoing Releases: Webinar
Talk
Emerging AAMI Cloud Change Control Guidance: Webinar
