Good Machine Learning Practice (GMLP) for Medical Devices: A Framework for Safe AI in Healthcare
Artificial Intelligence (AI) and Machine Learning (ML) are transforming healthcare by enabling advanced diagnostic tools and predictive analytics that improve patient care. However, they also bring unique challenges, such as ensuring safety, effectiveness, and patient benefit. To address these challenges, Good Machine Learning Practice (GMLP) principles have been created.
In October 2021, the U.S. Food and Drug Administration (FDA), Health Canada, and the UK's Medicines and Healthcare products Regulatory Agency (MHRA) published 10 guiding principles for GMLP. These principles provide a framework for developing high-quality AI/ML medical devices and managing their complexities. By applying these principles, developers can ensure that AI-driven healthcare technologies are safe, reliable, and deliver real value to both patients and healthcare providers.
1. Multi-Disciplinary Expertise Throughout the Product Life Cycle
Developing an AI model that can impact patient care needs input from a wide range of experts: data scientists to build the algorithms, engineers to ensure its technical function, clinicians to offer real-world perspectives, and regulatory specialists to navigate compliance. Each brings something crucial to the table.
Take, for example, an AI system designed to detect cancer in medical images. Data scientists can build the algorithm, but clinicians are needed to address practical issues, like how the tool will be used effectively in a busy hospital. By collaborating, the team can create a device that is both accurate and useful for healthcare professionals. It is also very important to have the experts provide their inputs at the right stage of the process.
2. Implement Good Software Engineering and Security Practices
For AI in Medical Devices, following basic engineering and security steps is essential to keep the device safe and reliable. This means using good software design, ensuring the data is accurate, organising it well, and protecting it from security risks. Inputs from international standards such as IEC 62304, ISO 42001 helps in setting the base. Implementing security practices as early as possible by adapting to NIST and OWASP guidelines will ensure efficient development. Encryption, Authentication, Audit Trails etc are some good requirements that could be adapted to initiate building security.
3. Ensure Data Is Representative of the Intended Patient Population
Conducting clinical studies to prove efficacy of the data is very important in the overall development of AI Medical Devices. Data collection protocol should ensure that the study includes a diverse group of participants who represent the intended patient population, considering factors like age, gender, race, and ethnicity. The sample size should be large enough to provide reliable results. This approach helps ensure that the findings can be applied to the real patient population, reduces bias, and ensures the model works well for everyone.
4. Keep Training and Test Datasets Independent
Training and test datasets should be separate from each other. The data used to train the model should not overlap with the data used to test it. Any factors that could link the two datasets, like patient details or how the data was collected, should be carefully handled to keep them independent. Ensure appropriate data set versioning is implemented using tools such as DVC, Dagshub etc.
5. Use the Best Available Reference Data
The best methods for creating a reference dataset ensure that the data collected is clinically relevant and well-understood. These reference datasets should be used in model development and testing to show that the model is strong and can work well for the intended patient population. Ensure the reference data set is in line with the intended use and is from well-known sources.
6. Design Models Tailored to Available Data and Intended Use
When designing an AI medical device, it's important to align the model's architecture and functionality with the available data and its intended purpose. This alignment will help to manage potential risks such as overfitting, performance inconsistencies, and security vulnerabilities. The model's benefits and risks need to be clear such that performance goals can be set, ensuring it works safely and effectively. It's important to understand how AI in medical devices will perform in different scenarios, for various patient groups, and in different clinical settings.
7. Optimise Human-AI Collaboration
When there is a "human in the loop," the goal is to focus on how well the AI and the human work together, not just how well the AI medical device performs on its own. This means making sure that the AI's outputs are easy for the human to understand and use. Evaluate and address the usability aspects with respect to 'Human in the Loop'.
8. Test Under Clinically Relevant Conditions
Testing should show how the device works in a clinical setting. This means creating tests that don't rely on the AI's training data. The tests need to consider who the device is for (patients), how it will be used in a clinical setting, how doctors or healthcare workers will interact with the device, the measurements needed, and any factors that could affect the results. The goal is to make sure the device works well and safely in a clinical environment.
9. Provide Clear and Essential Information to Users
Healthcare providers and patients need to understand how an AI model works, including its strengths, limitations, and any potential risks. Clear communication is crucial to ensure the technology is used safely and effectively.
The relevant information may include the following:
- The product's intended use and indications for use
- Performance of the model for appropriate subgroups
- Characteristics of the data used to train and test the model
- Acceptable inputs
- Known limitations
- User interface interpretation
- Clinical workflow integration of the model
10. Monitor Deployed Models and Manage Re-Training Risks
Once the device is in use, it needs to be regularly monitored to ensure it continues to work safely and effectively in real-world conditions. If the model is updated or retrained, there should be checks in place to prevent problems like the model becoming too focused on old data, developing bias, or not adapting well to new data. These measures help make sure the model stays reliable and safe for the people using it.