Insufficient Governance of AI Used in Medical Technologies Risks Inappropriate Care Decisions
January 29, 2024 | Evaluations & Guidance
Artificial intelligence (AI) functionality is being incorporated into a wide range of devices and systems in healthcare—from workflow aids (e.g., automated scheduling software) to diagnostic support systems that interpret medical images or analyze a patient's medical record. AI offers the promise of speeding up processes and assisting in clinical decisions, with the goal of improving patient safety and clinical outcomes.
But there are risks: AI systems are only as good as the algorithms they use and the data on which they are trained. Shortcomings in either area can lead to inappropriate patient care decisions, impacting patient safety. For instance, an algorithm that places too much weight on one factor, or not enough weight on another factor, could reach an inappropriate conclusion. Similarly, AI algorithms that are trained on data derived from one group of...