Artificial Intelligence in Healthcare: Challenges and Risks
Laura M. Cascella, MA, CPHRM
| Artificial Intelligence in Healthcare |
|---|
| Demand for AI technologies in healthcare is growing rapidly due to various factors, such as increasing expectations for personalized medicine and value-based care, expansion of digital information, healthcare provider shortages, and so on.1 Current applications of AI in healthcare include clinical decision support systems, surgical robots, ambient scribes, automated patient flow optimization, telehealth technologies, image analysis, drug development research techniques, and more. Researchers continue to study how various types of AI — such as machine learning, deep learning, neural networks, and natural language processing — can help improve diagnosis, treatment, patient experience, healthcare operations, public health initiatives, cybersecurity, and other aspects of healthcare. As newer forms of AI emerge, and as existing AI applications evolve, so too will regulations, standards of care, and best practices associated with their use. |
When envisioning the future of healthcare, artificial intelligence (AI) is a preeminent part of the picture. Daily stories trend in the media related to AI applications and their widespread potential for revolutionizing medical practice and patient care. Yet, akin to the promises of electronic health records in the early 21st century, the excitement surrounding AI has sometimes led to an idealistic view of its capabilities while marginalizing technological and operational challenges as well as safety and ethical concerns.2
As the healthcare industry explores using AI to augment, or even replace, the roles and responsibilities of human healthcare workers now and in the future, balancing utopian expectations with reality is imperative. As with any new technology, approaching AI with critical questioning and a healthy dose of caution can help prevent unfounded optimism, false confidence, and subsequent disillusionment.
Some of the major challenges and risks associated with AI that healthcare organizations should consider include:
- Biased data and functional issues. A major red flag associated with AI is the potential for bias. Bias can occur for various reasons; for example, the data that developers use to train AI applications and the rules they use to build algorithms might be biased. Additionally, bias might occur because of a variance in the training data or environment and how the AI program or tool is applied in real life. Read more about biased data and functional issues.
- Black-box reasoning. Many of today’s cutting-edge AI technologies — particularly machine learning systems that offer great potential for transforming healthcare — have opaque algorithms, making it difficult or impossible to determine how they produce results. This unknown functioning is referred to as “black-box reasoning” or “black-box decision-making,” and it presents concerns for patient safety, clinical judgment, and liability. Read more about black-box reasoning.
- Automation bias. Humans, by nature, are vulnerable to cognitive errors resulting from knowledge deficits, faulty heuristics, and affective influences/situativity. In healthcare, these cognitive missteps are known to contribute to medical errors and patient harm, particularly in relation to incorrect or delayed diagnoses. When AI is incorporated into clinical practice, healthcare providers might be susceptible to a type of cognitive error known as “automation bias.” Read more about automation bias.
- Data privacy and security. With the digitalization of health information, healthcare organizations and providers have faced growing challenges with securing increasing amounts of sensitive and confidential information while adhering to federal and state privacy and security regulations. AI presents similar challenges because of its dichotomous nature — it requires massive quantities and diverse types of data but is vulnerable to privacy and security issues. Read more about data privacy and security.
- Patient expectations. AI offers vast potential for improving patient outcomes through advances in population health management, risk identification and stratification, diagnosis, and treatment. Yet, even with this promise, questions arise about how patients will interact with and react to these new technologies and how these advances will change the provider–patient relationship. Read more about patient expectations.
- Training and education. The emergence of AI, its anticipated expansion into healthcare, and its massive scope point to significant training and educational needs for medical students and practicing healthcare providers. These needs go far beyond developing technical skills with AI programs and systems; rather, they call for a shift in the paradigm of medical learning. Read more about training and education.
In Summary
Healthcare is at the dawn of a digital revolution driven by the power and potential of AI. The opportunities for advancement span the healthcare spectrum and offer promises of optimized patient care and experience, streamlined clinical and operational processes, workforce solutions and efficiencies, advances in cybersecurity, and more.
Amidst the fervor for AI are pragmatic calls to approach these technologies with measured caution and optimism. Although AI will undoubtedly have a major impact on healthcare, it is still relatively new and rapidly changing. Healthcare organizations implementing AI systems and programs — and healthcare providers incorporating these technologies into daily practice — should be aware of AI’s capabilities, limitations, and potential risks.
For more information about this topic, see MedPro’s Risk Resources: Artificial Intelligence.
Endnotes
1 Grandview Research. (2025). AI in healthcare market summary. Retrieved from https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-healthcare-market
2 Chustecki, M. (2024). Benefits and risks of AI in health care: Narrative review. Interactive Journal of Medical Research, 13, e53616. doi: https://doi.org/10.2196/53616
TOOLS & RESOURCES
- Articles
- Booklets
- Checklists
- Claims Data by Specialty
- Claims Data by Topic
- Guidelines
- Product Catalog
- Reference Manual
- Resource Lists
- Risk Management Review
- Risk Q&A
- Risk Tips
- Sample Dental Informed
Consent Forms - Sample OMS Informed
Consent Forms - Senior Care Resources from
Pendulum
