Avoiding Overreliance on Artificial Intelligence in Healthcare

Laura M. Cascella, MA, CPHRM

In the not-too-distant past, the concept of artificial intelligence (AI) seemed abstract and futuristic. Even 5 years ago, the power and potential of AI were largely unknown to the general population, and its implementation was only just beginning. Fast forward, and AI is a burgeoning reality across many industries, including healthcare.

A Health Affairs study found that about 65 percent of U.S. hospitals are using AI-assisted predictive models, and an American Medical Association/Center for Digital Health and AI survey found that 81 percent of physicians are using AI for various tasks, such as charting, billing, patient summaries, translation, diagnosis, and more.¹

As AI becomes more mainstream in healthcare, concerns are arising about the issues and risks that might accompany these technologies — such as bias, errors, misinformation, privacy violations, and ethical dilemmas. One question in abundance is how AI will change the role and purpose of healthcare professionals. Will it replace clinicians, usurp their authority, and/or degrade the provider–patient relationship?

Perhaps even more concerning is whether AI will result in an overreliance on technology (automation bias) that prevents healthcare providers from recognizing when these technologies are making errors, failing to account for nuance or context, undergoing distributional shifts, or experiencing “model collapse.” These concerns are not unwarranted; a 2026 survey found that 74 percent of clinicians fear overreliance on AI.²

If healthcare providers fall prey to automation bias, their clinical judgment and reasoning skills may deteriorate (deskilling), which could lead to misdiagnosis, poor treatment decisions, perpetuation of health disparities, and other negative consequences with far-reaching effects. An article in AI Ethics explains that automation bias may result in clinical dangers, including “eroded vigilance, impoverished therapeutic relationships, and potentially poorer outcomes regarding overall well-being.”³

Preventing these issues necessitates managing expectations around AI and recognizing that, although the technology is transformative, it is not infallible (nor does it have the same capabilities as humans). Strategies that can help healthcare organizations and providers mitigate risks associated with automation bias and deskilling include the following:

• Develop AI governance policies that include clear expectations related to using AI tools, monitoring for potential quality and safety issues (including how monitoring will occur), and auditing for compliance with organizational policies and procedures.
• Establish a human-in-the-loop approach to AI that requires careful oversight of AI systems and validation of AI outputs. “A human-in-the-loop approach can ensure personal sensitivities are accounted for and patient preferences are best met. Clinicians’ expertise steeped in experience may be nearly irreplaceable for appropriately including idiosyncratic human factors in medical decision making.”⁴
• Make sure that AI tools are used only for their intended purposes and not to perform functions outside of the scope that the organization has specified. For example, if the organization permits use of large language models to gather information on general medical queries, providers should not use them to input specific patient data for diagnostic or treatment purposes.
• Consider implementing cognitive forcing functions into workflows to encourage critical thinking and prevent faulty heuristics that may lead to overreliance on AI tools. Cognitive forcing functions are designed to help clinicians think analytically and self-monitor decisions to avoid errors. Examples include implementing diagnostic timeouts, using checklists, and asking providers to explicitly rule out alternatives.
• Consider other approaches that may help reduce reliance on AI and avoid complacency, such as delaying the presentation of AI recommendations to give clinicians time to think or having providers make initial interpretations and decisions prior to seeing AI guidance.
• Establish thorough procedures to handle situations in which AI tools and systems are not available (e.g., during outages, natural disasters, or cyberattacks). Make sure providers and staff receive comprehensive training on these procedures and have the necessary skills to convert to backup processes. ECRI has listed “unpreparedness for a digital darkness event” as one of its top 10 health technology hazards for 2026.⁵
• Educate healthcare providers about automation bias, subsequent deskilling, and the potential outcomes. Make sure providers are aware of situations in which AI might not perform optimally (e.g., when training data do not align with the patient population, when patients have rare or complex conditions, when subtle variations in context occur, or when health trends shift over time).
• Provide training and mentorship opportunities, particularly for newer clinicians, to help foster clinical judgment and reasoning skills. “Senior physicians who openly question AI recommendations demonstrate that thoughtful skepticism is an expected part of safe clinical practice.”⁶ Consider a range of training formats including simulation scenarios, reflective practice, case studies, debriefings, and more.
• Emphasize the importance of soft skills — such as thoughtful communication, emotional intelligence, and empathetic listening — as AI becomes more ubiquitous. These skills can help preserve the provider–patient relationship and prevent clinicians from elevating their machine interactions above their human connections.⁷

Automation bias and clinician deskilling are legitimate concerns for healthcare professionals, organizations, and patients. When these issues occur, they may not be immediately apparent, but their consequences could be disastrous. Addressing overreliance on AI requires a careful approach to the technology that appreciates its abilities, recognizes its limitations, and vigilantly scrutinizes its outputs.

Further, taking deliberate steps to preserve the uniquely human aspects of healthcare — such as the ability to understand nuance and context, offer compassion, and show empathy — will help maintain trust, uphold the ethical principles of medicine, and support optimal care delivery.

For more information on various topics associated with AI, see MedPro’s Risk Resources: Artificial Intelligence.

Endnotes

¹ Nong, P., Adler-Milstein, J., Apathy, N. C., Holmgren, A. J., & Everson, J. (2025). Current use and evaluation of artificial intelligence and predictive models in US hospitals. Health Affairs, 44(1), 90–98. doi: https://doi.org/10.1377/hlthaff.2024.00842; American Medical Association & Center for Digital Health and AI. (2026, March). 2026 physician survey on augmented intelligence. Retrieved from www.ama-assn.org/system/files/physician-ai-sentiment-report.pdf

² Wolters Kluwer. (2026). Patients, doctors, and nurses on AI: Similar tools, different pathways, one destination. 2026 Future Ready Healthcare Survey Report. Retrieved from www.wolterskluwer.com/en/know/future-ready-healthcare

³ Saadeh, M. I., Janhonen, J., Beer, E., Castelyn, C., & Hoffman, D. N. (2025). Automation complacency: Risks of abdicating medical decision making. AI Ethics 5, 5783–5793. doi: https://doi.org/10.1007/s43681-025-00825-2

⁴ Ibid.

⁵ ECRI. (2026). Executive brief: Top 10 health technology hazards for 2026. Retrieved from https://home.ecri.org/blogs/ecri-thought-leadership-resources/top-10-health-technology-hazards-for-2026-executive-brief

⁶ Siwicki, B. (2026, July 15). New AI risks: “Cognitive spoofing” and fake expertise. Healthcare IT News. Retrieved from www.healthcareitnews.com/news/new-ai-risks-cognitive-spoofing-and-fake-expertise

⁷ Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5, CSCW1, Article 188, 1–21. doi: https://doi.org/10.1145/3449287; McBride, B. S., & Menack, S. (2025, July 14). AI in healthcare: Opportunities, enforcement risks and false claims, and the need for AI-specific compliance. Morgan Lewis. Retrieved from www.morganlewis.com/pubs/2025/07/ai-in-healthcare-opportunities-enforcement-risks-and-false-claims-and-the-need-for-ai-specific-compliance; Korkmaz, S. (2024). Artificial intelligence in healthcare: A revolutionary ally or an ethical dilemma? Balkan Medical Journal, 41(2), 87–88. doi: https://doi.org/10.4274/balkanmedj.galenos.2024.2024-250124; Wolters Kluwer. (2026, January 27). AI survey insights: Newer providers concerned about deskilling. Retrieved from www.wolterskluwer.com/en/expert-insights/ai-survey-insights-newer-providers-concerned-about-deskilling; Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., Bauer, R., & Bauer, M. (2026). Artificial intelligence and deskilling in medicine. The British Journal of Psychiatry, 1–3. Advance online publication. doi: https://doi.org/10.1192/bjp.2025.10496; Saadeh, et al., Automation complacency: Risks of abdicating medical decision making.

---

Prevention & Education Center home