Artificial Intelligence Risks: Automation Bias

Laura M. Cascella, MA, CPHRM

The concept of bias in relation to artificial intelligence (AI) usually is discussed in terms of biased data and algorithms, which pose significant ethical and safety issues. However, another type of bias also raises concern. Automation bias occurs when “clinicians accept the guidance of an automated system and cease searching for confirmatory evidence . . . perhaps transferring responsibility for decision-making onto the machine . . .”¹ Similarly, clinicians who use generally reliable technology systems might become complacent and miss potential errors, particularly if they are pressed for time or carrying a heavy workload.

Automation bias might result from risk homeostasis, a theory that suggests that individuals adjust their behavior based on their perceived level of risk. The less safe an activity seems, the more careful a person will be and vice versa. If AI introduces a perceived level of accuracy or infallibility, clinicians might be more likely to accept incorrect or suboptimal recommendations (errors of commission) or fail to act entirely without automated guidance (errors of omission).² For example, a recent study of endoscopists found that their ability to spot polyps and other abnormalities during colonoscopies decreased after they had grown accustomed to using an AI system for polyp detection.³

In light of the black-box issues associated with AI, automation bias is particularly concerning if providers rely too heavily on these technologies without having a clear indication of how they work, how the end results are produced, and the probability of their accuracy. Further, disease patterns and standards of care change over time. AI can become less effective if it does not receive and adapt to updated data, posing patient safety threats if providers fail to recognize these discrepancies.

Failure to identify and address errors of commission and omission that occur because of automation bias and complacency can perpetuate these issues and lead to patient harm and the erosion of clinicians’ clinical judgment and decision-making skills.

Unfortunately, automation bias — much like other cognitive biases — does not have a simple, universal solution. It will likely require a combination of strategies, such as providing ongoing education to raise awareness, encouraging critical thinking skills, using team-based approaches to care, finding novel ways to engage patients and families, experimenting with debiasing techniques, and implementing other best practices as they are identified.

As an article in Forbes notes, “The real promise of AI lies in its ability to improve human decision-making. By implementing strategies that emphasize the complementary roles of humans and machines, organizations can enhance decision-making processes, ensuring that they are both efficient and resilient.”⁴

To learn more about other challenges and risks associated with AI, see MedPro’s article Artificial Intelligence in Healthcare: Challenges and Risks.

Endnotes

¹ Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019, March). Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28(3), 231-237. doi: 10.1136/bmjqs-2018-008370

² Gretton, C. (2017, June 24). The dangers of AI in health care: risk homeostasis and automation bias. LinkedIn. Retrieved from www.linkedin.com/pulse/dangers-ai-healthcare-risk-homeostasis-automation-bias-gretton-md/

³ Budzyń, K., Romańczyk, M., Kitala, D., Kołodziej, P., Bugajski, M., Adami, H. O., Blom, J., . . . Mori, Y. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: A multicentre, observational study. The Lancet. Gastroenterology & Hepatology, S2468-1253(25)00133-5. doi: https://doi.org/10.1016/S2468-1253(25)00133-5

⁴ Hoffman, B. (2024, March 10). Automation bias: What it is and how to overcome it. Forbes. Retrieved from https://www.forbes.com/sites/brycehoffman/2024/03/10/automation-bias-what-it-is-and-how-to-overcome-it/

---

Prevention & Education Center home