overreliance
In plain wordsLeaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong.
Reliance on an AI system's outputs beyond what its actual reliability warrants, so that users accept errors they could have caught. It is the behavioral result that automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them. and miscalibrated trust produce, and is usually defined against 'appropriate reliance': accepting the AI's output when it is right and rejecting it when it is wrong. With large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., fluent and confident prose, hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. and sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful. make overreliance easy, and explanations can increase it when users read any explanation as a sign of competence. Mitigations studied include cognitive forcing (asking users to decide before seeing the AI's answer), communicating uncertainty, showing sources and designing for easy verification. Sustained overreliance can also contribute to deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed..
Reviewers approved nearly every AI-drafted summary, errors included, a clear sign of overreliance.
Origin
Long used in the human-automation literature, the term became standard in human–AI interaction research around 2020–2022; OpenAI's GPT-4 system card (March 2023) listed overreliance among the risks of language models, and it has since been a routine heading in model risk documentation.
Usage note
The opposite failure is under-reliance, including algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.; both are failures to match trust to trustworthiness. Overreliance concerns the user's behavior, whereas hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. and sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful. are properties of the model that invite it.
Related
- automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
- algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.
- deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
- sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
- trustworthy AItrustworthy AI, n.: AI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust.
- human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked.
- algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.
Sources
- Buçinca, Z., Malaya, M. B. & Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proc. ACM on Human-Computer Interaction 5 (CSCW1).
- Passi, S. & Vorvoreanu, M. (2022). Overreliance on AI: Literature Review. Microsoft Research.
Mentioned in
loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., advanced driver-assistance systemadvanced driver-assistance system, n.: Car features that help the driver but do not replace them, such as automatic emergency braking, lane-keeping help, adaptive cruise control and blind-spot warnings., AI literacyAI literacy, n.: Knowing enough about AI to use it sensibly and judge it critically: what it can and cannot do, how it goes wrong, and how it affects people., automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them., deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed., human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked., jagged frontierjagged frontier, n.: The uneven edge of what AI can do: it may excel at some hard tasks yet fail at seemingly easier ones, so how hard a task looks is a poor guide to whether AI can handle it., trustworthy AItrustworthy AI, n.: AI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust., workslopworkslop, n.: AI-generated work that looks polished but does not actually get the job done, quietly passing the real effort to whoever receives it, who must then puzzle it out or start over., algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster., botshitbotshit, n.: A blunt term for false or made-up chatbot output that people pass on and use as if it were true, especially at work., cognitive debtcognitive debt, n.: The idea that letting AI do your thinking or writing saves effort now but weakens your own thinking, memory and learning over time.