Skip to content
Last Dictionary
Plane
Core ConceptsInterpretability & SteeringGovernance, Policy & LawSociety, Ethics & Economy
figurative

black box

nounAlso: black-box model, black-box AI, black-box access, opacity

In plain wordsA system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.

  1. 1

    AI

    A model or system whose internal decision process cannot be inspected or understood by the people who use, oversee or are affected by it. The sociologist Jenna Burrell distinguished three sources of opacity: deliberate corporate or state secrecy, technical illiteracy among those affected, and the intrinsic complexity of models such as deep neural networksdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., whose billions of parametersparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. encode no humanly readable rules. Opacity hampers debugging, challenges to automated decisions and safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable. assurance. Responses include inherently interpretable modelsinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed., post hoc explanationexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application. methods, mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., documentation, transparency obligationstransparency obligations, n.: EU rules requiring that people be told when they are dealing with AI: chatbots must say they are bots, AI-made content must be marked, and deepfakes must be labeled. and audits. In evaluation, black-box access means querying only a model's inputs and outputs, as opposed to white-box access to its weights and activations.

    Regulators objected that the credit-scoring model was a black box even to the bank that deployed it.

  2. 2

    Engineering and testing

    Any device or system treated purely as a relation between what goes in and what comes out, its internal mechanism unknown or deliberately set aside, as in cybernetics, system identification and black-box software testing, which checks behavior against a specification without reading the code.

Origin

An engineering and wartime-electronics term of the 1940s for sealed units, given theoretical standing in cybernetics by W. Ross Ashby, whose An Introduction to Cybernetics (1956) has a chapter on the black box. Critics applied it to neural networks from the 1990s; Frank Pasquale's The Black Box Society (2015) and Burrell's 2016 article made it central to debates on algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it..

Usage note

A system can be a black box for legal reasons (trade secrets) while technically simple, or technically opaque while fully published, as with open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.. The aviation 'black box' (flight recorder) is an unrelated sense.

Related

See also
  • interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
  • explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
  • mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
  • AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision.
  • algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it.
  • right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.
  • interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
Contrast with
  • interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.

Sources

  1. Burrell, J. (2016). How the Machine 'Thinks': Understanding Opacity in Machine Learning Algorithms. Big Data & Society, 3(1).
  2. Pasquale, F. (2015). The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press.
  3. Ashby, W. R. (1956). An Introduction to Cybernetics. Chapman & Hall.

Mentioned in

deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.