Skip to content
Last Dictionary
Plane
Language Models & TransformersSafety & AlignmentMathematics & Statistics

semantic entropy

nounAlso: semantic uncertainty

In plain wordsA way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing.

  1. A measure of a language model's uncertainty about what to say, as distinct from how to phrase it. Several answers are sampled for the same input and grouped into clusters whose members entail one another, as judged by a natural language inferencenatural language inference, n.: Deciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'? model or an LLM; the entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy. of the probability mass across clusters is then calculated. Paraphrases of one answer fall into a single cluster, so high semantic entropy indicates that the model is choosing among genuinely different answers — a signature of confabulationconfabulation, n.: When an AI fills a gap in its knowledge with an invented but plausible-sounding answer, much as some people with memory disorders sincerely 'remember' things that never happened.. It requires no labeled examples or task-specific training, but each query costs several extra generations; probes trained to predict it from hidden states have been proposed as a cheaper approximation.

    We show an answer only when semantic entropy across ten samples is below the threshold.

SE(x)=−∑cp(c∣x)log⁡p(c∣x),p(c∣x)=∑s∈cp(s∣x)\mathrm{SE}(x) = -\sum_{c} p(c \mid x) \log p(c \mid x), \qquad p(c \mid x) = \sum_{s \in c} p(s \mid x)

Origin

Introduced by Lorenz Kuhn, Yarin Gal and Sebastian Farquhar at the University of Oxford in 'Semantic Uncertainty' (February 2023; ICLR 2023), and applied to detecting confabulations by Farquhar, Jannik Kossen, Kuhn and Gal in Nature (June 2024).

Related

See also
  • confabulationconfabulation, n.: When an AI fills a gap in its knowledge with an invented but plausible-sounding answer, much as some people with memory disorders sincerely 'remember' things that never happened.
  • hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.
  • entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
  • abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.

Sources

  1. Kuhn, L., Gal, Y. & Farquhar, S. (2023). Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. ICLR 2023. arxiv.org/abs/2302.09664
  2. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630. doi.org/10.1038/s41586-024-07421-0

Mentioned in

abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake., confabulationconfabulation, n.: When an AI fills a gap in its knowledge with an invented but plausible-sounding answer, much as some people with memory disorders sincerely 'remember' things that never happened., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., natural language inferencenatural language inference, n.: Deciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'?