Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsLanguage Models & Transformers

ColBERT

proper nounAlso: ColBERTv2, Contextualized Late Interaction over BERT

In plain wordsA search model that compares a question and a document word by word — matching each question word to its best partner in the document — while still letting documents be processed ahead of time for speed.

  1. A neural retrieval model that encodes queries and documents separately with BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. into one contextual vector per token and scores a pair by late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast.: for each query token it takes the maximum similarity with any document token (MaxSim), then sums over the query tokens. Because document token vectors are computed offline, ColBERT approaches cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list. quality at a fraction of the cost and can search a whole collection with vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's. indexes rather than only re-rank a shortlist; its paper reported query processing two orders of magnitude faster, with about four orders of magnitude fewer FLOPs per query, than BERT-based rankers. Its drawback is storage — one vector per token — which ColBERTv2 cut 6–10-fold with residual compression. ColBERT-style scoring has since been extended to many languages and to images of document pages.

    ColBERT's token-level matches showed exactly which words of the passage drove its score.

s(q,d)=∑i=1∣q∣max⁡1≤j≤∣d∣qi⊤djs(q,d)=\sum_{i=1}^{|q|}\max_{1\le j\le |d|}\mathbf{q}_i^{\top}\mathbf{d}_j

Origin

Introduced by Omar Khattab and Matei Zaharia of Stanford University in 'ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT' (arXiv, April 2020; SIGIR 2020). ColBERTv2 (Keshav Santhanam, Khattab and colleagues, December 2021) added compression and denoised supervision.

Related

See also
  • late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast.
  • cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
  • bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
  • dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
  • rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
  • DSPyDSPy, prop. n.: A free software framework in which programmers describe what each step of an AI pipeline should do, and the framework automatically works out good prompts and examples instead of hand-writing them.

Sources

  1. Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. arXiv:2004.12832. arxiv.org/abs/2004.12832v1
  2. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C. & Zaharia, M. (2022). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022. arXiv:2112.01488. arxiv.org/abs/2112.01488

Mentioned in

cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first., late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast., MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models.