Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsDeep Learning

Matryoshka embedding

noun/mat-ree-OSH-kuh/Also: Matryoshka representation learning, MRL, nested embedding, truncatable embedding

In plain wordsAn embedding built like a set of Russian nesting dolls: its first few numbers already give a rough summary, so you can cut it short to save space and still search reasonably well.

  1. An embedding vector trained so that its leading dimensions form usable lower-dimensional embeddings on their own: the first 64, 128, 256 … dimensions are each a coarse-to-fine version of the full vector. Matryoshka representation learning achieves this by applying the training loss to several nested prefixes at once and summing the results, which pushes the most important information into the earliest dimensions at little cost to full-size quality. Users can then truncate (and re-normalize) vectors to trade accuracy for storage and speed, or use short prefixes for a fast first pass and full vectors to re-score the shortlist, sometimes called adaptive retrieval. The original paper reported representations up to 14 times smaller at equal ImageNet accuracy and up to 14 times faster large-scale retrieval.

    We keep 256-dimension Matryoshka prefixes in memory and the full 3,072-dimension vectors on disk for re-scoring.

LMRL=∑m∈Mcm L(z1:m),M={8,16,32,…,d}\mathcal{L}_{\mathrm{MRL}}=\sum_{m\in\mathcal{M}} c_m\,\mathcal{L}\big(\mathbf{z}_{1:m}\big),\qquad \mathcal{M}=\{8,16,32,\dots,d\}

Origin

Introduced by Aditya Kusupati, Gantavya Bhatt, Aniket Rege and colleagues in 'Matryoshka Representation Learning' (arXiv, May 2022; NeurIPS 2022). OpenAI's text-embedding-3 models (January 2024) were trained with the technique and exposed a dimensions parameter for shortening embeddings, and many open embedding models followed.

Related

See also
  • text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
  • embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are.
  • vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's.
  • product quantizationproduct quantization, n.: A compression trick for embeddings: chop each long list of numbers into short pieces and replace each piece with the ID of its closest typical piece, shrinking storage many times over while keeping search workable.
  • dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands.
  • approximate nearest neighbor searchapproximate nearest neighbor search, n.: A family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster.

Sources

  1. Kusupati, A. et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. arXiv:2205.13147. arxiv.org/abs/2205.13147v4
  2. OpenAI (2024). New embedding models and API updates, 25 January 2024. openai.com/index/new-embedding-models-and-api-updates

Mentioned in

dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands., embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are., text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.