two-tower model
In plain wordsA design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison.
A neural architecture for large-scale retrieval in which two separate networks (towers) map the query side, such as a user and their context or a search query, and the candidate side, such as an item or document, into embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. in a shared vector space, scoring relevance by dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles. or cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.. Because candidates are encoded independently of the query, every item embedding can be precomputed and indexed, so retrieval from millions or billions of items reduces to approximate nearest neighbor searchapproximate nearest neighbor search, n.: A family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster. taking milliseconds. The towers are trained on logged interactions with a contrastive or sampled-softmax loss, commonly using the other items in a mini-batch as negatives, with corrections for the popularity bias that this sampling introduces. The price of speed is that query and item features never interact before the final score, so two-tower models usually serve as the candidate-generation stage of a recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on., followed by a ranking model that can use cross features. In text retrieval the same design is called a bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash. or dual encoder.
The two-tower model retrieves 500 candidates in under ten milliseconds before the ranker takes over.
Origin
Descends from Microsoft's Deep Structured Semantic Model (DSSM) for web search (Po-Sen Huang and colleagues, 2013), which encoded queries and documents with separate networks. YouTube's 2016 deep candidate-generation model and a 2019 Google paper by Xinyang Yi and colleagues on correcting sampling bias helped make 'two-tower' the standard industry term.
Usage note
Contrast cross-encoderscross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list. and ranking models, which score each query–item pair jointly: more accurate, but far too slow to apply to a whole catalog.
Related
- recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
- bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
- approximate nearest neighbor searchapproximate nearest neighbor search, n.: A family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
- dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
- matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together.
- cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
Sources
- Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A. & Heck, L. (2013). Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. Proc. CIKM 2013.
- Covington, P., Adams, J. & Sargin, E. (2016). Deep Neural Networks for YouTube Recommendations. Proc. RecSys 2016.
- Yi, X. et al. (2019). Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. Proc. RecSys 2019.
Mentioned in
recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on., bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash., click-through rate predictionclick-through rate prediction, n.: Estimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., generative recommendationgenerative recommendation, n.: A newer approach to recommendation that uses language-model-style AI to predict the next item a person will want, one step at a time, much as a chatbot predicts the next word.