WordPiece
In plain wordsGoogle's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.
A subword tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with. method created at Google, first for voice search and now best known as the tokenizer of BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.. Its vocabulary is built bottom-up from single characters, as in byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., but each merge is chosen by how much it improves a unigram language model of the training text rather than by raw pair frequency; in effect it favors pairs whose parts seldom occur apart. At inference, BERT's implementation segments each word by greedy longest-match-first lookup, marks word-internal continuation pieces with a '##' prefix, and maps anything it cannot segment to an unknown-token symbol. BERT's English vocabulary had about 30,000 WordPieces. Open-source trainers are reconstructions, since Google's own training code was never published.
The WordPiece vocabulary splits most chemical names into a dozen fragments.
Origin
Introduced by Mike Schuster and Kaisuke Nakajima for Japanese and Korean voice search (ICASSP 2012), then used as 'wordpieces' in Google's neural machine translation system (2016) and in BERT (2018).
Related
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
- SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
Sources
- Schuster, M. & Nakajima, K. (2012). Japanese and Korean Voice Search. ICASSP 2012.
- Wu, Y., Schuster, M., Chen, Z., Le, Q. V. et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144. arxiv.org/abs/1609.08144
Mentioned in
BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.