SentencePiece
In plain wordsFree software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.
An open-source tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. library from Google that learns and applies subword vocabularies straight from untokenized sentences. It makes no assumption that words are separated by spaces: the space is stored as an ordinary visible symbol ('▁') inside the pieces, so the original string can always be restored exactly, and languages written without spaces, such as Japanese, Chinese and Thai, need no separate word splitter. Two training algorithms are offered — byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size. and the unigram language-model method, which starts from a large candidate vocabulary and prunes the pieces whose removal least reduces corpus likelihood — and the library can sample alternative segmentations for subword regularization. A byte-fallback option encodes unknown characters as UTF-8 bytes. It has been used by T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design., LLaMA 1 and 2, Gemma and many multilingual models.
We trained a unigram SentencePiece model with byte fallback so no character maps to unknown.
Origin
Released by Taku Kudo and John Richardson of Google in 2018 (EMNLP 2018 system demonstrations), building on Kudo's paper of the same year introducing the unigram-model subword algorithm and subword regularization.
Usage note
SentencePiece is a toolkit, not a separate algorithm: a 'SentencePiece tokenizer' is a unigram or BPE model trained with it.
Related
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
- WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
Sources
- Kudo, T. & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018 (System Demonstrations). arxiv.org/abs/1808.06226
- Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. arxiv.org/abs/1804.10959
Mentioned in
byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.