The Stack
In plain wordsA large free collection of computer source code from public repositories, selected mainly for permissive licenses and released from 2022 onward by the BigCode project for training coding models.
An open dataset of source code for training code-generatingcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs. language models, built by BigCode, an open research collaboration led by Hugging Face and ServiceNow. The first release (v1.0, 2022) held 3.1 TB of files in 30 programming languages from public GitHub repositories whose detected licenses are permissive (such as MIT and Apache 2.0) — plus, in that release only, three weak-copyleft licenses — and excluded GPL-style and unlicensed code; v1.1 grew to 358 languages and about 6 TB. It was near-deduplicated and paired with a governance process in which developers can search for their code ('Am I in The Stack') and request its removal from future versions. The Stack v2 (2024), built with the Software Heritage archive, spans more than 600 languages, also admits files with no detected license, and yielded a training set about four times larger than the first StarCoder's; it trained the StarCoder2 models (3, 7 and 15 billion parameters) on 3.3–4.3 trillion tokens.
We removed repositories whose owners had opted out of The Stack before training.
Origin
Introduced in 'The Stack: 3 TB of Permissively Licensed Source Code' by Denis Kocetkov, Raymond Li, Loubna Ben Allal and colleagues (arXiv, November 2022); The Stack v2 appeared with StarCoder2 in February 2024 (Anton Lozhkov, Raymond Li, Loubna Ben Allal and colleagues).
Usage note
Its license filter and opt-out made it a model for consent-aware data collection, but permissive licenses still require attribution and license detection is error-prone, so legal questions about code models remain.
Related
- code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs.
- opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in.
- data licensingdata licensing, n.: The legal permissions that say who may use a dataset or a body of content, for what purposes and on what terms — including whether it may be used to train AI.
- deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online.
- Hugging FaceHugging Face, prop. n.: A company whose website is the main public library for sharing AI models and datasets — often called the GitHub of machine learning — and which makes widely used free AI software.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
Sources
- Kocetkov, D. et al. (2022). The Stack: 3 TB of Permissively Licensed Source Code. arXiv:2211.15533. arxiv.org/abs/2211.15533
- Lozhkov, A. et al. (2024). StarCoder 2 and The Stack v2: The Next Generation. arXiv:2402.19173. arxiv.org/abs/2402.19173
Mentioned in
opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in.