Data & DatasetsGovernance, Policy & LawSociety, Ethics & Economy

datasheet for datasets

nounAlso: datasheets for datasets, datasheet, dataset card, data card

In plain wordsA standard document that travels with a dataset, describing why it was made, what is in it, how it was collected and labeled, and what it should and should not be used for.

A documentation template for datasets, named by analogy with the specification sheets supplied with electronic parts: a fixed questionnaire that the dataset's creators fill in, organized in seven sections that run from why the dataset was built and what it contains, through how it was collected, cleaned and labeled, to the uses it suits or does not suit and how it is distributed and maintained. The questions prompt creators to record who funded the work, what each instance represents, whether the data contain personalpersonally identifiable information, n.: Information that can identify a specific person, such as a name, address, phone number or ID number — or several facts that together point to one person. or sensitive information, whether consent was obtained, how representative the sample is, known errors and redundancies, and who maintains the data. The aim is to make tacit choices explicit, so that users can judge fitness for purpose and creators weigh risks before release. Datasheets inspired Hugging Face dataset cards and Google's Data Cards, and similar documentation duties appear in regulation, such as the EU AI Act's technical-documentation requirements for high-risk systems.
The datasheet's collection section revealed that all the speech was recorded in just two cities.

Origin

Proposed by Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford in 'Datasheets for Datasets' (arXiv, March 2018; Communications of the ACM, 2021). Related 2018 proposals include Emily Bender and Batya Friedman's data statements for NLP and the Dataset Nutrition Label.

Usage note

Datasheets document data; model cardsmodel card, n.: A short document published with an AI model explaining what it is for, how it was trained and tested, how well it works for different groups of people, and its known limits. document trained models; system cardssystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place. document deployed systems and their safety evaluations. Adoption is uneven, and retrospective datasheets, such as one for BookCorpusBookCorpus, prop. n.: A collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later., have exposed problems in widely used datasets.

Sources

Gebru, T., Morgenstern, J., Vecchione, B., Wortman Vaughan, J., Wallach, H., Daumé III, H. & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. arxiv.org/abs/1803.09010
Bender, E. M. & Friedman, B. (2018). Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the ACL 6.

Mentioned in

datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision., data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system., Emily M. BenderEmily M. Bender, prop. n.: An American linguist who argues that language models only mimic the form of language without understanding it, and who co-coined the phrase 'stochastic parrots'., model cardmodel card, n.: A short document published with an AI model explaining what it is for, how it was trained and tested, how well it works for different groups of people, and its known limits., reproducibilityreproducibility, n.: Whether other people, or the same people later, can get the same results by rerunning an AI experiment. Many published AI results have proved hard to reproduce, which makes real progress harder to judge., responsible AIresponsible AI, n.: An umbrella name for the practices organizations use to keep their AI systems fair, safe, private and accountable, from the first design meeting to everyday use., system cardsystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place., Timnit GebruTimnit Gebru, prop. n.: An Ethiopian-born American computer scientist who exposed racial and gender bias in AI systems, co-wrote the 'stochastic parrots' paper, and founded an independent AI research institute., BookCorpusBookCorpus, prop. n.: A collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later., Kate CrawfordKate Crawford, prop. n.: An Australian scholar who studies the social, political and environmental costs of AI, co-founded the AI Now Institute, and wrote 'Atlas of AI'.