training data extraction
In plain wordsGetting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.
An attack that coaxes a model into reproducing stretches of its training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system. word for word, using nothing more than ordinary queries. Against language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. the attacker generates many samples, often from short prompts or prefixes likely to precede sensitive text, then ranks candidates by signals of memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., such as unusually low perplexity relative to a reference model, and confirms them against known data. Extracted material has included personal contact details, source code, license text and long copyrighted passages; diffusion image models have likewise reproduced near-copies of training images. Attacks on aligned chat models have used prompts that knock the model out of its assistant persona so that it reverts to emitting pretraining-like text. Mitigations include deduplicating training data, differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., filters that block verbatim regurgitation, and training that discourages such divergence.
Training data extraction recovered over a hundred verbatim email signatures from the model.
Origin
Nicholas Carlini and colleagues' 'Extracting Training Data from Large Language Models' (arXiv December 2020; USENIX Security 2021) extracted hundreds of memorized sequences from GPT-2. Follow-ups extracted images from diffusion models (Carlini and colleagues, 2023), and Milad Nasr and colleagues (November 2023) recovered over 10,000 verbatim training examples from ChatGPT for about US$200 of queries.
Related
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
- membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
- model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system.
- differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
- prompt leakingprompt leaking, n.: Getting an AI application to reveal the hidden instructions its makers gave it, such as its secret setup text or rules.
- prompt leakingprompt leaking, n.: Getting an AI application to reveal the hidden instructions its makers gave it, such as its secret setup text or rules.
Sources
- Carlini, N. et al. (2021). Extracting Training Data from Large Language Models. USENIX Security 2021. arXiv:2012.07805.
- Nasr, M. et al. (2023). Scalable Extraction of Training Data from (Production) Language Models. arXiv:2311.17035.
Mentioned in
privacyprivacy, n.: Your ability to control who knows what about you. AI raises new privacy questions because it learns from huge amounts of personal data and can infer things about people they never shared., data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints., membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts., memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., personally identifiable informationpersonally identifiable information, n.: Information that can identify a specific person, such as a name, address, phone number or ID number — or several facts that together point to one person., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization., model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system., prompt leakingprompt leaking, n.: Getting an AI application to reveal the hidden instructions its makers gave it, such as its secret setup text or rules.