privacy budget
In plain wordsThe total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.
The cap on cumulative privacy loss, measured by the parameters ε (and δ) of differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., permitted across all computations on a dataset. Because privacy loss composes — under basic composition the ε values of successive analyses add, and tighter accounting methods (advanced composition, Rényi differential privacy, numerical privacy-loss-distribution accountants) give smaller totals — each query, release or training step 'spends' part of the budget, and a curator must stop or add more noise once it is exhausted. Choosing the budget is a policy decision balancing accuracy against risk, with no universal threshold. In DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step. the budget is spent over training steps and depends on noise level, sampling rate and number of epochs.
We split the privacy budget, spending ε = 1 on the summary statistics and ε = 4 on model training.
Origin
Implicit in the composition theorems of the founding differential-privacy papers (2006); the budget framing was in use by the late 2000s, and Frank McSherry's PINQ system (2009) was an early platform to make it explicit, tracking and enforcing a budget across queries. The US Census Bureau's public debate over the global privacy-loss budget for its 2020 census data brought the term to policy audiences.
Usage note
Reported ε values are comparable only when δ, the unit of privacy (example, user or document) and the accounting method are also stated.
Related
- differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
- DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step.
- local differential privacylocal differential privacy, n.: A privacy method in which each person's device scrambles their data with random noise before sending it anywhere, so the company collecting it never sees anyone's true answer but can still estimate overall trends.
- membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
Sources
- McSherry, F. (2009). Privacy Integrated Queries: An Extensible Platform for Privacy-Preserving Data Analysis. ACM SIGMOD 2009.
Mentioned in
differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step., local differential privacylocal differential privacy, n.: A privacy method in which each person's device scrambles their data with random noise before sending it anywhere, so the company collecting it never sees anyone's true answer but can still estimate overall trends., PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes.