AI audit
In plain wordsAn independent check of an AI system, or of the company using it, against agreed standards — for example, testing whether a hiring tool treats women and men fairly.
A structured examination of an AI system, or of the organization that develops or deploys it, against defined criteria, producing findings that others can rely on. Audits differ in object and depth: technical or model audits test behavior directly (accuracy, robustness, biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., privacy leakage, dangerous capabilities); process or governance audits check documentation, risk management and controls against standards such as ISO/IEC 42001ISO/IEC 42001, prop. n.: An international standard for how an organization should run its AI work responsibly — policies, risk checks, roles and records — which outside auditors can certify, much as ISO 9001 does for quality. or the NIST AI Risk Management FrameworkNIST AI Risk Management Framework, prop. n.: A free, voluntary guide from the US government's standards agency that helps organizations spot, measure and reduce the risks of the AI systems they build or use.; and outcome audits examine real-world effects. They may be internal, commissioned from an independent firm, or adversarial — carried out by researchers or journalists without the developer's cooperation. Legally required examples include annual bias audits of automated employment decision toolsautomated employment decision tool, n.: Software that scores, ranks or screens job candidates or employees. New York City requires employers using such tools to have them checked for bias every year and to tell applicants. in New York City and the conformity assessmentsconformity assessment, n.: The check, done by the maker or by an approved outside body, that a high-risk AI system meets the EU's legal requirements before it can be sold or used. of the EU AI ActEU AI Act, prop. n.: The European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models.. Effective audits need adequate access (black-box queries are often insufficient), auditor independence and mature standards, all of which remain in short supply.
The AI audit found the screening model's selection rate for older applicants was well below the benchmark.
Origin
Grew from the 'auditing algorithms' agenda in social science (Christian Sandvig and colleagues, 2014) and from fairness research such as Joy Buolamwini and Timnit Gebru's Gender Shades study (2018) and Inioluwa Deborah Raji and Buolamwini's 'Actionable Auditing' (2019); Raji and colleagues proposed an internal audit framework in 2020.
Usage note
Distinguish from an alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss., which investigates whether a model has hidden objectives, and from third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is. of frontier capabilities, although the boundaries blur.
Related
- third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is.
- algorithmic impact assessmentalgorithmic impact assessment, n.: A structured review, done before and during the use of an automated system, of whom it could affect and how — such as unfair treatment or privacy harms — and what will be done about it.
- algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it.
- conformity assessmentconformity assessment, n.: The check, done by the maker or by an approved outside body, that a high-risk AI system meets the EU's legal requirements before it can be sold or used.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- automated employment decision toolautomated employment decision tool, n.: Software that scores, ranks or screens job candidates or employees. New York City requires employers using such tools to have them checked for bias every year and to tell applicants.
- alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss.
Sources
- Raji, I. D. & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Proceedings of AIES 2019.
- Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D. & Barnes, P. (2020). Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. Proceedings of FAT* 2020.
Mentioned in
AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect., algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., AI liabilityAI liability, n.: Who has to pay or answer in court when an AI system causes harm — the company that built it, the business that used it or the person who relied on it — and under which laws., AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision., algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it., algorithmic impact assessmentalgorithmic impact assessment, n.: A structured review, done before and during the use of an automated system, of whom it could affect and how — such as unfair treatment or privacy harms — and what will be done about it., conformity assessmentconformity assessment, n.: The check, done by the maker or by an approved outside body, that a high-risk AI system meets the EU's legal requirements before it can be sold or used., Gender ShadesGender Shades, prop. n.: A 2018 study showing that commercial face-analysis software guessed the gender of darker-skinned women far less accurately than that of lighter-skinned men., ISO/IEC 42001ISO/IEC 42001, prop. n.: An international standard for how an organization should run its AI work responsibly — policies, risk checks, roles and records — which outside auditors can certify, much as ISO 9001 does for quality., Joy BuolamwiniJoy Buolamwini, prop. n.: A Canadian-American computer scientist and activist whose research showed that face-analysis software works far worse on darker-skinned women, and who founded the Algorithmic Justice League., responsible AIresponsible AI, n.: An umbrella name for the practices organizations use to keep their AI systems fair, safe, private and accountable, from the first design meeting to everyday use., third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is., AI Incident DatabaseAI Incident Database, prop. n.: A free public online collection of reports about real cases in which AI systems caused or nearly caused harm, kept so that builders and regulators can learn from past failures., Algorithmic Justice LeagueAlgorithmic Justice League, prop. n.: Joy Buolamwini's organization, known for showing that face-analysis software made far more mistakes with darker-skinned women, which campaigns to hold the makers of biased AI systems accountable., alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss., automated employment decision toolautomated employment decision tool, n.: Software that scores, ranks or screens job candidates or employees. New York City requires employers using such tools to have them checked for bias every year and to tell applicants., notified bodynotified body, n.: An independent testing and certification organization, approved by an EU government, that checks whether certain risky products — including some AI systems — meet the law before they can be sold.