AI security

The field concerned with the confidentiality, integrity and availability of AI systems, and with the security consequences of AI capabilities. It covers attacks across a model's life cycle — data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations. and backdoorsbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time. during training, evasionevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through. and jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. at inference, prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. against applications and agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., privacy attacks such as membership inferencemembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts., and theft of weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. — together with defenses such as adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., confidential computingconfidential computing, n.: Running computations inside a hardware-sealed compartment of a chip, so that data and AI models stay protected even from the cloud company or administrator operating the machine. and supply-chain controlsAI supply chain security, n.: Protecting everything an AI system is built from — downloaded models, datasets, code libraries and tools — against tampering, so a poisoned or fake part cannot sneak into the finished product.. It also studies AI as a tool for attackers and defenders (AI-enabled cyberattackAI-enabled cyberattack, n.: A computer attack in which criminals or spies use AI to help — writing convincing scam messages, finding weak spots, producing malicious code, or even running much of the hacking by itself., AI-assisted vulnerability discoveryAI-assisted vulnerability discovery, n.: Using AI to find security holes in software — and often to suggest fixes — faster than human experts alone, so that defenders, or attackers, can act on them.). Work is organized around an explicit threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough. and shared catalogs such as MITRE ATLASMITRE ATLAS, prop. n.: A free online reference from the US nonprofit MITRE that maps, step by step, how attackers go after AI systems, with examples drawn from real attacks and security tests. and the OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one..

The AI security review flagged that the support agent could both read customer records and send email.