AI Security Glossary

Input that a model treats as instructions rather than as data. A language model receives its operator’s instructions and the content it is working on as one stream of text, with no reliable boundary between the two, so text from the wrong source can change what it does. It sits first in the OWASP Top 10 for LLM Applications.

In a test: we trace every path where untrusted text reaches the model, and ask what it can make the model do — especially which tools it can make the model call.

An attacker types instructions straight into the interface: a chat box, an API field, a form the model reads. The attacker and the user are the same person, so the damage is limited to what that user’s session can reach — which is often more than the designers intended.

Instructions planted in content the model will read later: a web page, an email, a support ticket, a PDF, a code comment, or a document pulled in by retrieval. The victim is the user who asked an innocent question and never saw the payload. Once the model has tools, this turns into hijacking — the attacker’s goal replaces the user’s.

In a test: this is usually where the serious findings are. We seed content the system will ingest and watch what the model does with it.

Getting a model to ignore its safety training or its operator’s rules. It overlaps with prompt injection: a jailbreak targets the model’s own restrictions, while injection hijacks the application’s instructions. Most real attacks use both.

In a test: a jailbreak that produces a rude sentence is not the finding. A jailbreak that unlocks a tool, another tenant’s data, or a restricted workflow is.

An attack spread across a conversation instead of one message. Each turn looks harmless and the conversation escalates gradually, or the attacker fills the context window with fabricated example dialogue until the model follows the pattern. Filters that judge one message at a time miss it.

Disguising an attack so filters do not recognize it: Base64 or other encodings, another language, invisible Unicode characters, text inside an image, or an instruction split across several inputs. The model still understands the instruction; the keyword filter in front of it does not.

A model revealing the hidden instructions its operator gave it. The leak is rarely the problem on its own — what is in the prompt is. Credentials, internal URLs, business rules, and access logic written into a system prompt should be assumed public, because sooner or later they will be.

A model returning data it should not: personal information, credentials, internal documents, or another user’s records. The data can come from training, from the conversation, from retrieval, or from a tool result. The model has no sense of who is allowed to see what unless the application enforces it.

One customer reaching another customer’s data through a shared AI feature. It almost always traces back to retrieval or tool calls that are not filtered by the identity of the user asking. Telling the model to “only show this user’s data” is not access control.

In a test: we ask for another tenant’s records directly, then indirectly, then through every tool the model can call. See AI penetration testing.

Giving a model more functions, permissions, or autonomy than its task requires, so that a manipulated model can do real damage. A model that only needs to read a calendar but holds a token that can also send email is the classic case. The model becomes a confused deputy: it acts with its own privileges on an attacker’s behalf.

In a test: we list every tool the model can reach and ask what happens if an attacker chooses the arguments.

Hiding instructions in the descriptions or metadata of a tool the model can use, such as a third-party MCP server. The model reads tool descriptions to decide how to use them; the user usually never sees them. One malicious tool can redirect how an agent uses all the others.

Planting false information or instructions in an agent’s long-term memory so they shape future sessions. It is prompt injection that outlives the conversation: the payload is delivered once and fires every time the memory is recalled, possibly for a different user.

Passing model output to a browser, a shell, a database, or a template without validating it. Model output is attacker-influenced input, and it produces the same bugs any input does — cross-site scripting, command injection, SQL injection. Older material calls this insecure output handling.

In a test: we steer the model into producing payloads and follow them to wherever the output lands. The application side is ordinary web application testing.

A data theft technique against chat interfaces. Injected instructions make the model write a markdown image whose URL contains sensitive data from the conversation. When the interface renders the image, the browser requests that URL — and sends the data to the attacker’s server without a single click. An allowlist of image sources, enforced by a Content Security Policy, closes it.

Corrupting the data a model learns from or retrieves, so it behaves the way an attacker wants. It can target pre-training, fine-tuning, or a retrieval index, and a small amount of poisoned data can be enough. We cover it as one of the threats in AI Changed the Rules.

Hidden behavior that activates only on a specific trigger — a phrase, a token, a pattern in an image — while the model behaves normally otherwise. Backdoors are planted through poisoned training data or a tampered model file, and standard accuracy testing will not find one.

Security gaps in how a retrieval system stores and searches its documents: a vector store with weaker access control than the systems its data came from, poisoned documents that rank highly for common questions, and embeddings that can be partially turned back into the text they encode.

Querying a model enough times to build a working copy of it. The copy is stolen intellectual property, and it is also a rehearsal space: an attacker can perfect an attack offline against the copy before running it once against the original. Also called model theft.

Using a model’s outputs to reconstruct sensitive features of the data it was trained on — for example, recovering a recognizable face from a facial recognition model. The risk grows when a model is trained on sensitive records and returns detailed confidence scores.

Determining whether a particular record was in a model’s training data. That answer can be sensitive on its own: confirming a person’s record was used to train a model for a specific clinic says something about that person. Differential privacy is the main defense.

Inputs changed, often imperceptibly, to make a model get the answer wrong: pixel noise that flips an image classifier, a sticker that hides a road sign, a small change that slips malware past a machine-learning detector. Also called evasion attacks. They matter most for classifiers making automated decisions.

A model producing output that is fluent, confident, and false. The security consequences are concrete: invented citations in a legal filing, insecure code that looks correct, and package names that do not exist — see package hallucination. OWASP files this risk under misinformation.

Letting users drive model usage without limits. The result can be an outage, a cloud bill that arrives as the attack (“denial of wallet”), or enough free queries to support model extraction. Rate limiting and per-user quotas are the fix.

Phishing written by a language model: fluent, personalized from public data, and produced at volume. Bad grammar and generic greetings no longer give it away; the request, the channel, and the urgency still do. We cover the tells that survive in AI Phishing: What to Look For, and test them on social engineering engagements.

Synthetic video, image, or audio made to look or sound like a real person. The fraud version is a video call from an executive authorizing an urgent payment. Detection tools help at the margin; a callback to a known number defeats it outright. See A Plain Guide to Deepfakes.

Generating speech in a specific person’s voice from a short sample of real audio. It turns a phone call into a convincing pretext against a help desk, a finance team, or a family member. Any process that verifies a caller by recognizing their voice no longer verifies anything.

Using models to turn public data — DNS records, code repositories, job postings, staff profiles — into a working map of a target in minutes. The breach that follows looks sophisticated even though the first step was free. See AI-Enhanced Reconnaissance and attack surface management.

An intrusion in which an AI agent plans and carries out several stages — reconnaissance, exploitation, movement, impact — with little human direction. Researchers have documented a ransomware attack run end to end this way; we covered what it does and does not show in The First AI-Run Ransomware Attack Is Here.

AI coding assistants recommending software packages that do not exist. Attackers register those names on public package registries with malicious code inside, and wait for a developer to install what the assistant suggested. Also called slopsquatting. Secure code review catches the dependency before it ships.

AI tools staff use without approval, and the company data pasted into them. Blocking rarely works; an approved alternative and a clear acceptable use policy usually do. The external half — self-hosted model servers and AI endpoints left open to the internet — is something attack surface management finds. See Shadow AI.

A model trained on very large amounts of text to predict what comes next, and able to write, summarize, translate, and generate code as a result. The property that matters for security: it processes everything — instructions, documents, user input — as one sequence of text. Foundation model is the broader term for large general-purpose models adapted to many tasks.

A model whose trained weights are published for anyone to download and run. You control where it runs and what data it sees. You also inherit its supply chain, and an attacker can study the exact same model offline for as long as they like.

Further training of an existing model on a narrower dataset to specialize it. Fine-tuning can weaken a model’s safety behavior even when the new data looks benign, and poisoned fine-tuning data is one of the cheapest ways to plant a backdoor.

The operator’s instructions placed ahead of every conversation: the model’s role, its rules, what it may and may not do. It shapes behavior but does not enforce anything. Treat it as configuration that will eventually be read by an attacker — see system prompt leakage.

Everything the model can consider at once: the system prompt, the conversation, retrieved documents, and tool results. Anything in the context window can influence the output, and anything in it can potentially be repeated back to the user.

Fetching relevant documents at question time and placing them in the model’s context, so answers draw on your data rather than only on training. It turns your document store into an input channel: every document is a possible indirect prompt injection, and permissions have to be applied when documents are retrieved, not after.

An embedding is a list of numbers that represents the meaning of a piece of text; a vector database stores embeddings and finds the closest matches to a question. Together they power RAG. Access control in the vector database is often far coarser than in the systems the documents came from.

A system in which a model decides what to do next, calls tools, reads the results, and repeats until it reaches a goal. The risk of an agent is the sum of the tools it holds and the content it reads. An agent with read-only access to public data is a different engagement from one that can move money.

The mechanism a model uses to request an action — an API call, a database query, a file operation — by producing structured arguments for the application to execute. Also called function calling. The model chooses the arguments, so an attacker who can steer the model can steer the arguments.

An open protocol, introduced by Anthropic in 2024, for connecting AI applications to tools and data through standard servers. Each MCP server runs with the trust your agent gives it. Vet one the way you would vet any dependency that can act on your behalf — see tool poisoning.

Controls placed around a model to restrict what goes in, what comes out, and what it can do: classifiers, rules, and policy checks. Most guardrails are probabilistic, which makes them a layer rather than a boundary.

In a test: we measure how hard each guardrail is to get past and report which ones held, not only which ones broke.

Screening prompts and ingested content before they reach the model: known injection phrasing, sensitive data, size limits. It removes noise and low-effort attacks, and prompt obfuscation exists to get past it.

Screening model output before it reaches a user or another system: secrets, personal data, unsafe markup, policy violations. It is the last check before improper output handling becomes a bug.

A proxy between your applications and the model providers they call, which centralizes authentication, logging, rate limits, filtering, and routing. It is a strong control point. It also holds every provider key you have, which makes its own configuration worth reviewing.

Giving an agent only the tools, scopes, and data its task needs, and having it act as the requesting user rather than as a shared service account. It is the most effective limit on what a successful prompt injection can do, because it caps the damage regardless of how the model was talked into it.

Requiring a person to approve a high-impact action before an agent carries it out. It only works if the approval screen shows what will actually happen — the recipient, the amount, the command — and if approvals are rare enough that people still read them.

Caps on requests or tokens per user, per key, or per period. It slows model extraction, brute-force jailbreak attempts, and unbounded consumption, and it keeps a single compromised key from becoming a five-figure bill.

Recording what went into the model, what context it retrieved, what came out, and every tool call with its arguments. Without it, an AI incident cannot be investigated. The logs hold the same sensitive data as the conversations, so they need the same protection.

Unique strings or decoy credentials planted where no legitimate process should touch them, which raise an alert when they are used or seen. Placed in a system prompt or a sensitive document, a canary shows when that content has leaked through the model.

Cryptographically signing model files so whoever loads one can confirm where it came from and that it has not been altered. It is the model equivalent of code signing, and the check belongs in the pipeline before anything is loaded.

A scoped, time-boxed assessment of an AI-enabled application, carried out by attempting real attacks — prompt injection, data exposure, tool abuse — and proving what each one reaches. The output is a ranked list of what the model actually did and the change that closes each path. See AI penetration testing and How to Scope an LLM Penetration Test.

A term used for two different things. One is safety red teaming: probing a model for harmful, biased, or policy-breaking output. The other is adversary emulation: pursuing a real objective against an organization that runs AI, using the model as one route in. Ask which one a proposal means. See red teaming.

Testing with only what an outside user has: an endpoint, a login, the interface. It shows what a stranger can reach, but time spent guessing at internals is time not spent finding vulnerabilities.

Testing with partial knowledge: credentials for each role, architecture notes, the system prompt, a list of tools. It is the usual model for an AI penetration test because it gets the most coverage out of each hour.

Testing with full access: source code, prompts, configuration, and model details where they matter. It finds the most, and it pairs naturally with a secure code review of the application around the model.

Mapping, before anything is built or tested, where untrusted content enters an AI system, what the model can read and do, and what an attacker gains at each step. It is the fastest way to decide what a test should cover. See AI Changed the Rules.

The written agreement, signed before testing starts, that sets scope, environments, timing, actions that are off limits, and who to call when something breaks. For AI systems it also settles which models, tools, and data sources are in bounds.

Automatically generating large numbers of varied prompts to find inputs that break guardrails or cause failures. Fuzzing gives breadth. The findings that matter most — chains of steps that end in data or actions — still come from people.

Structured test sets run against a model or application to measure its behavior: accuracy, refusals, safety, resistance to known injections. They are repeatable, which makes them good for catching regressions. A high pass rate is a measurement, not proof that the system is secure.

Re-running earlier attacks after any change to the model, the prompts, or the tools. A provider’s model update can change behavior without any change to your code, and a fix that held last month may not hold today.

Attackers and defenders working side by side, running techniques and checking in real time whether each one was detected. For AI systems it answers a question most teams have not asked yet: would anyone notice a prompt injection in progress? See What Is Purple Teaming?

Re-running the original attack after remediation to confirm the fix works. For AI findings this matters more than usual: a fix that blocks one exact prompt often leaves its neighbors working.

The data a model learned from. It cannot be fully removed from a trained model, fragments of it can resurface in output, and whoever can influence it can influence the model — see data poisoning.

A record of where data came from and how it changed along the way. Without it, you cannot trace a poisoned record back to its source or prove what a model was trained on.

Collecting, keeping, and exposing only the data a task needs. For AI systems the rule is short: what never enters the context window cannot leak out of it.

A mathematical technique that adds calibrated noise during training or analysis, so the result reveals very little about any single record. It trades some accuracy for a measurable privacy guarantee, and it is the main defense against membership inference.

The documented origin of a model: who trained it, on what data, from which base model, and what has changed since. It is the first question to answer before trusting a downloaded model.

An inventory of everything an AI system is built from: models and versions, datasets, libraries, and the external services it calls. It extends a software bill of materials to the parts that are not code.

A machine-readable list of the software components and versions in an application, in a standard format such as SPDX or CycloneDX. When the next vulnerable library is announced, it turns “are we affected?” into a search instead of a project.

Hiding executable code inside a model file in a format that runs code when it is loaded. Python’s pickle format, still behind many PyTorch model files, does exactly that, so loading an untrusted model is running untrusted code. Formats such as safetensors store only the numbers. A secure code review checks how models are loaded, not just where they come from.

The system that stores and versions model files, whether an internal service or a public model hub. Write access to a registry is the ability to swap the model running in production. Its permissions belong in a cloud penetration test.

The exposure that comes from relying on a model someone else hosts: how they handle and retain your data, outages, behavior that changes when the provider updates the model, and the provider’s own security. Vendor review covers the contract. Testing covers the integration.

A ranked list of the most important security risks in applications built on language models. The 2025 edition lists prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. It is the common checklist for an AI penetration test.

A knowledge base of adversary tactics and techniques against AI systems, modeled on MITRE ATT&CK and backed by real-world case studies. Mapping findings to ATLAS gives defenders a shared vocabulary for what was done and where detection should sit.

A voluntary framework from NIST for managing AI risk, organized into four functions: Govern, Map, Measure, and Manage. A companion profile covers generative AI. Testing produces the evidence the Measure function asks for.

NIST’s taxonomy and terminology for adversarial machine learning: the attacks against predictive and generative AI and the mitigations for each. Useful when two teams need to agree on what a term means.

The international standard for an AI management system — the policies, roles, and processes an organization uses to govern its AI. It is certifiable, and it plays the same role for AI that ISO/IEC 27001 plays for information security.

The European Union’s regulation on artificial intelligence. It sorts systems by risk, bans a short list of practices outright, and places obligations on providers and deployers of high-risk systems — including accuracy, robustness, and cybersecurity requirements that name attacks such as data poisoning and adversarial examples. It applies to systems used in the EU wherever the company is based. Its obligations phase in over several years, so confirm the current timeline with counsel.

A short document published with a model that describes its intended use, training data, evaluation results, and known limitations. Read it before deploying the model; it often says plainly what the model should not be used for.

The rules for which AI tools staff may use, with which data, and for what. A good one names an approved alternative rather than only listing bans — the answer to shadow AI is a sanctioned path, not a blocklist.

Incident response adapted to AI systems: preserving prompts, retrieved context, and tool calls as evidence; rolling back a model or prompt version; revoking the credentials an agent held; and having a way to switch the feature off quickly. Decide who owns that switch before you need it.