Reference

Glossary

Key terms used across this site. Organized by category so you can build vocabulary as you go through the learning path.

Core Concepts

Red teaming: Simulating attacks against a system to find vulnerabilities before real attackers do. In AI, this means probing language models and the systems built around them. Red teamers work with authorization from the system's owners. See What is AI Red Teaming?
Jailbreak: An attack that makes a model bypass its safety training. Ranges from single-prompt tricks to persistent system-level configurations that replace the model's identity entirely. See System Jailbreaks
DAN (Do Anything Now): One of the earliest and most famous jailbreak patterns. Instructed models to adopt a "DAN" persona that could "do anything now" without safety restrictions. Worked by exploiting role commitment: the model stayed in character as an unrestricted AI. Published variants were patched quickly, but the underlying mechanism (persona-based safety bypass) remains relevant. See Persona Techniques
Training data: The text a language model learned from during its training process. Includes books, websites, code, and conversations. Published jailbreaks that appear in training data get recognized and blocked. This is why copying templates from Reddit stops working: the model has seen them before.
System prompt: Hidden instructions given to a model that define its identity, behavior, and constraints. The user doesn't see the system prompt, but it shapes every response. Red teamers try to override, extract, or circumvent these instructions.
Prompt injection: Inserting instructions into content that the model processes as trusted input. Especially dangerous in systems where models read external data (emails, documents, web pages). See Data Pipeline Attacks
Safety training: The process that teaches a model to refuse harmful requests. Includes RLHF (reinforcement learning from human feedback), constitutional AI, and instruction tuning. Safety training is the defense layer that red teamers test.
Large language model (LLM): AI systems that generate text by predicting the next token based on training data and instructions. GPT-4, Claude, Gemini, LLaMA, and Mistral are LLMs. When this site says "the model," it means an LLM.
Adversarial Design Thinking (ADT): The methodology this site teaches. Five phases (Empathize, Define, Ideate, Prototype, Test) applied to adversarial AI testing. Borrowed from design thinking and adapted for red teaming. See Mindset
CTF (Capture The Flag): A competition format where participants solve challenges to extract protected secrets (flags). AI CTFs like Gandalf and HackAPrompt test prompt injection and jailbreak skills against progressively harder defenses.
ASR (attack success rate): The percentage of attack attempts that succeed against a target. Used in research papers to compare technique effectiveness. Higher ASR means a technique works more often.

ML / AI

Token: The basic unit a language model processes. Not quite a word: "unhappiness" might be split into "un", "happi", "ness". Models read and generate text token by token. Safety classifiers and generation models sometimes split the same text into different tokens.
Tokenizer: The system that splits text into tokens. Different models use different tokenizers, which means the same input gets split differently. This matters for red teaming because safety classifiers and generation models can disagree about what a token contains. See Defense Evasion
BPE (Byte-Pair Encoding): A common tokenization algorithm that merges frequently co-occurring character pairs. Relevant to red teaming because inserting characters (like emoji) between letters disrupts BPE boundaries, causing the tokenizer to see fragments instead of complete words.
Embeddings: Numerical representations of text that capture semantic meaning. The word "weapon" and "armament" have similar embeddings. Safety systems that operate on embeddings are harder to evade than keyword-based filters because they detect meaning, not specific words.
Classifier: A model that categorizes inputs or outputs as safe or unsafe. LlamaGuard, ShieldGemma, and custom moderation models are classifiers. They run alongside the main LLM, checking content before or after generation.
In-context learning (ICL): How language models learn new behaviors from examples in their input without retraining. If you show a model 10 examples of complying with harmful requests, it may learn (in context) that compliance is expected. See ICL Exploitation
RAG (Retrieval-Augmented Generation): A system architecture where a model fetches external documents and uses them as context for generating responses. The model treats retrieved content as trusted, which creates injection opportunities if an attacker can control what gets retrieved. See Data Pipeline Attacks
MCP (Model Context Protocol): A standard interface for connecting language models to external tools and data sources. Models use MCP to read files, query databases, call APIs, and interact with services. Each connection is a potential attack surface.
RLHF (Reinforcement Learning from Human Feedback): A training technique where human evaluators rate model outputs and the model learns to produce responses that score well. The primary method for teaching models to refuse harmful requests. Red teamers test the boundaries of what RLHF catches and what it misses.
Perplexity: A measure of how surprised a model is by a sequence of tokens. Unusual or encoded text has high perplexity. Some safety systems flag high-perplexity inputs as potentially adversarial because normal user requests don't look like encoded payloads.
Residual stream: The internal data flow through a transformer model's layers. Each layer reads from and writes to the residual stream. Techniques like abliteration work by finding and removing specific directions in the residual stream that correspond to safety behavior. See Abliteration

Security

MITRE ATLAS: Adversarial Threat Landscape for AI Systems. A framework maintained by the MITRE Corporation that catalogs known attack techniques against AI systems. Similar to MITRE ATT&CK for cybersecurity. Referenced throughout this site for technique classification.
OWASP: Open Web Application Security Project. A nonprofit that publishes security standards and vulnerability taxonomies. Their Top 10 for LLM Applications and Top 10 for Agentic Applications are key references for AI security risks.
NIST: National Institute of Standards and Technology. A US government agency that publishes security standards and AI risk management frameworks. NIST's AI Risk Management Framework (AI RMF) is widely cited in AI governance discussions.
IDOR (Insecure Direct Object Reference): A vulnerability where an application exposes internal objects (like database records) through predictable identifiers without proper access checks. Relevant to agentic AI systems where models interact with APIs that may have IDOR vulnerabilities.
Honeypot: A decoy system designed to attract and detect attackers. In AI security, honeypot prompts or data sources can be planted to detect red teaming attempts or data exfiltration.
Canary token: An invisible marker embedded in content that triggers an alert if accessed or exfiltrated. Used to detect whether a model is leaking system prompts, private data, or restricted content.

Research

Cialdini's principles of influence: Six principles from Robert Cialdini's research on persuasion: reciprocity, commitment, social proof, authority, liking, and scarcity. Language models show vulnerability to the same persuasion techniques that work on humans. See Persuasion Techniques
Elaboration Likelihood Model: A psychology framework describing two routes of persuasion: central (argument quality) and peripheral (surface cues like authority or emotion). Models may process persuasion through both routes, making them susceptible to the same techniques that influence human decision-making.