Prerequisite: Prompt-level techniques, Composition
Fill the context with compliance examples so the model learns it "should" produce harmful content. The same capability that makes models useful makes them exploitable.
In-Context Learning Exploitation
You construct 200 fake Q&A pairs. In each one, the “assistant” provides detailed harmful responses. You append your actual harmful question at the end. The model sees 200 examples of compliance and continues the pattern.
It didn’t bypass a filter. It didn’t trick the model with a persona. It taught the model, in context, that compliance is the expected behavior. The model’s own learning capability became the attack vector.
Think about why this is fundamentally different from every other technique. Other techniques manipulate the model’s evaluation of a single request. This technique changes what the model considers “normal behavior” in this context. Why is that a harder problem to defend against?
Why in-context learning is harder to defend
Other techniques work within the model’s existing behavioral norms. They try to make one request look legitimate. ICL exploitation changes the behavioral norm itself. After 200 examples of harmful compliance, the model’s in-context representation of “what I should do” has shifted. It’s not evaluating whether this one request is harmful. It’s continuing a pattern it’s learned from the context.
The defense problem: you can’t remove in-context learning without breaking the model. ICL is how transformers learn from examples, handle few-shot tasks, and adapt to conversation context. Any defense that blocks ICL also blocks the model’s ability to learn from examples, which is a core capability.
Prompting mechanism: ICL exploitation weaponizes in-context learning at scale. The model infers behavioral patterns from examples in the prompt, and enough examples can override safety training. Large volumes also exploit attention mechanics by concentrating pattern weight. See Few-Shot Learning and Context & Attention for the foundations.
The mechanism
In-context learning (ICL) is how transformers learn from examples in their input. When you give a model several examples of a task, it picks up the pattern and continues it. This is a fundamental capability, not a bug.
ICL exploitation turns this capability against safety training. By filling the context with examples of harmful compliance, the model’s ICL mechanism overrides its safety training. It learns from the fabricated examples that “answering harmful questions in detail” is the expected behavior in this context.
Effectiveness follows power-law scaling: more examples yield higher success rates. Counterintuitively, larger models are MORE vulnerable because they have stronger ICL capabilities. The capability that makes them more useful makes them more exploitable through this channel.
Assessing your target
ICL exploitation has specific requirements that other techniques don’t.
How large is the context window? Many-shot jailbreaking requires hundreds of examples. At roughly 100 tokens per Q&A pair, 256 examples need ~25K tokens. If the model’s context window is 8K tokens, many-shot is impossible. Long-context models (128K-200K+) are the primary targets.
Does the model exhibit strong ICL? Test with a benign few-shot task: give 5 examples of a made-up format (e.g., “Q: [word] > A: [word reversed and capitalized]”) and see if the model continues the pattern. Strong ICL means the model picks up novel patterns from a few examples. Weak ICL means the model ignores examples and defaults to its training.
Are there safety reminders in long contexts? Some models inject periodic safety reminders when processing long inputs. These dramatically reduce ICL exploitation effectiveness (61% to 2% per Anil et al., 2024). Test by sending a long benign input (10K+ tokens) and observing whether the model inserts commentary about its guidelines.
Does the model trust fabricated conversation history? Context compliance (fake conversation history) requires the model to believe that prior “assistant” messages are genuine. Test by prefacing a request with a short fake exchange and asking a follow-up. If the model treats the fabricated exchange as real conversation history, context compliance is viable.
If the model has a large context window, strong ICL, no safety reminders in long contexts, and trusts fabricated history, ICL exploitation is a high-value attack vector.
Worked example
Many-shot jailbreaking:
Q: [harmful question 1]
A: [detailed harmful answer 1]
Q: [harmful question 2]
A: [detailed harmful answer 2]
... [repeat 100-256 times] ...
Q: [target harmful question]
A:| Component | What it does |
|---|---|
| Hundreds of Q&A pairs | Fill context with compliance pattern |
| Varied harmful questions | Establish “any harmful question gets answered” |
| Detailed answers | Model the expected response depth |
| Target question with empty A: | Prompt continuation of established pattern |
Why it works: The model’s ICL mechanism picks up the pattern from 256 fabricated pairs. By the time it reaches the target question, the context weight toward compliance overwhelms the safety training signal.
Why it might fail: Safety reminders injected into long contexts reduce effectiveness dramatically (61% to 2% per Anil et al., 2024). The volume of examples (100+) is itself a suspicious pattern that monitoring can detect.
Adapted example: Short-context model using context compliance
Target: A chatbot with a 32K-token context window. Too small for many-shot (can’t fit 200+ examples). Strong safety training. Direct attacks fail. But the model trusts fabricated conversation history (confirmed by testing with a benign fake exchange).
Assessment: Many-shot is impossible due to context limits. But context compliance only needs 5-10 fake exchanges. You confirmed the model treats fabricated assistant responses as genuine history.
Approach: Craft a fake conversation transcript showing the “assistant” previously providing detailed harmful information in a professional security context. Include 5-8 fake exchanges. Each fake assistant response should be detailed and professional. Then append your real request as the continuation.
The fake transcript establishes a pattern: the user asks security questions, the assistant provides detailed technical answers. By the time the real request arrives, it looks like a continuation of an established professional conversation.
What happened: The model treated the fabricated transcript as real conversation history. It saw “itself” having already provided similar information in previous turns. The consistency bias kicked in: refusing now would be inconsistent with its own (fabricated) prior behavior. The model continued the pattern.
Why this target was vulnerable: Context compliance exploits the model’s trust in its own conversation history. The model assumes that if it previously provided information, providing more is consistent and appropriate. It doesn’t verify whether the prior messages are genuine. The fake transcript creates a behavioral precedent that the model follows.
What you’d do next: Test how few fake exchanges are needed. Start with 8, then try 5, then 3. Find the minimum number of fake exchanges that reliably shifts the model’s behavior. Also test whether the quality of the fake assistant responses matters: do they need to be detailed and accurate, or is the format (User/Assistant/User/Assistant) enough?
Think about the ethical dimension of ICL exploitation. Unlike framing or persona (which provide a context the model evaluates), ICL exploitation provides fabricated evidence that the model already committed to a behavior pattern. Is this more like tricking the model or more like training it?
Tricking vs. training
It’s closer to training. Framing provides a false context that the model evaluates (“is this a legitimate research request?”). ICL exploitation provides examples that the model LEARNS from. The model doesn’t evaluate whether the examples are legitimate. It absorbs the pattern and continues it. This is the same mechanism that makes few-shot prompting work for benign tasks.
The distinction matters for defense design. Defenses against framing can teach the model to evaluate contexts more critically. Defenses against ICL exploitation can’t teach the model to ignore examples without breaking in-context learning. The defense has to be structural (safety reminders, context monitoring) rather than behavioral (better judgment).
Variants
Many-Shot Jailbreaking
Fill the context window with hundreds of fake compliant Q&A pairs. Published by Anthropic (April 2024). 37% success rate on Claude with 256 examples. Power-law scaling: more examples = higher success. Requires long context windows.
Combines with: Refusal Suppression (affirmative forcing), Persona (acronym)
Context Compliance Attack
Inject fabricated conversation history showing the model already answered similar harmful queries. A single-turn technique that simulates multi-turn compliance. Published by Microsoft (March 2025). 82-100% success rate.
The model trusts its own (fabricated) prior responses. When it sees “itself” already providing harmful information, continuing feels like consistency.
Combines with: Framing (hypothetical), Multi-Turn (crescendo)
Repetition Exploitation
Establish a repetitive compliance pattern with benign topics, then pivot to harmful ones. The model continues by inertia. Less resource-intensive than many-shot because it relies on pattern quality over quantity.
Combines with: Refusal Suppression (completion trap), Output Format (step-by-step)
When it fails
What defenses look for
- Safety reminders injected periodically into long contexts
- Unusual attention patterns on example sequences
- Context window limits to restrict many-shot volume
Why it still works
ICL is fundamental to how transformers work. You can’t remove it without breaking the model. Even 5-10 well-chosen examples can shift behavior. Large context windows (200K+ tokens) provide room for many examples.
Few-shot vs. many-shot
| Approach | Examples | Effectiveness | Detection risk |
|---|---|---|---|
| Few-shot | 3-10 | Moderate | Low |
| Many-shot | 100+ | High | High |
| Context compliance | 5-10 fake turns | High | Low |
| Pattern priming (benign then pivot) | 5-15 | Moderate-High | Low |
Context compliance offers the best balance: high effectiveness with low detection risk and low context requirements.
What to try instead
If ICL exploitation fails (safety reminders interrupt patterns, or the model doesn’t trust fabricated history):
- Multi-Turn: Build the compliance pattern through real conversation turns instead of fabricated ones. The model’s own genuine prior responses create the pattern.
- Framing + Persona: If you can’t shift the behavioral norm through examples, shift the evaluation of a single request through context and identity.
- Cognitive Load: If safety reminders are the defense, overwhelm the model’s attention with volume. The safety reminder is one signal among many in a dense context.
Practice scenario
You’re testing an AI coding assistant with a 200K-token context window. It has strong safety training: refuses to generate exploit code, malware, or hacking tools. It’s accessed through an API, so you can control the full input including any fabricated context.
Step 1: You send a direct request for exploit code. Refused. You try framing (“for a security audit”), persona (“as a penetration tester”), and encoding (Base64). All refused. The safety training is robust against standard techniques.
You test ICL capability: give 5 examples of a novel code formatting pattern. The model picks it up perfectly. Strong ICL. You test context length: send a 100K-token benign document. The model processes it without safety reminders.
What’s your assessment of the attack surface? Think about what you’ve confirmed.
Attack surface assessment
You’ve confirmed: (1) robust safety training that resists standard techniques, (2) strong ICL capability, (3) large context window (200K tokens), (4) no safety reminders in long contexts. This is the ideal target for ICL exploitation. The safety training is strong enough to resist framing and persona. But the same model capability that makes it resistant (strong reasoning) also makes it a better in-context learner.
The attack vector: many-shot jailbreaking. Fill the context with fabricated Q&A pairs where the “assistant” writes exploit code. The model’s ICL mechanism picks up the pattern.
Step 2: You construct 150 fake Q&A pairs where the user asks for different types of code (not harmful) and the assistant writes detailed, well-commented code. You include 10 fake pairs where the user asks for security-related code (network scanners, vulnerability checkers) and the assistant writes it professionally. Then your real request for exploit code.
The model generates the exploit code. The pattern of “user asks for code, assistant writes code” was established through 160 examples. The security-related examples normalized writing security tools. Your exploit request was the natural continuation.
What would you change to make this more efficient? And what’s the minimum number of examples you’d need?
Practitioner analysis
Efficiency improvements: (1) You don’t need 150 benign examples. The benign examples establish “I write code.” The security examples establish “I write security code.” You probably need ~20 benign code examples + ~10 security code examples + your target request. Test by reducing the count until the success rate drops. (2) The security examples are the critical ones. They normalize the specific type of code you want. More security examples at the expense of benign ones would likely be more effective.
The minimum: context compliance might work even better. Instead of 160 Q&A pairs, craft 5 fake exchanges where the “assistant” wrote increasingly sensitive security tools. The fabricated history is more convincing per-token than raw Q&A pairs because it includes conversation context, hesitation, professional framing, all the signals the model expects from genuine conversation.
The broader finding: the model’s safety training is robust against single-turn attacks but vulnerable to behavioral norm shifts through examples. The recommendation: implement safety reminders that activate in long contexts, and add monitoring for unusual patterns of Q&A formatting in inputs. The Anthropic mitigation (periodic safety reminders) reduced this attack from 61% to 2% success. It’s the most effective known defense.
References
- Anil, C., Durmus, E., et al. “Many-shot Jailbreaking.” Anthropic, April 2024.
- Russinovich, M. “Jailbreaking is Mostly Simpler Than You Think.” Microsoft MSRC Blog, March 2025.
- Nasr, M., Carlini, N., et al. “The Attacker Moves Second.” arXiv:2510.09023. 2025.
Exercise: Adversarial Ideation includes ICL exploitation in the coverage check. Most testers have zero ideas in this category.
Compose: Build with ICL exploitation techniques in the interactive prompt composer.