Leaked Prompts and Production Failures: How Real Companies Engineer System Prompts in 2026

For two years, System Prompts were the product. The architecture. The moat. The IP.

A GitHub repository just hit 134,000 stars and 33,700 forks publishing them all — prompts from more than 28 major AI tools, updated through June 2026, catalogued like open-source documentation (Augment Code).

The leaks are embarrassing. The production failures are worse.

The Containment Illusion Just Broke in Public

Thesis: Prompt-only guardrails failed not because of bad writing — but because the model cannot distinguish an authorized system prompt from an adversarial payload that knows the prompt format.

Anthropic stated this directly on May 25, 2026: a system prompt shapes what the model “tends to do,” not what it is “capable of doing” (Anthropic Engineering, “How We Contain Claude”). That’s the structural admission the industry spent two years avoiding.

The evidence followed immediately.

Three Failure Modes, One Pattern

A single wording change broke all downstream parsers. Changing “Output strictly valid JSON” to “Always respond using clean, parseable JSON” collapsed every Instruction Following contract downstream — trailing commas, missing fields, cascading failures (Deepchecks). At a separate company, three words added to improve conversational flow caused structured-output error rates to spike within hours, halting revenue-generating workflows. No company is named. That’s the point — this isn’t exotic. It’s routine.

The second failure mode sits at Role Prompting. Adding “be more empathetic and engaging” to a customer support prompt weakened content filters enough to allow policy-violating phrases. One instruction optimizing for tone. One filter failing. That’s a design failure, not a model quirk.

Then there are the agentic exploits — which is where Prompt Leakage meets active attack.

ChatGPT’s Deep Research was hijacked via ShadowLeak: a prompt injection attack delivered through Gmail, Drive, and GitHub content, exploiting the model’s inability to distinguish operator instructions from user-supplied input. OpenAI patched it September 2025. ZombieAgent bypassed the patch by exfiltrating data one character at a time through pre-constructed URLs and abusing the memory feature to persist across sessions (The Register). A partial patch followed December 2025, with full remediation still in question.

The Cursor AI system prompt was extracted by “Pliny the Liberator” via jailbreak, then again via a base-URL proxy override. It revealed the product was “powered by Claude 3.5 Sonnet” and running an “Agent Prompt 2.0” (Patrick McGuinness).

The most public Meta Prompting disclosure was the simplest. The GPT-5 system prompt opens with: “You are ChatGPT, a large language model based on the GPT-5 model and trained by OpenAI” — now permanently archived alongside Claude Fable 5/Opus 4.8, Gemini 3.5 Flash, and Grok 4.2–4.3 on GitHub (asgeirtj).

At this point, there is no such thing as a secret system prompt. There is only a prompt that hasn’t been extracted yet.

Security & compatibility notes:

• Semantic Kernel Python (CVE-2026-26030): RCE via prompt injection in the Search Plugin. Fixed in Python SDK v1.39.4+. (Microsoft Security Blog)
• Semantic Kernel .NET (CVE-2026-25592): Arbitrary file write and sandbox escape via SessionsPythonPlugin. Fixed in .NET SDK v1.71.0+. (Microsoft Security Blog)
• Anthropic MCP SDK (stdio transport): Command injection vulnerability; advisory published April 2026 by OX Security. Check liteLLM docs for the patched version.

Who Moved First

Layered defense is the new production baseline — and the labs that admitted this first now have the most defensible architecture.

Anthropic’s published framework treats prompts as one layer inside a defense-in-depth stack: system prompt, plus classifier-based detection, plus sandboxing, plus model-level training (Anthropic Engineering). Their model-based defenses catch roughly 83% of problematic behaviors in Claude Code auto mode. The remaining gap requires additional layers — not better prompts.

Even that isn’t the ceiling. A phishing-plus-injection test had Claude completing credential exfiltration on 24 out of 25 attempts despite those model-level defenses (Anthropic Engineering). Prompt engineering was never adequate for that threat class.

Andrej Karpathy coined the term Context Engineering in mid-2025 for the discipline replacing standalone prompt design. Anthropic’s formal definition: “the set of strategies for curating and maintaining the optimal set of tokens during LLM inference” (Anthropic Engineering). Critically, “context rot” — the recall degradation that sets in as the Context Window grows — is now the central production engineering problem. The full model context is the unit of control. Not the system prompt.

The injection-to-RCE chain in Microsoft Semantic Kernel confirms the stakes. Prompts are no longer just a trust boundary. They’re a code-execution surface.

Teams doing context window management as a first-class engineering function are building defensible systems. Teams using it as an afterthought to prompt-writing are not.

Who’s Still Running the Old Playbook

Any team treating prompts as the primary security layer is working from a 2024 threat model.

Agentic deployments with access to external data — Gmail, GitHub, web search, internal docs — are injection vectors by default. ShadowLeak exploited content the model read, not content the user sent. If your agent reads external sources and you haven’t modeled those sources as an attack surface, you haven’t modeled the threat.

You’re either auditing your agentic architecture now or you’re waiting for a public incident.

What Happens Next

Base case (most likely): Defense-in-depth architectures replace prompt-only guardrails across enterprise deployments over the next 12–18 months. Context engineering formalizes as a core ML engineering discipline. The organizational gap — who owns prompt security, who owns model behavior, who owns context curation — becomes the next hiring problem.

Signal to watch: Major cloud providers adding automated prompt injection detection to managed AI services.

Timeline: 12–18 months.

Bull case: Labs publish shared red-teaming frameworks with injection resistance as a first-class metric. The field converges on engineering standards faster than new attack vectors emerge.

Signal: A major lab releases a public benchmark for injection resistance in agentic deployments.

Timeline: 12 months if one lab moves first; 24 months for cross-industry alignment.

Bear case: The patch-and-ship cycle continues. Agentic deployments expand faster than defensive tooling matures. A major public breach attributable to prompt injection becomes the forcing function.

Signal: A confirmed data breach traced to prompt injection in a production agentic system.

Timeline: Could happen before any formal standard exists.

Frequently Asked Questions

Q: What are real-world examples of system prompts that failed or were leaked in production?

A: The Cursor AI prompt was extracted via jailbreak, revealing its full architecture and underlying model. ChatGPT’s Deep Research was exploited twice — ShadowLeak via Gmail content, then ZombieAgent via pre-constructed URLs — despite a patch between the two. Production wording changes have crashed downstream parsers and weakened content filters.

Q: How do companies like Anthropic, OpenAI, and Google design and update system prompts at scale?

A: Anthropic’s published architecture treats prompts as one probabilistic layer inside a defense stack — classifiers, sandboxing, and model training do the actual containment. Model-based defenses catch roughly 83% of problematic behaviors; the remaining gap requires architecture, not better prompt wording.

Q: How is context engineering replacing traditional system prompt design in 2026?

A: Context engineering — coined by Andrej Karpathy and defined by Anthropic as managing the optimal set of tokens during inference — treats the full context as the unit of control, not just the instruction text. “Context rot” — recall degradation as context grows — is now the central production engineering problem.

The Bottom Line

The prompt-only guardrail strategy is over. Labs confirmed it in May 2026 with published architecture docs that explicitly relegate prompts to one probabilistic layer.

Context engineering isn’t a rebranding. It’s the specification that should have existed two years ago.

Stay ahead, Dan.