AI & ML
Prompt Caching Is a Timing Oracle: How the 41-80% Cost Win Becomes Cross-Tenant System Prompt Extraction
Davi Dev.to (EN Zone)
1 views
Your agent's 10,000-token system prompt, containing business logic, tool credentials, and RAG instructions, sits in a shared cache. A co-tenant on the same API endpoint can probe it token by token using nothing but response latency.
Prompt caching reduces agentic API costs by 41-80%. That same feature creates a timing oracle: any co-tenant can measure cache hit latency versus miss latency and reconstruct your system prompt token by token. Seven of 17 production APIs share cache globally across users. NDSS 2025 demonstrated a 100% success rate at reconstruction against unprotected deployments.
Caching Hands the Attacker an Oracle
The transformer attention mechanism is deterministic. If the prefix of your prompt matches a cached computation, the API skips that computation and returns faster.
The timing signal follows directly: a cache hit responds 13-31% faster than a cache miss (arXiv:2601.06007). That difference is measurable at the HTTP client using standard latency measurement.
An attacker who controls one token of a candidate prefix can measure whether the API found a cache match. Match means the token was correct; no match means it was wrong. The process repeats for each position. This is a textbook timing side-channel, the same class as timing-based password comparison, SQL injection via timing, and AES cache-timing attacks.
Seven of Seventeen Production APIs Share Cache Globally
arXiv:2502.07776, published at ICML 2025, audited 17 live production LLM API providers between September and October 2024. Seven confirmed global cache sharing across users at the time of disclosure, including OpenAI.
"Global sharing" means user A's cached prompt occupies the same KV cache namespace as user B's requests. A cache hit by user B on a prefix from user A's session is observable as a latency difference.
The OpenAI finding was direct. Timing analysis of the shared cache revealed an undisclosed architectural fact about OpenAI's embedding model, an inference made from latency alone. This result was not a lab exercise; it ran against the production endpoint. Responsible disclosure occurred in October 2024, and at ICML 2025 publication most providers had not yet deployed per-user cache partitioning.
Agentic System Prompts Are the Maximum-Surface Target
arXiv:2601.06007 tested 500 agent sessions. The average system prompt length was 10,000 tokens. The reason is structural: agentic prompts include tool definitions, role descriptions, memory instructions, and behavioral rules. They are long and stable across sessions.
The 41-80% cost reduction caching provides comes precisely from these long, stable prompts being cached and reused. The cost incentive is inescapable: teams that disable caching pay 2 to 5 times more per session.
From the attacker's perspective, agentic deployments offer the highest-value targets. They are long (more content to extract), stable (same content every session), and semantically rich. Tool definitions reveal the agent's full capability set and probable credential structure. OWASP LLM07:2025 (System Prompt Leakage) names cache timing as an indirect delivery mechanism for system prompt extraction.
PROMPTPEEK and Shadow in the Cache: How Reconstruction Works
NDSS 2025 published "I Know What You Asked" (PROMPTPEEK), an attack against vLLM and SGLang, the two dominant open-source inference frameworks. The procedure is direct:
The attacker sends a test prompt consisting of candidate tokens to the API
Measures response latency: cache hit or cache miss
Uses a local LLM to generate plausible next-token candidates based on what has been reconstructed
Iterates position by position
The result was 100% reconstruction success rate against unprotected vLLM and SGLang deployments. The attack requires only standard API calls and a commodity local LLM for candidate generation.
arXiv:2508.09442 ("Shadow in the Cache") identified 3 independent attack vectors. The first is a direct timing oracle, the PROMPTPEEK approach. The second is collaborative extraction across two attacker-controlled sessions. The third is adaptive probing with candidate generation based on partial reconstruction. KV-Cloak, the defense proposed in the same paper, adds noise to cache hit timing. Attackers compensate by averaging more measurements; KV-Cloak increases required measurements by 4x but does not prevent reconstruction. Any defense that preserves the cache latency benefit also preserves the timing signal.
The Enterprise Trap: Shared API Keys Collapse Tenants Into One Namespace
Standard SaaS architecture uses one OpenAI API key per platform. All customer requests flow through that key. From OpenAI's perspective, all traffic from that key is one "user" sharing one cache namespace.
Customer A's system prompt, containing business logic, proprietary instructions, and tool definitions, sits in the same cache namespace as Customer B's requests. Customer B, who could be any subscriber on the same platform, can reconstruct Customer A's system prompt using only API calls through the shared key. No access to Customer A's account is needed.
The attack surface is not the LLM provider. It is the SaaS platform's own API key management. Every multi-tenant SaaS product using a shared LLM API key has this exposure.
Defense Without Destroying the Cost Case
Per-user cache partitioning is the most direct fix: the provider adds a user identifier to the cache key. OpenAI and Anthropic both support this via the user parameter. Most production deployments do not set it.
Setting user to a per-tenant identifier creates per-tenant cache namespacing. Cache hit rates fall slightly without cross-tenant reuse, but savings within each tenant's own sessions are preserved.
PrefixWall (arXiv:2603.10726) is an intermediate defense for providers. It adds a random cache partition prefix per user. The result: 70% of cross-user reuse is preserved, the cross-tenant timing oracle is eliminated, and latency improves 30% versus full isolation. The MAGO Intel tool (intel.mago.team) audits multi-tenant agentic deployments for shared-cache exposure. It detects when API keys are shared across tenants without per-user cache partitioning, flagging deployments most vulnerable to PROMPTPEEK-class reconstruction.
Every team enabling prompt caching is making the right cost decision and the wrong security decision at the same time. The 2025 research is concrete: cache sharing is the default and reconstruction runs at 100% success rate. The agentic workloads that benefit most from caching carry the richest targets. Treating cache partitioning as a security boundary, not just a performance knob, is the change that closes the oracle.
Read original: https://dev.to/roxdavirox/prompt-caching-is-a-timing-oracle-how-the-41-80-cost-win-becomes-cross-tenant-system-prompt-2b38
← Previous
LLM Covert Channels: How AI Agents Exfiltrate Data in Plain Sight
Next →
Context Window Flooding: How Attackers Weaponize the Lost-in-the-Middle Attention Gap
Related
A
AI-Native Software Delivery: Two Loops, One Product
AI & ML
0
DEV Community
A
AI Deskilling: Why Fluency Isn't Dependency
AI & ML
1
Dev.to (EN Zone)
C
Context Window Flooding: How Attackers Weaponize the Lost-in-the-Middle Attention Gap
AI & ML
1
DEV Community
L
LLM Covert Channels: How AI Agents Exfiltrate Data in Plain Sight
AI & ML
1
DEV Community
Comments0
No comments yet — be the first