The Semantic Cache Poisoning Problem: Why One Wrong Answer Serves Confidently Wrong Results at Scale

The Optimization That Quietly Changed Your Failure Model

Every team that puts an LLM in front of real traffic eventually reaches for a semantic cache. The economics are irresistible: instead of paying for a fresh inference on every query, you embed the incoming request, find a previously answered query that is close enough in vector space, and serve the stored answer. Latency drops from seconds to milliseconds, cost drops by whatever fraction of your traffic is repetitive, and the demo looks great. What almost nobody notices is that you have just changed the failure model of your entire system.

A traditional key-value cache has a simple, well-understood pathology: it can serve stale data for a key you explicitly wrote. A semantic cache has a far stranger one. It serves data for keys you never wrote at all -- because "close enough in embedding space" means one stored answer now covers an entire neighborhood of questions, including questions whose correct answer is different. The instant a wrong answer enters that neighborhood, it stops being one bad response and becomes the authoritative response for every query that lands nearby. That is semantic cache poisoning, and it is a category of bug that request-key caching simply cannot produce.

Why Semantic Caches Poison Differently

Start with what a cache hit actually means here. In a conventional cache, a hit is an identity match: the key you looked up is the key that was stored. In a semantic cache, a hit is a similarity judgment: the incoming query's embedding is within some threshold distance of a stored query's embedding. Those are not the same guarantee. Two questions can sit close in vector space and still have genuinely different correct answers -- "how do I cancel my subscription" and "how do I pause my subscription" are neighbors in embedding space and opposites in intent. Set your similarity threshold a hair too loose and the cache will confidently serve the cancel answer to the pause question.

Now add the poisoning dynamic. Suppose one inference produces a subtly wrong answer -- a hallucinated policy detail, a stale price, a misread of the retrieved context. In a stateless system that error affects exactly one response and then evaporates. In a semantic cache it gets written to the store, and from that moment every semantically similar query is a cache hit against the poisoned entry. One bad generation has been promoted from a transient blip to a persistent, high-confidence, low-latency wrong answer served to a whole cluster of users. The cache did not just remember a mistake; it amplified it across the neighborhood.

The embedding layer makes it worse, because the boundaries of that neighborhood are invisible and they drift. You cannot enumerate which future questions will hit a given cached answer -- the blast radius is defined by a distance threshold in a space no human reads directly. And if you ever upgrade your embedding model, every stored entry's position shifts underneath you, so a cache that was safe yesterday can start returning different neighbors today. That is the same class of silent corruption behind retrieval freshness in enterprise RAG, where your index rots while every dashboard stays green, and it means a semantic cache is not a static store you can reason about once -- it is a live index that decays.

The Dashboards Stay Green While It Rots

The reason semantic cache poisoning survives so long in production is that it fires every metric your monitoring loves. Cache hit rate is up. P50 and P99 latency are down. Token spend is down. Error rate is flat, because a poisoned cache hit is not an error -- it is a successful, fast, cheap response that happens to be wrong. Every signal on the dashboard says the optimization is working, which is exactly the profile of a failure that hides inside a success metric. It is the AI-systems version of the problem enterprise teams keep relearning: observability for AI is nothing like traditional APM, and the dangerous failures are the ones that never trip an alarm.

This is also a textbook case of a system passing all its liveness checks while its intelligence quietly degrades -- a binary "is it up" health check will never catch a cache that is up, fast, and wrong. Catching it requires wiring a quality signal into the cache path rather than trusting hit-rate alone, the same reason mature teams demand eval-driven development, because testing an AI system is harder than building it. Without that signal, the poisoned entry serves indefinitely, and the first real detection is a customer complaint or a compliance finding -- long after the wrong answer has been served ten thousand times.

Engineering a Cache That Fails Closed

A semantic cache is worth having. The point is not to rip it out but to stop treating it as a simple speed knob and start treating it as a stateful component with its own correctness requirements, its own eviction policy, and its own observability. The goal is a cache that fails closed -- that returns a miss when it is unsure rather than a confident wrong hit.

Separate your thresholds by cost of error. A single global similarity threshold is a bet that every query neighborhood tolerates the same fuzziness, which is false. Queries whose wrong answers are cheap can run a loose threshold; queries touching policy, pricing, or compliance should run a threshold tight enough that only near-identical questions hit, or should bypass the cache entirely. Route by intent before you route by distance, the same discipline behind semantic routing that classifies intent before dispatching.

Never let an unvalidated generation write to the cache. The poisoning entry point is the write path, so guard it. An answer earns a place in the cache only after it clears the same validation you would apply before shipping it downstream -- schema checks, policy checks, and confidence gates. This is where structured output engineering and an explicit guard layer pay for themselves: a response that cannot pass production guardrails must never become the cached authority for its neighborhood.

Make cached answers expirable and invalidatable by meaning, not just by key. When a policy changes or a price updates, you need to evict every semantically related entry, not just the one exact string you can name. That requires tagging cache entries with the source facts they depend on, so a change to an underlying fact invalidates the whole cluster that leaned on it -- the caching analog of data contracts that pin down what each layer is allowed to assume.

Instrument the cache path with a quality signal, not just a hit-rate signal. Sample cached responses and re-score them against a fresh generation or an eval, so a rising divergence between what the cache serves and what the model would now produce becomes a visible metric. That is canary analysis applied to your cache: you are watching for the moment a stored answer drifts away from correct, before your users find it for you. Pair it with drift detection on the pipeline so an embedding-model upgrade that shifts every neighborhood triggers a revalidation instead of a silent corruption.

Cache the Speed, Not the Mistakes

Semantic caching is one of the highest-leverage optimizations available to a production LLM system, and it is also one of the easiest ways to build a machine that serves wrong answers faster and cheaper than ever. The difference between those two outcomes is entirely in the engineering around the write path, the threshold policy, and the observability. A semantic cache with no validation on writes, one global threshold, and hit-rate as its only metric is a poisoning incident waiting for its trigger. A cache that validates before it stores, routes by intent, invalidates by meaning, and watches its own output quality is what lets you keep the latency and cost wins without betting your correctness on a distance threshold nobody is watching.

Bigyan Analytics builds production LLM infrastructure where the caching layer is treated as a correctness-critical component, not a speed hack -- validated writes, intent-aware thresholds, and quality signals wired into the cache path. Book a working session to pressure-test your caching architecture.