Skip to main content

Stop trusting your cache hit rate. 7 tests show why

An evaluation method for semantic caches that measures false-positive rate beside hit rate, with seven failure modes and their fixes.

Updated 9 min readMarkdown
Stop trusting your cache hit rate. 7 tests show why

This article was originally published on the Actian blog.

A semantic cache decides whether to reuse an LLM response with one number, the similarity between a new query and a stored one. Hit rate measures how often that number clears the threshold. It says nothing about whether the stored response answers the new question. A banking assistant that returns cancellation steps to "How do I prevent my bank card from being cancelled?" still logs a hit.

This guide measures that gap. It runs seven tests on a semantic cache built with Actian VectorAI DB and tracks the false-positive rate (FPR), the share of hits that return a wrong answer, beside the hit rate. It covers the following:

  • It defines an evaluation method with a labeled dataset, a wrong-hit definition, and a calibration split.
  • It shows where a single threshold fails and why tuning can't fix negation or changed values.
  • It measures how VectorAI DB filters enforce tenant, intent, and freshness boundaries.
  • It tests local verification and length-aware thresholds as extra safeguards.
  • It turns the results into an operating approach for a production cache.

The code is in the companion repo.

Architecture overview

Semantic cache architecture

The cache embeds each query, retrieves the closest stored question from VectorAI DB, and serves the stored response when the cosine similarity meets the threshold. A miss goes to the model. The threshold controls how readily the cache accepts a match, but it doesn't establish that two queries share an intent.

Component Choice
Vector store VectorAI DB Community Edition in Docker (5,000-vector limit)
Cached questions 3,075 from Banking77, labeled with 77 intents
Lookup questions 770 from the same dataset
Embedding model sentence-transformers/all-MiniLM-L6-v2
Default threshold 0.80 cosine similarity

Evaluation method

A wrong hit is one where the cached question has a different intent, belongs to another simulated tenant, or refers to a source that changed. FPR is wrong hits divided by all hits, with a target of 5% or lower. Thresholds are calibrated on one half of the lookup set and evaluated on the other.

The tenants and source changes are simulated, and the negation and number pairs come from templates. The results show how each failure mode behaves under controlled conditions, so treat them as evidence of the failure modes and validate the numbers on your own traffic.

Running the tests

You need Docker with Compose, Python 3.11 or 3.12, and uv.

git clone https://github.com/actian-devs/developer-content.git
cd developer-content/blog/semantic-cache-fp
docker-compose up -d
uv sync
uv run python test1_threshold_sweep.py --threshold 0.80
uv run python test2_negation.py --threshold 0.80
uv run python test3_numbers_dates.py --threshold 0.80
uv run python test4_partitions.py --threshold 0.80
uv run python test6_query_length.py --threshold 0.80
uv run python test7_stale_answers.py --threshold 0.80

Test 5 runs a verification model in two steps.

uv run python test5_decider_verify.py \
  --threshold 0.80 \
  --export-hits data/test5_lookups.json
uv run --extra decider python test5_decider_verify.py \
  --threshold 0.80 \
  --model Mapika/decider-0.8b \
  --max-hits 100 \
  --hits-file data/test5_lookups.json
uv run python make_table.py

Results land in results/. The first run downloads the dataset and embedding model.

Test 1: The threshold sweep

The sweep looks up 770 questions at every threshold from 0.50 to 0.99. At 0.80, the cache hit 610 of 770 lookups (79.2%), and 39 hits were wrong (6.4% FPR). "Why was I unable to do a transfer?" matched "Why was I not able to complete my transfer?" at 0.921, across the declined_transfer and failed_transfer intents.

Hit rate and false-positive rate across similarity thresholds

At 0.83, FPR fell to 4.8% on held-out queries and hit rate fell to 70.1%. A strict threshold sends reusable queries to the model, and a loose one accepts matches that need different answers, so choose the operating point against a correctness target.

Tests 2 and 3: Negation, numbers, and dates

Measure Negation Changed numbers and dates
Pairs tested 75 negated questions 38 changed-value pairs, 44 paraphrase pairs
Median similarity of bad pairs 0.946 0.906
Median similarity of paraphrases 0.936 0.911
Served at 0.80 74 of 75 (98.7%) 38 of 38 (100%)
Threshold-only fix No threshold up to 0.99 blocked 95% 0.97 blocked 95%, and paraphrase rate fell from 95.5% to 13.6%
Targeted check Reject when only one query contains negation Exact match on numbers and date words
With the check 4.0% served, 72.1% paraphrase rate 0% served, 95.5% paraphrase rate

Examples include "Should I not cancel my card?" matching "Should I cancel my card?" at 0.990, and "Set my daily card limit to $2500" matching the $250 version at 0.962. Because bad pairs score as high as paraphrases, no threshold separates them. The checks are deliberately simple, and the value check doesn't treat "fifty dollars" and "$50" as equal.

Test 4: Tenant and intent boundaries

Configuration FPR Hit rate
No filters 69.7% 79.2%
MUST filter on tenant 9.1% 61.4%
MUST filters on tenant and predicted intent 4.7% 58.4%
Tenant MUST plus top-two intents as SHOULD 7.9% 60.6%
Per-intent thresholds 9.6% Not reported

Without filters, 409 hits (67.1% of all hits) crossed tenant boundaries. The tenant filter removed all of them, and the remaining errors matched neighboring intents because each tenant partition held fewer examples per intent. The intent classifier was right 88.4% of the time. Per-intent thresholds had only about five calibration questions per intent, which limits that result. Similarity can't act as access control, so the lookup has to enforce boundaries before it picks a response.

Test 5: Local verification

A small model, decider-0.8b, read both questions in 100 sampled hits on an Apple Silicon M3 MacBook.

Cut-off Wrong hits caught Good hits rejected
0.5 4 of 6 25 of 94 (26.6%)
0.3 1 of 6 2 of 94

Median latency was 106 ms per hit, with a 116 ms 95th percentile at batch size one. FPR in the sample fell from 6.0% to 2.8%. With six wrong hits in the sample, this is provisional, and a full run across all 610 hits is still needed.

Test 6: Query length

Length Share of lookups FPR at 0.80 Calibrated threshold
Eight words or fewer 43% 9.4% 0.87
Nine to eleven words Not reported 2.9% 0.80
More than eleven words Not reported 4.6% 0.83

Short queries caused 27 of the 39 wrong hits. Length-aware thresholds brought FPR to 2.0% with a 64.2% hit rate on held-out queries, against 4.8% and 70.1% for a single 0.83 threshold.

Test 7: Stale answers

Each intent acts as a source, and 15 of 77 sources changed after caching. FPR rose to 24.9%, with 113 hits returning the right intent and an outdated answer. One filtered set_payload call marked the 600 affected entries as stale, and a MUST NOT filter excluded them. FPR fell to 11.4% with a 67.4% hit rate. The remaining errors matched neighboring intents, so invalidation needs a refill or intent filtering, followed by another evaluation.

Results summary

Test Mitigation Result
1. Threshold sweep Calibrate against a 5% FPR target 4.8% FPR, 70.1% hit rate at 0.83
2. Negation Negation check 98.7% served fell to 4.0%
3. Numbers and dates Exact value check 100% served fell to 0%
4. Boundaries MUST filters on tenant and intent 4.7% FPR, 58.4% hit rate
5. Verification decider-0.8b on sampled hits 6.0% fell to 2.8% FPR (provisional)
6. Query length Length-aware thresholds 2.0% FPR, 64.2% hit rate
7. Stale answers Stale flag with MUST NOT filter 24.9% fell to 11.4% FPR

The 6.4% overall FPR hid failures concentrated in specific query types and cache conditions. These numbers depend on the embedding model, the Banking77 sample, and the wrong-hit definition, and intent labels are an imperfect proxy for whether two questions can share a response.

Operating a cache you can trust

Practice How to apply it
Set the threshold against a correctness target Label a sample of your traffic, sweep thresholds, and pick the lowest that meets the target
Report FPR beside hit rate Put both on every dashboard and rerun the check on a schedule
Partition the cache Store tenant, intent, and source on each entry and filter on them during search
Invalidate stale entries Mark changed sources stale and exclude them with MUST NOT
Verify high-stakes hits Compare negation, amounts, and dates, and add local verification for refunds and cancellations

Conclusion

Wrong hits clustered in negated requests, changed values, another tenant's data, and outdated policies, and a threshold alone missed them. A correctness target, VectorAI DB filters, and targeted checks addressed each one.

Where to take this next

No comments yet. Be the first to comment.