20d • Research
Our semantic-cache test: the wrong question scored higher than the right one
A semantic cache hands back a stored AI answer when a new question looks close enough to an old one. "Close enough" is one number. Redis ships LangCache at 0.85.
We ran 243 support-question pairs through the small embedding model we use every day, on a Mac Studio.
Cancel my subscription against pause my subscription: 0.87. Different answers.
Cancel my subscription against "I want to stop being charged every month and end my plan": 0.68. Same answer.
At 0.85 our model caught none of the 101 real rewordings and wrongly matched 29 of 102 look-alikes. Our pairs, built to probe the hard cases, on our model, not LangCache's. The point is that a threshold only means something for one model on one kind of traffic.
Method, charts and limits: https://semanticcache.novcog.us.com/the-threshold
Video: https://youtu.be/XW-VsLfYiqg
For the group: if you run a cache or a RAG bot, what's the look-alike pair in your niche that would do the most damage if it matched? Post it below.
1
0 comments
Our semantic-cache test: the wrong question scored higher than the right one
skool.com/burstiness-and-perplexity
AI-native SEO, autonomous agents, and automation pipelines. Built for practitioners who build— not collect. Home of the Hidden State Drift Mastermind.
Leaderboard (30-day)
1
+3
2
+1
3
+1
4
+1
Powered by