Embeddings are not search
Cosine similarity finds things that vibe with your query, which is not the same as things that answer it. Why every RAG stack still needs a keyword path, with diagrams.
Confession that will save you a quarter: my first serious RAG
build used embeddings only, because embeddings are magic and
keyword search is my dad's technology. Then a user searched
for ERR_CONN_4012 and the vector index confidently returned
three documents about connection philosophy, none containing
the string. My dad's technology would have found it in four
milliseconds.
Retrieval is the problem, we established that. Today's more specific claim: embeddings alone are a bad retriever, structurally, and the failure modes are predictable enough to draw.
what similarity actually optimizes
An embedding maps text to a point such that related meanings
land nearby. "Related meanings" is doing sneaky work in that
sentence. Similar-meaning is symmetric: "how do I cancel my
subscription" sits near "canceling your plan", great. But
queries are not statements of meaning; they are requests with
exact anchors: error codes, invoice numbers, function names,
version strings. Exact anchors are precisely what dense
vectors blur, because the whole point of the representation
is generalization. The model has never seen ERR_CONN_4012
enough times to give it a home, so it gives it a
neighborhood, and neighborhoods answer questions nobody
asked.
The reverse failure is quieter and worse: high similarity, zero relevance. "Reset your password" and "change your password" are near-identical vectors and materially different procedures. Cosine 0.93, support ticket reopened.
the boring fix that ships
Hybrid retrieval: run BM25 (or your database's full-text search, which is fine) beside the vector query, merge with reciprocal rank fusion, optionally rerank the top slice. Every serious search team converged here years ago, which should have been the tell. The lexical path catches the anchors, the dense path catches the paraphrases, and the fusion means neither has to be perfect.
The second diagram shows the shape that has survived production at three companies I can speak for: two cheap retrievers, one fusion step, one optional reranker, and evals on top deciding the weights, because the eval harness is where retrieval quality actually gets decided. My hybrid weights came out of a hundred labeled queries, not a blog post, including this one.
the part people skip
Log the queries that fail. Not aggregate recall; the actual strings users typed before rating the answer garbage. Mine sorted into three piles: exact anchors (lexical fixes it), vocabulary mismatch (synonyms in the index fix it), and questions the corpus genuinely does not answer, which no retriever fixes and every retriever gets blamed for. The third pile was 40%. Nobody wants to hear that the search is fine and the docs are missing. The logs say it anyway.
Embeddings are a wonderful component and a terrible strategy. Search was always a systems problem. Your dad was right.
tags: #rag #embeddings #retrieval