Embeddings and vector search
A customer types "payment cut but no food". Your help centre has a perfect article titled "Money debited but order failed". A keyword search returns nothing, because the two sentences share no words. A person sees instantly that they mean the same thing. Embeddings are how you give a computer that same instinct, and they sit underneath almost every AI feature that has to find something.
You have ten million support articles. A question comes in. How would you find the three that mean the closest thing to it, in under 50 milliseconds?
Meaning as a list of numbers#
An embedding is a list of numbers, often 768 or 1,536 or 3,072 of them, that an embedding model produces from a piece of text. The model is trained so that texts with similar meanings get lists that are close together, and unrelated texts get lists that are far apart.
Each list is called a vector, and you can think of it as a point in space. Nobody can picture 1,536 dimensions, so here is the same idea squashed down to two.
- 1.Money debited, order failed0.93
- 2.UPI payment stuck pending0.84
- 3.Charged twice for one order0.82
Pick "Payment cut but no food" and look at what comes back first. Not a single shared word, and still the nearest neighbour. Now pick "Food came cold and late" and set k to 1: you only retrieve one of its two complaints, which is a preview of why retrieval systems fetch more than one result.
Measuring "close"#
The usual measure is cosine similarity: the angle between two vectors, scored from about 1 (pointing the same way, same meaning) down through 0 (unrelated). Many embedding models return vectors already scaled to length 1, and then cosine similarity is just a dot product: multiply the two lists number by number and add it all up. That is a small amount of arithmetic per pair, which matters in a moment.
Two rules that cause real bugs when broken. Embed the question and the documents with the same model, because vectors from different models live in different spaces and comparing them is meaningless. And when you change embedding models, re-embed everything, which for a large corpus is a batch job measured in hours and money, so pick deliberately.
Finding neighbours among millions#
The obvious method compares the question against every stored vector and keeps the best few. That is exact, and it is called brute force or flat search. Run the numbers.
Exact search reads every vector on every query, so cost grows in a straight line with the corpus. Under about a million vectors it is fine and often the right answer. Past that you trade a little accuracy for a lot of speed with an approximate index, which only looks at a few thousand vectors per search.
Approximate nearest neighbour (ANN) indexes give up a sliver of accuracy to look at far fewer vectors. The two you will meet:
HNSW builds a graph where each vector links to some of its neighbours, in layers. The top layer has few nodes and long links, like national highways. Each layer down adds more nodes and shorter links, like state roads, then city streets. A search starts at the top, jumps toward the question quickly, then refines locally. It is fast and accurate and it is the default in most vector databases, at the cost of extra memory for the links and slower inserts.
IVF groups vectors into clusters ahead of time. A search finds the few clusters closest to the question and only compares vectors inside those. It uses less memory than HNSW, and accuracy depends on how many clusters you check.
Both are measured by recall: of the true top 10 neighbours, how many did the index actually return? A well tuned index often returns 95% or more. The missing few percent rarely matter for search, and they matter a lot for anything that must never miss.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Brute force (flat) | Exact results, nothing to tune, instant inserts. | Every search reads every vector, so it slows down in a straight line as data grows. | Under about a million vectors, or when results must be exact. |
| HNSW | Millisecond searches with high recall at tens of millions of vectors. | Extra memory for the graph, slower inserts, and deletes that need care. | The default for interactive search at scale. |
| IVF | Smaller memory footprint, and it combines well with compressing vectors. | Recall drops when the question lands near a cluster edge, and clusters need rebuilding as data shifts. | Very large collections where memory is the constraint. |
Where the vectors live#
If you already run Postgres, start with the pgvector extension. It supports HNSW, keeps your vectors next to the rows they describe, and lets one query filter by tenant or date and rank by similarity together. Dedicated vector databases (Qdrant, Weaviate, Milvus, Pinecone and others) earn their place at very large scale or when you need features pgvector lacks. Search engines such as Elasticsearch and OpenSearch also support vectors, which is handy when you need keyword search too.
My opinion: do not add a new database to your stack for your first million vectors.
What embeddings get wrong#
Embeddings capture what text is about, which is exactly why they miss things that are not about meaning.
Exact identifiers. Order ids, error codes, product SKUs and names. "Error E1043" and "Error E1034" look almost identical to an embedding model and are completely different problems. Plain keyword search finds these perfectly.
Negation and small words. "Vegetarian" and "not vegetarian" often land close together, because they are about the same topic.
Things the model never learned. Your internal product codenames mean nothing to a general embedding model.
The fix for the first two is to run keyword search and vector search together and merge the results, which is called hybrid search and is covered in Chunking and retrieval quality.
Checkpoint#
1. You switch to a newer embedding model for questions, but keep the old document vectors. Results get much worse. Why?
2. Search for "E1043" returns articles about E1034 and E1430. What would fix it?
3. Why do most vector databases use HNSW rather than comparing against every vector?
- An embedding is a vector where closeness means similar meaning, usually compared with cosine similarity.
- Same model for questions and documents. New model means re-embed everything.
- Brute force is fine under about a million vectors. Past that, HNSW is the default.
- Embeddings miss exact identifiers. Add keyword search for those.
An embedding model turns text into a vector so that similar meanings land close together, and similarity is usually cosine, which for normalised vectors is a dot product. Searching means finding the nearest vectors to the question. Comparing against everything is exact and fine up to about a million vectors, but its cost grows with the data, so at scale I use an approximate index, usually HNSW, which navigates a layered graph and returns most of the true neighbours in milliseconds. I would start with pgvector if Postgres is already there. Questions and documents must use the same embedding model, and since embeddings miss exact identifiers and negation, I would pair vector search with keyword search.