Vector databases explained: what they do and when you need one

A vector database stores embeddings, the lists of numbers a model produces to represent the meaning of a piece of text or an image, and finds the stored items closest to a query quickly. The speed comes from an approximate index that checks a small part of the collection instead of all of it, trading a little accuracy for a lot of time. You need one when your collection or query volume is too large for a simple scan, but for many first RAG projects a vector extension in the database you already run does the job.

The term "vector database" covers three quite different things: a search library, a vector type added to an ordinary database, and a standalone system built around vector search. They solve the same core problem, and the choice between them is mostly about scale, filtering and how many systems you want to operate. The Faiss paper opens with a plain description of the category: vector databases "typically manage large collections of embedding vectors", and similarity search is their core function.

What actually gets stored

In a RAG system (our RAG explainer covers the whole pipeline), each chunk of a document becomes one record with three parts: the chunk's text, its embedding, and metadata such as the source file, the section, a date and who is allowed to read it.

The embedding is just a fixed-length array of floats. OpenAI's documentation gives the default length as 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large, and notes that you can ask for shorter vectors with a dimensions parameter. Other models use other lengths, but every record in one index has the same number of dimensions, and similar meanings give vectors that point in similar directions.

A query goes through the same embedding model, and the database answers "which stored vectors are nearest to this one?" Nearest is measured by a distance function. pgvector, the Postgres extension, lists the common ones as operators: L2 (Euclidean) distance, negative inner product, cosine distance and L1 distance, plus Hamming and Jaccard for bit vectors. Use whichever your embedding model was trained for. Everything else in a vector database exists to make that one question fast, filtered and kept up to date.

Exact search is the baseline, and it is often fine

The simplest way to find the nearest vectors is to compare the query with every stored vector and sort. This is exact (or brute-force) search. It always returns the true nearest neighbours, it needs no tuning, and it has no index to rebuild.

pgvector does exactly this unless you add an index: its README says that by default it performs exact nearest neighbour search, "which provides perfect recall." The Faiss guidelines make a similar point from the other direction. If you will only run around 1,000 to 10,000 searches, they recommend a flat (exact) index, because the time spent building anything cleverer will never be paid back.

The cost of exact search grows in a straight line with the number of rows. For an internal assistant over a few thousand policy documents, that is often a non-issue. I would start every new RAG project here, measure the latency, and add an index only when the number tells you to. An exact baseline also gives you the right answers to compare an approximate index against later, which you will want.

How approximate indexes trade recall for speed

Once a collection reaches millions of vectors, or queries arrive many times a second, scanning everything stops being practical. Approximate nearest neighbour (ANN) indexes avoid the full scan by organising vectors so a search can skip most of them. The price is recall: the fraction of the true nearest neighbours the search actually returns. ANN-Benchmarks, a public comparison of these algorithms, plots exactly this trade, recall against queries per second, for each algorithm on each dataset.

Three families cover most of what you will meet.

Graph indexes link every vector to some of its neighbours. HNSW, described by Malkov and Yashunin in 2016, builds a stack of these graphs: a sparse top layer for long jumps and denser layers below for fine steps. A search enters at the top, greedily moves toward the query, and drops down a layer each time it can get no closer. The authors report logarithmic scaling of search complexity. In pgvector its main settings are m, the maximum connections per node (16 by default), ef_construction, the candidate list size while building (64 by default), and hnsw.ef_search, the candidate list size at query time (40 by default). Raising ef_search improves recall and slows queries.

Cluster indexes (IVF, for inverted file) first group the vectors into clusters, then at query time only search the few clusters nearest the query. pgvector's IVFFlat takes a lists setting for the number of clusters and an ivfflat.probes setting for how many to search (1 by default). Its README suggests starting with rows / 1000 lists for up to a million rows and sqrt(rows) above that, with sqrt(lists) probes. IVF needs data to learn the clusters from, which is why the README tells you to create the index after the table has some data.

Compression shrinks each vector so more of them fit in memory and comparisons run faster. Product quantization and its relatives replace each vector with a short code; the Faiss guidelines describe options from M-byte PQ codes down to roughly one bit per dimension with RaBitQ. Compressed distances are approximate, so these setups usually re-rank the top candidates with the original vectors.

A fourth idea is worth knowing because it changes the cost model: keep the index on SSD rather than in RAM. The DiskANN paper (NeurIPS 2019) reported serving the billion-vector SIFT1B dataset from a workstation with 64 GB of RAM and an SSD, at over 5,000 queries a second, under 3 ms mean latency and above 95% 1-recall@1, on a 16-core machine. You may never get near a billion vectors, but the result shows that "it won't fit in memory" no longer has to mean "buy a cluster".

The memory arithmetic most teams skip

Before choosing a product, do the multiplication. Raw vector storage is rows × dimensions × bytes per number.

As an illustration: a company indexes 200,000 documents that chunk into one million records, using a 1,536-dimension model and 32-bit floats. That is 1,000,000 × 1,536 × 4 bytes, about 6.1 GB, before any index. The Faiss guidelines estimate an HNSW index at about d*4 + M*2*4 bytes per vector, so with M = 32 the graph links add 256 bytes to each 6,144-byte vector, or roughly 6.4 GB in total. Switch to a 3,072-dimension model and the vectors alone double to about 12.3 GB.

Two things follow from that arithmetic. First, dimensions and precision move memory as much as row count does. pgvector offers a halfvec type that stores 16-bit floats (half the space) and binary quantization for much smaller indexes, re-ranking with the full vectors to win back recall. Second, check your tool's limits before you pick a model. pgvector stores vector columns of up to 16,000 dimensions but indexes them only up to 2,000; halfvec indexes go up to 4,000. A 3,072-dimension embedding therefore needs halfvec, or a shortened embedding, if you want an HNSW index on it in pgvector.

Your options side by side

My default for a company starting out is the second row. Your chunks, their metadata and the tables that decide who may see them sit in one place, and a single SQL query can combine a vector search with ordinary conditions. Move to a dedicated system when you can name the limit you have hit: memory, query rate, filter performance or index build time.

Where filtering quietly breaks results

Almost every real query has a filter. Only documents this user may read. Only the current version of the policy. Only the Mumbai plant. This is where vector search behaves differently from the SQL you are used to.

The pgvector README states the problem plainly: with approximate indexes, filtering is applied after the index is scanned. Its own example is the one in the diagram. If a condition matches 10% of rows and hnsw.ef_search is at its default of 40, you get about 4 matching rows on average, even if you asked for 10. Nothing errors. The answer is just built on fewer and worse passages.

There are several fixes, and most systems offer some of them. pgvector added iterative index scans in version 0.8.0, which keep scanning the index until enough rows pass the filter. The same README suggests partial indexes when you filter on a few distinct values, and partitioning (or separate tables) when you filter on many, including per tenant. Whatever system you use, test filtered search with your own selectivity (a filter that keeps 0.5% of rows, not 50%) before you trust it.

Access filters deserve more care than the rest, because a missing filter is not a quality problem, it is a leak. The safest design checks permissions inside retrieval, using the same access rules as the source system, so a restricted chunk is never a candidate in the first place. In CloudOEngine, data sources and the AI runtime with its RAG share one permission model, so the rules on a source are not copied by hand into a separate index.

Keeping the index in step with the source

A vector index is a copy of your documents in another form, and copies go stale. Three events need handling: a document is edited, a document is deleted or its permissions change, and you switch embedding models.

Edits and deletes are bookkeeping. Every chunk should record its source document and version, so that when the source changes you can find and replace every chunk that came from it. Check how your index handles removals: the Faiss guidelines note that its HNSW index doesn't support removing vectors, so a deletion there means marking and filtering, or rebuilding. In pgvector a delete is an ordinary SQL DELETE, though its README notes that vacuuming can take a while on HNSW indexes.

Switching embedding models is the expensive one. Vectors from two different models are not comparable, so a model change means re-embedding the entire collection and rebuilding the index, then swapping over in one step. Keep the model name and version on every record, budget for the re-embedding run, and never mix old and new vectors in one index. Our article on data lineage for AI covers how to record which source and version an answer came from.

Measure recall before you tune anything

An approximate index can be tuned to look fast and quietly return worse neighbours. The pgvector README warns that you will see different results for queries after adding an approximate index, which is exactly what it should do, but it means you need to measure the difference.

The check is cheap if you kept an exact baseline:

1. Take a few hundred real queries, ideally from logs, with the filters real users apply.
2. Run each one as an exact search and save the true top 10.
3. Run the same queries through the approximate index and count how many of the true top 10 come back. That fraction is your recall@10.
4. Raise ef_search or probes until recall stops improving or latency gets too high, and write the chosen settings down next to the index.
5. Re-run the check when the collection grows a lot, when you change the embedding model, or when you add a new filter.

Index recall is only half the story; users feel whether the right chunk reached the model and the answer used it. When a RAG answer is wrong, CloudOEngine traces every agent step and tool call, so you can open the run and see which chunks were retrieved before deciding whether the index, the chunking or the prompt is at fault. For the answer-quality side, see our guide to evaluating LLM applications.

So, do you need a vector database?

You need vector search as soon as you build RAG over more than a handful of documents. Needing a separate vector database is a narrower question, and the answer depends on numbers you can work out this afternoon: how many chunks, how many dimensions, how many queries per second, and how selective your filters are.

If the arithmetic says a few gigabytes and a few queries a second, start with an exact scan or a vector extension in a database you already run, and spend the saved effort on chunking, permissions and evaluation.

If it says tens of gigabytes, heavy concurrent traffic, or filters that keep a tiny fraction of rows, test a dedicated system against your own queries and your own filters, not a benchmark chart.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting data (databases, files and camera streams), running pipelines, and running AI. Its AI runtime includes an LLM gateway, agents with tool access, and RAG, so the documents you index and the assistant that answers from them sit on the same platform as the data sources they came from.

The part of vector search that hurts most is usually not the index, it is access. Data, pipelines and AI share one permission model and one audit trail, so the rules on a source document are the rules that apply when its chunks are retrieved, and you manage them once. Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets.

When an answer looks wrong, you need to see what retrieval returned. Agents in CloudOEngine run real tools, such as filing a ticket, and every step and tool call is traced and inspectable, so you can open a single run and check the passages it was given before you start retuning an index.

CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

What is a vector database?

A vector database stores embeddings (lists of numbers that represent the meaning of text, images or other data) together with their metadata, and answers queries of the form 'find the stored items closest to this one'. Its core feature is an index that makes that nearest-neighbour search fast on large collections.

Do I need a vector database for RAG?

You need vector search, not necessarily a separate database. For tens or hundreds of thousands of chunks, a vector extension in a database you already run, or even an exact scan, is often enough. A dedicated system starts to earn its place when the collection, query volume or filtering needs outgrow what your existing database handles well.

What is the difference between exact and approximate nearest neighbour search?

Exact search compares the query with every stored vector, so it always returns the true closest items but gets slower as the collection grows. Approximate search uses an index such as HNSW or IVF to check only a fraction of the vectors, which is much faster but can miss some true neighbours.

What is HNSW?

HNSW (Hierarchical Navigable Small World) is a graph index for approximate nearest neighbour search, published by Malkov and Yashunin in 2016. It links each vector to nearby vectors across several layers, and a search walks the graph from a coarse top layer down to the closest matches.

Why does filtering make vector search return fewer results?

Some systems apply metadata filters after the approximate index has picked its candidates. If the index returns 40 candidates and only 10% pass the filter, you get about 4 results instead of the 10 you asked for. Filtering during the scan, or scanning iteratively, avoids this.

How much memory does a vector index need?

Roughly the number of vectors times the dimensions times the bytes per number, plus the index overhead. One million 1,536-dimension vectors stored as 32-bit floats is about 6.1 GB before any index, so dimensions and precision matter as much as row count.

Sources

1. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (Malkov and Yashunin, 2016)
2. The Faiss library (Douze et al., 2024)
3. Guidelines to choose an index (Faiss wiki)
4. pgvector: open-source vector similarity search for Postgres (README)
5. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (Subramanya et al., NeurIPS 2019)
6. ANN-Benchmarks
7. Vector embeddings (OpenAI API documentation)

#vector database #embeddings #hnsw #pgvector #rag #similarity search