Vector Databases Explained: How Embeddings, Similarity Search, and AI Retrieval Really Work

Large language models do not continuously browse the live internet during inference. Their internal parameters remain frozen from the moment their pre-training run concludes. Yet every day, enterprise AI systems answer questions about contracts uploaded ten minutes ago, debug code written this morning, and retrieve newly published clinical guidelines.

The language model is only one half of the equation. Behind the scenes lies a dedicated retrieval engine designed to store, index, and query billions of mathematical representations of meaning rather than exact keywords: The Vector Database.

This deep dive breaks down the internal architecture, mathematical indexing algorithms, and real-world trade-offs of modern vector storage engines like Milvus, Qdrant, Pinecone, and pgvector.

---

1. The Core Limitation of Relational & Lexical Databases

Traditional databases (PostgreSQL with B-Trees, Elasticsearch with Inverted Indexes, SQLite) excel at structured queries and exact keyword matching:

• WHERE user_id = 42 (B-Tree lookup: $\mathcal{O}(\log N)$)
• WHERE title LIKE '%kubernetes%' (Inverted Index lookup)

However, lexical search fails when semantics and intent diverge from exact word choices:

To solve this, Machine Learning models (e.g., text-embedding-3-large, bge-large-en) project text, images, and audio into high-dimensional geometric spaces (typically 768, 1,536, or 3,072 dimensions). In these spaces:

• Synonyms and semantically related concepts cluster closely together.
• Mathematical distance directly corresponds to semantic similarity.

---

2. The Complete Vector Retrieval & RAG Pipeline

In a modern Retrieval-Augmented Generation (RAG) architecture, unstructured enterprise data is transformed, ingested, and retrieved through a multi-stage pipeline:

---

3. High-Dimensional Distance Metrics

Vector similarity search measures the mathematical geometric proximity between dense vectors in $d$-dimensional vector space:

Cosine Similarity

Measures the cosine of the angle between two non-zero vectors. Invariant to vector magnitude:

$\text{Cosine Similarity}(\vec{u},\vec{v})=\frac{\vec{u}\cdot \vec{v}}{\Vert \vec{u}\Vert \Vert \vec{v}\Vert }$

Euclidean Distance (L2)

Measures the straight-line distance between two vector coordinate points in Euclidean space:

$d(\vec{u},\vec{v})=\sqrt{{\sum }_{i=1}^n(u_i-v_i{)}^2}$

Dot Product (Inner Product)

Used for normalized unit vectors:

$\text{Dot Product}(\vec{u},\vec{v})=\vec{u}\cdot \vec{v}={\sum }_{i=1}^n{u}_i{v}_i$

---

4. Vector Indexing Strategies & Internal Data Structures

Exact $k$-Nearest Neighbor ($k$-NN) brute-force scanning requires $\mathcal{O}(N\cdot D)$ time complexity per query, which quickly degrades when searching millions of vectors. Vector databases use Approximate Nearest Neighbor (ANN) indexing:

1. Hierarchical Navigable Small World (HNSW)

HNSW builds a multi-layer graph structure where upper layers have longer skip-links for fast coarse routing, and lower layers have dense local clustering for fine-grained nearest neighbor discovery:

• Query Complexity: $\mathcal{O}(\log N)$
• Tradeoff: Exceptional recall (>98%) and low latency at the expense of high memory (RAM) usage.

2. Inverted File with Product Quantization (IVF-PQ)

• Inverted File (IVF) partitions vector space into Voronoi cells via $k$-means clustering.
• Product Quantization (PQ) compresses high-dimensional vectors into compact byte codes.
• Tradeoff: Reduces RAM consumption by up to 90% with minor recall degradation.

3. DiskANN (Graph-on-Disk)

Stores compressed representations in memory for graph navigation, while retrieving full-precision vectors from NVMe SSDs, enabling billion-scale search on a single node.

---

5. Production Vector Database Comparison Matrix

---

6. Hybrid Search: Combining Dense Vectors with Sparse Lexical BM25

Pure vector search can fail on exact part numbers, code snippets, or proper nouns (e.g., CVE-2024-3094 or x86_64). Modern production deployments use Hybrid Search combining dense neural embeddings with sparse BM25 keyword matching, merged via Reciprocal Rank Fusion (RRF):

$\text{RRF Score}(d\in D)={\sum }_{m\in M}\frac{1}{k+r_m(d)}$

---

7. Conclusion & Architectural Best Practices

1. Normalize vectors upfront so Dot Product can replace costly Cosine trigonometric calculations.
2. Apply pre-filtering / metadata filtering inside the graph traversal rather than post-filtering discarded results.
3. Use Hybrid Search with Cross-Encoder Reranking for maximum RAG accuracy in mission-critical applications.