vecdb-bench

A Bun + TypeScript benchmark evaluating three embedded vector database stacks for hybrid semantic search, targeting code indexing/RAG and fantasy book RAG use cases.

Stacks Evaluated

Environment Variables

Copy .env.example to .env and fill in your API keys and model configuration. See .env.example for all required variables.

Methodology

• Datasets: 20 code snippets + 20 fantasy book passages, each with 20 queries and ground-truth relevance labels
• Search methods: vector-only, FTS-only, hybrid (RRF fusion), hybrid + reranker
• Metrics: Precision@5, Recall@5, MRR, NDCG@5, average latency, P95 latency, indexing time
• Latency measurement: Only DB query time is measured — embedding and reranker API latency is excluded so results reflect pure database performance
• Scoring: Combined score = 70% quality (weighted MRR/NDCG/Recall/Precision) + 30% performance (inverse latency)

Setup

git clone https://github.com/lemon07r/vecdb-bench.git
bun install
cp .env.example .env  # add your API keys and model config
bun run src/bench.ts

Options

Example with custom dimensions:

bun run src/bench.ts --dimensions 512
bun run src/bench.ts -d 256

Results

Models Used

• Embedding: Qwen/Qwen3-Embedding-0.6B (1024 dimensions) via SiliconFlow API
• Reranker: Qwen/Qwen3-Reranker-0.6B via SiliconFlow API

Code Search Dataset (20 docs, 20 queries)

Fantasy Books Dataset (20 docs, 20 queries)

Aggregate Scores

Results with 8B Models (4096 dimensions)

Models Used

• Embedding: Qwen/Qwen3-Embedding-8B (4096 dimensions) via Nebius API
• Reranker: Qwen/Qwen3-Reranker-8B via SiliconFlow API

Code Search Dataset (20 docs, 20 queries)

Fantasy Books Dataset (20 docs, 20 queries)

Aggregate Scores

8B vs 0.6B Comparison

• 8B models improve hybrid+rerank quality — Code Search achieves perfect 1.000 NDCG@5 across all engines (vs 0.981 with 0.6B)
• Fantasy Books vector search improves — 0.996 NDCG@5 with 8B vs 0.958 with 0.6B
• DuckDB latency increases significantly with 4096d vectors — ~50ms vector search vs ~12ms at 1024d
• SQLite and LanceDB handle 4× larger vectors well — latency increase is minimal (1–5ms range)
• Overall ranking unchanged — SQLite > LanceDB > DuckDB regardless of model size

Reranking Impact Analysis

Comparing hybrid search quality with and without the reranker, and across model sizes (averaged across all three engines):

Key observations:

• Reranking consistently improves quality — MRR jumps to perfect 1.000 in all cases, meaning the correct document is always ranked first after reranking.
• 8B without reranker ≈ 0.6B with reranker on Code Search — 8B hybrid achieves 0.960 NDCG@5 vs 0.6B hybrid+rerank at 0.981. The gap is small enough that for latency-sensitive applications, using a larger embedding model without reranking can be a viable alternative to a smaller model with reranking.
• Fantasy Books benefits less from reranking — even 0.6B hybrid without reranking already scores 0.988 NDCG@5, leaving little room for improvement. The natural language in fiction is easier to match than code.
• The biggest quality gain from reranking is on Code Search with 0.6B — NDCG@5 jumps from 0.952 → 0.981 (+3.0%), while 8B reranking pushes it to a perfect 1.000.
• Diminishing returns at 8B — the reranker adds less value with 8B embeddings since the base retrieval is already stronger, especially on Fantasy Books where 8B hybrid+rerank (0.994) actually scores marginally lower than 8B hybrid alone on some engines.

Key Takeaways

• Quality is virtually identical across all three engines (~0.97 MRR, ~0.97 NDCG@5 on hybrid+rerank). The reranker equalizes any quality differences.
• SQLite is the fastest — 0.1–1.0ms per query, 4–5ms indexing. ~40× faster than DuckDB, ~4× faster than LanceDB.
• LanceDB has the best hybrid search quality on Fantasy Books — perfect 1.000 NDCG@5 on hybrid without reranker.
• DuckDB is slowest at 12–28ms per query and 211–230ms indexing, but still fast in absolute terms.

Recommendations