Cutting a 24GB RAG Index Down to 3GB with Turbovec, a Rust Take on Google's TurboQuant

The pager went off at 2:14am on a Tuesday because our internal RAG search service kept getting OOM-killed on the box that hosted the vector index. We'd grown the corpus from about 800k chunks to 4.2 million over three months as more teams piped their docs into it, and nobody had gone back to check what that did to memory. The index was a flat float32 array sitting in RAM next to the embedding model and the API process, and at 1536 dimensions per chunk, 4.2 million vectors works out to roughly 24GB just for the vectors themselves, before you count the graph structure, the id map, or anything else running on that box.

My first instinct was to reach for FAISS's IndexPQ, which is the standard answer to "my vectors are too big." It worked, recall was acceptable, but I didn't love babysitting a k-means training step that needed a representative sample of the corpus, needed re-running if the embedding distribution shifted, and needed careful handling every time we added a new tenant's docs before the codebook had seen anything like them. Around the same time I ran into turbovec, a Rust vector index with Python bindings built on top of Google Research's TurboQuant paper, and the pitch was specifically "no training step, no codebook drift." That was enough to make me actually try swapping it in instead of just reading the README and moving on.

This post is the migration as it actually happened: the math that got us there, the integration, one gotcha in the quantizer's calibration step that cost me an afternoon, some rough recall numbers from our own subset, and where I think this tool genuinely doesn't fit yet. It's not a benchmark press release — it's what happened when I put it in front of real traffic.

The 24GB wake-up call

Once I actually did the arithmetic it was obvious the float32 index was never going to survive another quarter of growth. 4.2 million vectors at 1536 dimensions, four bytes per float, comes out to roughly 24GB just for the raw vector data — and that's before FAISS's own overhead for its flat index wrapper, before the id-to-metadata lookup table, and before the embedding model itself, which also wanted a few gigabytes of headroom on the same box.

Re-embedding at a smaller dimension wasn't really on the table; we didn't own the embedding model and swapping it would have meant re-indexing every downstream consumer's assumptions about vector shape, not just ours. Sharding across multiple boxes was the "correct" distributed-systems answer, but it also meant standing up a routing layer, dealing with cross-shard merge logic for top-k, and doubling our on-call surface area for a search feature that, frankly, wasn't important enough to justify that investment yet. Quantization was the only lever that let a single box keep doing the job.

What TurboQuant actually does differently

Product quantization, the thing FAISS's IndexPQ implements, splits each vector into sub-vectors and runs k-means on each subspace to learn a codebook from your actual data. That's why it needs a training pass, and why the codebook can go stale as your data distribution shifts — the buckets it learned on last quarter's embeddings might not fit this quarter's.

TurboQuant, published by Google Research and presented at ICLR 2026, takes a different route. It applies a random orthogonal rotation to every vector first. After that rotation, each coordinate independently follows a Beta distribution that converges to a Gaussian as dimensionality grows — and crucially, this holds regardless of what your original data looked like, because the rotation is what induces the distribution, not the data itself. Once you know the distribution in advance, you can precompute the optimal bucket boundaries with the Lloyd-Max algorithm before you've seen a single real vector, then just quantize each coordinate against that fixed codebook.

The tradeoff for that convenience is a small, fixed distortion penalty — the paper works out that Lloyd-Max quantization against the asymptotic Beta distribution lands within roughly a 2.7x constant factor of the Shannon-optimal distortion rate. In practice that showed up as recall a percentage point or two off from a well-tuned, well-trained PQ codebook on some datasets, and slightly ahead on others, which tracked with what I saw in our own numbers below.

Swapping FAISS PQ for turbovec's IdMapIndex

The crate — maintained by a developer named Ryan Codrai — ships as both a Rust crate and a Python package with PyO3 bindings, and since our search service's hot path is already Rust, I went with cargo add turbovec directly instead of round-tripping through Python. Our old wrapper around FAISS looked roughly like this, including the training call we had to remember to run before any inserts would actually quantize well:

use faiss::index::io::write_index;
use faiss::{Index, index_factory, MetricType};

let mut index = index_factory(1536, "PQ192x8", MetricType::InnerProduct)?;
if !index.is_trained() {
    // needs a representative sample, ideally >= 30x the codebook size
    index.train(&training_sample)?;
}
index.add(&vectors)?;
write_index(&index, "docs.index")?;

The training sample was a recurring source of bugs on its own — forget to refresh it after onboarding a new tenant with a very different document style, and recall on that tenant's queries would quietly tank. Swapping in turbovec's IdMapIndex removed that step entirely, and gave us stable external ids across deletes for free, which we'd previously bolted on ourselves with a separate id-to-slot table:

use turbovec::IdMapIndex;

let mut index = IdMapIndex::new(1536, 4).unwrap();
index.add_with_ids(&vectors, &doc_ids).unwrap();

let (scores, ids) = index.search(&query_batch, 10);

index.remove(stale_doc_id);
index.write("docs.tvim").unwrap();

No training call, no separate id map to maintain, and the bit width — 4 in our case — is just a constructor argument instead of a string you have to get exactly right in a factory string like "PQ192x8". That last part sounds trivial but it removed an entire category of "which PQ config string did we actually use for this index file" archaeology six months later.

The calibration-freeze gotcha that cost me an afternoon

Not everything about "no training step" is free, though. Turbovec includes an extension on top of the base TurboQuant algorithm — the README calls it TQ+ — that fits a per-coordinate shift and scale during the very first add() call, to correct for the fact that the asymptotic Beta distribution the base algorithm assumes doesn't perfectly match real embeddings at finite dimensions, especially at low bit widths. That calibration is computed once and then frozen for the lifetime of the index.

I found this out the hard way because our migration script seeded the new index with a small batch of synthetic test vectors first, just to sanity-check the API before pointing real traffic at it. Those test vectors weren't remotely representative of our actual embedding distribution, and because calibration locks in on that first add, every real vector we indexed afterward got quantized against boundaries calibrated for garbage data. Recall on a validation set dropped by a few points and it took me an embarrassingly long time to connect it back to that seed batch instead of assuming it was a bug in my search code.

Once I fixed the migration script to build the real index straight from a representative first batch, recall came back in line with what I'd expect from the paper's numbers. It's a sharp edge, but it's a documented one, and it only bites you once you know to look for it.

Recall and memory numbers from our own run

I didn't reproduce the project's full benchmark suite — that would take longer than the migration itself — but I did run a quick, unscientific comparison on a 500k-vector subset of our real corpus, matching FAISS PQ's bit budget to turbovec's as closely as the two APIs allow. Take these as directional, not a rigorous study; your mileage will vary with your embedding model and dimensionality.

The 4-bit config ended up being our sweet spot: smaller than FAISS PQ's 8-bit codebook and marginally better recall on our data, with no training pass to maintain. The 2-bit config was tempting for the extra compression but the recall drop was a bit steeper than I wanted for a service where wrong answers erode trust fast. We landed on 4-bit across the board and kept 2-bit in reserve for a lower-priority archival tier where recall matters less than fitting the whole thing on one box.

Hybrid filtering for multi-tenant RAG

Our search service is multi-tenant, which means every query also needs a tenant scope, and normally that means either pre-filtering with a SQL query and passing IDs through, or post-filtering results and hoping enough survive to fill the top-k. Turbovec's search() takes an allowlist directly, and the filtering happens inside the SIMD scoring kernel rather than after the fact.

use turbovec::IdMapIndex;

// Stage 1: narrow to this tenant's document ids via our existing metadata store.
let allowed_ids: Vec = doc_store.ids_for_tenant(tenant_id)?;

// Stage 2: dense search restricted to that allowlist, no over-fetch-then-filter.
let (scores, ids) = index.search_filtered(&query_vector, 10, &allowed_ids);

The part that mattered for us in practice is that a narrow allowlist actually gets cheaper, not more expensive — blocks of vectors with no allowed ids get skipped before any scoring work happens, instead of the kernel computing scores for the whole index and discarding most of them. For our smaller tenants, where a single customer's docs might be a tiny fraction of the shared index, that turned what used to be our slowest queries into some of our fastest ones.

When I'd still reach for something else

I don't think turbovec is the right default for every vector search problem, and I'd be doing you a disservice if I pretended otherwise. It's fundamentally a quantized flat index — there's no graph structure like HNSW underneath it, so search cost still scales with corpus size even though each comparison got cheaper. For a lot of workloads that's a completely fine tradeoff, but not for all of them.

• If you're north of 50-100 million vectors and need sub-millisecond latency, you probably still want an ANN graph index or IVF-style clustering on top of quantization, not a flat scan no matter how fast the kernel is.
• If you need the index sharded and replicated across multiple machines out of the box, that's not what this crate does — you'd be building that orchestration layer yourself, same as you would with raw FAISS.
• If your team isn't comfortable running a young, single-maintainer crate in production, that's a legitimate reason to wait, regardless of how good the algorithm is — bus factor is a real operational risk, not a theoretical one.
• If your embeddings are very low-dimensional (think under 100 dims), the asymptotic Beta assumption the algorithm leans on is looser, and the recall gap versus a well-trained PQ codebook narrows or can even flip in PQ's favor.

None of that stopped me from shipping it, because our situation — a single box, a growing multi-tenant corpus, an on-call rotation that didn't want to own a codebook training pipeline — matched what it's good at almost exactly.

The migration bought us headroom: the index that used to eat 24GB now sits under 3GB at 4-bit, and the box that used to get OOM-killed under load hasn't paged me since. But I'm treating that as a starting point, not a finish line — I've got a shadow-query job comparing float32 and quantized recall on live traffic on a rolling basis, because a benchmark on last month's data doesn't tell you what happens when the corpus shifts six months from now. If you're staring down a similar memory problem on a single-box deployment and you're already comfortable operating Rust in production, it's worth an afternoon to try before you reach for a managed vector database or a bigger box — just build your real index from real data on the very first insert, and don't let a smoke test anywhere near it.

Sources: Ryan Codrai — turbovec on GitHub, Zandieh, Daliri, Hadian, Mirrokni — TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874)