A team reads that a newer embedding model tops the benchmarks, changes one line of config, and redeploys. Ingestion keeps working, no errors fire, latency looks normal. Two days later support tickets say the assistant is 'confidently wrong' about documents it answered perfectly last week. Nothing crashed: new documents were embedded by the new model, old ones by the old model, and every query now compares vectors that live in two different coordinate systems. The similarity scores are still numbers between 0 and 1. They just mean nothing.
Why can't you just swap the embedding model in a live RAG system?
Because an embedding model defines its own vector space, and distances only mean something between vectors from the same model. OpenAI's embeddings guide lists default sizes of 1536 dimensions for `text-embedding-3-small` and 3072 for `text-embedding-3-large`, so a store built on one cannot even accept the other. Even when dimensions happen to match, two models place the same sentence at unrelated coordinates, so cosine similarity between a new query vector and an old document vector is arbitrary.
This is the failure mode behind many 'the chatbot got worse' reports, and it is a cousin of the problems in why RAG chatbots return wrong answers. The fix is structural: treat the embedding model as part of the index's schema, version it, and never mix versions in one index.
What does re-indexing actually cost?
The embedding API line is usually the smallest number. OpenAI's guide puts `text-embedding-3-small` at roughly 62,500 pages per dollar and `text-embedding-3-large` at roughly 9,615 pages per dollar, assuming about 800 tokens per page. A 100,000-page corpus therefore costs on the order of a couple of dollars to tens of dollars to embed once, which is trivial.
At scale the picture changes. Gabriel Anhaia's migration write-up works through a 40M-chunk corpus averaging 300 tokens: roughly $10K of embedding inference at a premium model's rates, a 56-hour backfill at a throttled 200 chunks per second, storage that roughly doubles during dual-write, and p99 query latency of 1.3 to 1.6 times the old index on higher-dimensional vectors. Those figures come from one author's worked example, so treat them as an order-of-magnitude guide, not a quote. The hidden costs are parsing, replicas, reranker retraining, evaluation runs and the engineering time to do it safely.
What is the safe migration pattern?
A four-step dual-index approach keeps the old index live as the source of truth until the new one has earned the traffic:
- **Dual-write.** Every new or updated document is embedded by both models and written to both indexes. The old index stays authoritative; failures writing to the new one are logged and retried, never surfaced to users.
- **Shadow backfill.** Re-embed the historical corpus into the new index at a throttled rate, with a persisted cursor so the job is resumable and crash-safe. Use a stable sort key so chunks aren't skipped when documents change mid-run.
- **A/B evaluation in shadow.** Production queries still get answers from the old index while a fraction also query the new one silently, logging both result sets. Users see no change, and you accumulate real comparison data.
- **Slice-based cutover.** Route traffic per slice (tenant, document type, language) so a regression in one segment can be rolled back without reverting everyone.
How do you detect a recall regression without labelling every query?
Track overlap@k: the fraction of the top-k chunks that the old and new retrievers agree on, over a rolling window of live queries per slice. Anhaia's example alerts when mean overlap falls below 0.65 across 1,000 queries. Low overlap does not prove the new model is worse, since it may be better, but it tells you exactly which slices to review by hand.
Pair that with a labelled golden set for the slices that matter most, using the recall@k and faithfulness method in our RAG evaluation guide. Overlap finds where behaviour changed; the golden set decides whether the change is good.
Which implementation details cause the silent bugs?
- **Normalization mismatch.** OpenAI's guide notes its embeddings are normalized to length 1, which is why dot product equals cosine similarity. If you shorten dimensions manually, you must re-normalize, or scores drift. Apply the identical normalization at write and query time.
- **Metric mismatch.** Switching from cosine to dot product changes rankings unless vectors are unit length. Set the index metric to match the model's documentation.
- **Dimension shortening as a lever.** The same guide says a `text-embedding-3-large` vector shortened to 256 dimensions can outperform an unshortened `text-embedding-ada-002` at 1536 on MTEB, a real way to cut storage and latency, but re-run your own evaluation before trusting a public benchmark.
- **Stale reranker data.** After cutover, the retriever surfaces different chunks, so any reranker training or tuning data built on the old retriever's candidates needs rebuilding. See hybrid search with BM25, vectors and reranking.
- **Chunking changes riding along.** Change one variable at a time. Migrating the model and the chunking strategy in one release makes any regression impossible to attribute.
How AIBOOTSTRAPPER helps
Retrieval quality is the product in the RAG systems we build. For ComplyNexus, the Hong Kong compliance platform, the challenge was a team drowning in changing regulations tracked in spreadsheets; we built a RAG engine that monitors regulatory sources, maps new rules to the client's control library and keeps audit trails, which delivered 92% less manual review time and a regulatory-change turnaround cut from three weeks to two hours (see case studies). A system like that has to survive model upgrades without an auditor ever seeing a change in answers, which is why versioned indexes and evaluation sets are part of the design from day one.
If you are planning an embedding or vector-store change and want it scoped against your corpus and traffic, see our AI product development services or book a call.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.