Changing a RAG Embedding Model? Plan the Vector Migration Before You Switch
An embedding-model change affects both stored document vectors and incoming query vectors. Use this staged plan to inventory your setup, rebuild and validate an index, and cut over with a rollback path.
Treat an embedding change as a data migration
A RAG system typically embeds source content for storage and embeds a user’s query for retrieval. Changing the embedding model changes an important part of that process. Do not assume the vectors already stored in an index will work correctly with queries embedded by a different model or configuration.
The safe operational rule is to keep document and query embedding behavior aligned, and to verify any claimed compatibility before relying on it. Similarity scores from two different model setups should not be treated as directly comparable unless you have tested that comparison for your system.
This is why a model setting alone is not a migration plan. NVIDIA’s RAG guidance says documents must be re-ingested after changing the embedding model or dimensions following ingestion for accurate retrieval. Qdrant documents a migration approach that copies data to a new collection and re-embeds vectors while searches continue.
- Identify exactly which model and configuration produced the current document vectors.
- Plan how both stored documents and incoming queries will use the new embedding setup.
- Keep the existing working index available until the replacement has passed your checks.
Inventory the whole retrieval path before rebuilding
Start by recording how data moves from source content to retrieved passages. An index rebuild can be incomplete even when the embedding call succeeds: the source set may have changed, metadata may be omitted, or the query path may still use the previous configuration.
Write down the current and proposed settings in one place. Include model identifiers and any configuration that affects embedding output, expected vector dimensions, text extraction and chunking steps, metadata fields, and retrieval settings. Record how documents are identified and updated so you can compare the rebuilt index against the intended corpus.
Treat preprocessing as part of the embedding setup. If document text is split or normalized differently during the migration, retrieval changes may reflect those differences as well as the model change. Make such changes explicit rather than bundling them into an untraceable switch.
- Corpus: source locations, document identifiers, expected item or chunk counts, and exclusions.
- Embedding: current and target model, configuration, dimensions, and query-embedding path.
- Processing: extraction, chunking, text transformations, and handling of updates or deletions.
- Index and retrieval: collection or index name, metadata and filter fields, and search settings.
- Operations: the rebuild job, access permissions, available storage, and a way to inspect failures.
Choose a migration shape that preserves a working state
The right migration approach depends on your system’s ability to hold more than one index, the size and update rate of the corpus, and how much interruption your application can tolerate. The supplied vendor guidance supports a new-collection migration pattern, but it does not establish one strategy as best for every deployment.
A full rebuild can be straightforward when you can schedule a maintenance period and retain the old index until the new one is checked. A parallel index is useful when you need to compare retrieval behavior or keep serving from the existing index during the rebuild. A limited-scope migration can help test your pipeline on a representative subset, but it is not proof that the full corpus has been rebuilt or validated.
Choose deliberately, document the trade-offs, and avoid deleting or overwriting the only working index as the first step. If the platform cannot support the approach you want, account for that constraint before starting.
- Use a full rebuild when a controlled interruption is acceptable and you can preserve the previous index.
- Use parallel indexes when you need continued access to the existing retrieval path during rebuilding and validation.
- Use a limited pilot to test the process, then separately plan and verify the complete migration.
- Confirm capacity, write behavior during rebuilding, and the method for directing queries to the intended index.
Re-embed consistently and track coverage
Build the new index using the target configuration for document embeddings, and configure query embedding to match before testing retrieval against that index. Confirm that the produced dimensions match what the index expects. A dimension mismatch is a concrete incompatibility to catch before routing live queries.
Keep a record of what was processed and what failed. Compare the rebuilt contents with the intended corpus using stable document or chunk identifiers where available. Reconcile missing items, duplicates, skipped records, and documents changed during the job instead of treating a completed process as proof of complete coverage.
If source content can change while rebuilding, decide how those changes reach the new index. Depending on your system, this may mean pausing writes, recording changes for later application, or processing updates into the destination as part of the migration. Test the chosen method rather than assuming the destination stays current.
- Use the same target embedding configuration for all document vectors in the new index.
- Use the corresponding query embedding configuration when querying that index.
- Check dimensions and confirm the index accepts the expected vector format.
- Track processed, failed, skipped, changed, and duplicate records.
- Reconcile source updates made during the rebuild before cutover.
Validate retrieval before sending users to the new index
Validation should test more than whether the new index accepts queries. Prepare a set of representative questions and expected source documents or passages before the rebuild. Include common queries, queries that depend on important metadata filters, and cases where the correct source is less obvious.
Run those checks through the same query-embedding and retrieval path your application will use. Inspect the retrieved passages and their metadata, not just a final answer: a plausible answer can hide a missing, irrelevant, or incorrectly filtered source. Compare the old and new paths as separate systems, and define what counts as an acceptable result for your use case instead of comparing raw similarity scores as if they were on a shared scale.
Check corpus coverage alongside retrieval quality. A strong result on a small test set does not establish that every intended document was indexed. Record test cases, outcomes, and unresolved differences so the cutover decision is reviewable.
- Confirm representative documents are present in the new index.
- Test expected retrieval for representative queries and inspect the actual passages returned.
- Verify metadata filters and any important access or scoping behavior used by retrieval.
- Check empty, ambiguous, and failure cases relevant to your application.
- Agree on acceptance criteria before testing and record any exceptions.
Cut over with a rollback path
Make the index switch a deliberate operational step. Record which index and embedding configuration the application uses, then change the query path as a coordinated pair. Routing queries to the new index while leaving query embedding on the old configuration can invalidate the work you just completed.
Keep the prior index and its configuration intact through the agreed observation period. Define who can initiate a rollback, what evidence would trigger one, and how the application will be pointed back to the previous working path. If data changes continue during or after cutover, include those changes in the rollback plan so returning to the old index does not silently lose newer content.
Only clean up the previous index after the new path is accepted, rollback is no longer required, and retention or governance requirements have been considered. A backup is useful for recovery planning, but it is not a substitute for a tested index switch or a clear rollback procedure.
- Switch the query embedding configuration and target index as one controlled change.
- Verify the live path with a small set of known queries immediately after switching.
- Keep the previous index available until the new path has met the agreed acceptance criteria.
- Document rollback authority, trigger conditions, steps, and treatment of intervening data changes.
- Set cleanup timing only after acceptance and retention checks.
Assign ownership and preserve the migration record
An embedding migration does not end when the index is switched. Assign an owner for the model and preprocessing configuration, another accountable role for index operations if needed, and a process for deciding when future model or dimension changes require a rebuild.
Keep the migration record with the index configuration: source corpus scope, model and dimensions, preprocessing choices, completion and failure counts, validation results, cutover time, rollback instructions, and cleanup decision. This makes later troubleshooting less dependent on memory and helps distinguish an embedding change from a corpus or retrieval change.
Hosting and infrastructure management can reduce operational work, but they do not decide which documents belong in the index, which users may retrieve them, or whether retrieval quality is acceptable. For example, Airbip manages deployment of catalog applications as Docker workloads and provides configurable backups; teams still need to plan the RAG data migration, access choices, validation, and rollback for their own setup.
- Name an owner for embedding configuration and one for index operations.
- Version or otherwise preserve the model, preprocessing, and retrieval settings used for each index.
- Retain a migration log and representative validation set for future changes.
- Review data access, retention, and backup choices independently of model configuration.
Frequently asked questions
Do I need to re-embed documents when changing the embedding model?
Plan to re-embed or re-ingest the documents for the new setup unless compatibility is explicitly supported and validated for your system. NVIDIA’s RAG documentation says documents must be re-ingested after changing the embedding model or dimensions following ingestion for accurate retrieval.
Can the old index stay available while a new one is built?
That depends on the vector database and deployment design. Qdrant documents a migration process that copies data to a new collection, re-embeds vectors, and allows searching during migration. Check the behavior and operational requirements of your own setup before relying on a similar approach.
Can I compare similarity scores from the old and new embedding models?
Do not assume the scores share a directly comparable scale. Compare retrieved passages against representative queries and agreed acceptance criteria, and treat each model and index configuration as a separate retrieval path unless you have validated score comparability.
Is a successful rebuild enough to approve cutover?
No. Check coverage, failures, metadata and filters, and retrieval results for representative queries. Then switch the query embedding configuration and index together, verify the live path, and retain a rollback option until the new setup is accepted.