Fine-Tuning Embedding Models for Domain-Specific Retrieval

Off-the-shelf embedding models work well for general-purpose retrieval. But when your domain has specialized terminology — legal contracts, medical records, financial filings, internal codebases — general models leave significant retrieval quality on the table. Fine-tuning can close that gap, but only if done correctly.

When Fine-Tuning Is Worth It

Before investing in fine-tuning, verify that the problem is actually the embedding model:

1. Check your chunking first. Bad chunks produce bad embeddings regardless of the model. Fix chunking before fine-tuning.
2. Try a larger model. Switching from all-MiniLM-L6-v2 (384 dims) to bge-large-en-v1.5 (1024 dims) often closes the gap without any training.
3. Measure the baseline. You need retrieval metrics (recall@k, MRR) on a test set before you can claim fine-tuning helped.

Fine-tuning makes sense when:

• Your domain has vocabulary that doesn't appear in general training data (proprietary terms, abbreviations, jargon)
• Semantic similarity in your domain differs from general English (in legal text, "reasonable" and "unreasonable" are semantically close in general embeddings but should be far apart for contract analysis)
• You have at least 1,000 labeled query-document pairs (or can generate them)

Training Data Preparation

The quality of your training data determines the ceiling of your fine-tuned model. You need pairs of (query, relevant_document) and ideally (query, relevant_document, irrelevant_document) triplets.

Generating Training Pairs

If you don't have labeled data, generate it:

Hard Negative Mining

Hard negatives are documents that look relevant to the query but aren't. They're critical for training — without them, the model learns to distinguish relevant documents from obviously irrelevant ones (easy) but fails to distinguish relevant from almost-relevant (hard).

enriched_pairs.append({ "query": pair["query"], "positive": pair["positive"], "hard_negatives": hard_negatives, })

TripletLoss: Explicitly uses (anchor, positive, negative) triplets. More control over negative selection but requires careful margin tuning.

CachedMultipleNegativesRankingLoss: A memory-efficient variant that enables effective batch sizes of 65K+ by caching embeddings. This lets you train on a single GPU while getting the benefit of huge in-batch negative pools.

Practical recommendation: start with MultipleNegativesRankingLoss with batch size 128. If recall plateaus, switch to CachedMultipleNegativesRankingLoss for a larger effective batch size. Add explicit hard negatives if in-batch negatives aren't enough.

Training Recipe

Key hyperparameters:

• Learning rate: 1e-5 to 3e-5. Lower for larger models.
• Epochs: 1-5. Embedding models overfit quickly — monitor eval metrics and stop early.
• Batch size: As large as GPU memory allows. Larger batches = more in-batch negatives = better contrastive learning.
• Warmup: 10% of training steps. Prevents catastrophic forgetting of general knowledge in early steps.

Evaluation Metrics

Track these metrics on a held-out test set:

Before/After: Real Results

On a proprietary legal document retrieval task (12,000 documents, 500 test queries):

The biggest gains came from queries using domain-specific terminology. For general queries, the improvement was modest (5-8%). This is expected — fine-tuning teaches the model your domain's language, not general retrieval.

Matryoshka Embeddings

A practical consideration: fine-tuned models produce fixed-dimension embeddings. If you need to reduce dimensions later (for cost or storage), use Matryoshka Representation Learning during training:

This trains the model so that the first N dimensions are useful on their own. You can truncate embeddings to 256 dimensions at search time with only a small recall drop, cutting storage by 75%.

Deployment Considerations

After fine-tuning, you need to re-embed your entire corpus with the new model. Plan for this:

• Track embedding model versions in your vector store metadata
• Build a re-indexing pipeline that can run incrementally (embed new/changed docs) or fully (re-embed everything)
• Run A/B tests comparing retrieval quality between old and new embeddings before cutting over
• Keep the old index available for rollback

Fine-tuning embedding models is high-leverage work when the domain justifies it. A few thousand training pairs and a few hours of GPU time can produce retrieval improvements that would require fundamental architecture changes to achieve otherwise. But measure first, tune second, and always have a baseline to compare against.