Embedding Models

Embedding models are neural networks that convert text (or other data) into dense vector representations, optimized so that semantically similar inputs map to nearby points in vector space.

What It Really Means

Not all embedding models are created equal. The choice of embedding model is often the single biggest lever for quality in semantic search and RAG systems — more impactful than the choice of vector database, reranking algorithm, or even the LLM itself.

Embedding models differ along several dimensions:

• Architecture: BERT-based, T5-based, or custom architectures
• Training objective: Contrastive learning, masked language modeling, or instruction-tuned
• Dimensions: 384 to 3072 dimensions per vector
• Max input length: 512 to 8192 tokens
• Language support: English-only vs multilingual
• Task specialization: Retrieval, classification, clustering, or general-purpose

The MTEB (Massive Text Embedding Benchmark) leaderboard ranks models across diverse tasks. But leaderboard performance does not always predict performance on your specific domain. A legal embedding model will outperform the MTEB leader on legal search, even if it scores lower on the benchmark.

The key insight is that embeddings are a learned compression of meaning. Different models learn different compressions, and the best compression depends on what aspects of meaning matter for your task.

How It Works in Practice

Model Categories

API-Based Models:

• OpenAI text-embedding-3-small (1536d) — good balance of cost and quality
• OpenAI text-embedding-3-large (3072d) — highest quality from OpenAI, supports dimension reduction
• Cohere embed-english-v3.0 (1024d) — strong retrieval performance
• Voyage AI voyage-3 (1024d) — optimized for code and technical text

Open-Source Models:

• BAAI/bge-large-en-v1.5 (1024d) — top open-source general-purpose model
• intfloat/e5-mistral-7b-instruct (4096d) — instruction-tuned, highest quality
• sentence-transformers/all-MiniLM-L6-v2 (384d) — fast, lightweight, good for prototyping
• nomic-ai/nomic-embed-text-v1.5 (768d) — long context (8192 tokens), Matryoshka support

Model Selection Decision Tree

1. Budget: API models cost $0.02-0.13 per million tokens. Open-source models are free but need GPU hosting.
2. Latency: Smaller models (384d) are 5-10x faster than large ones (4096d).
3. Domain: If your domain is specialized (legal, medical, code), test domain-specific models.
4. Scale: At >100M documents, storage costs matter — lower dimensions save significantly.
5. Multilinguality: If you need cross-lingual search, choose multilingual models.

Benchmarking on Your Data

Implementation

Production Embedding Pipeline

Trade-offs

Small Models (384-512d)

• Fast inference, low storage
• Good for prototyping and cost-sensitive applications
• Lower quality on nuanced semantic distinctions
• Examples: all-MiniLM-L6-v2, text-embedding-3-small

Large Models (1024-4096d)

• Best quality, captures fine-grained semantics
• Higher latency and storage costs
• May be overkill for simple classification tasks
• Examples: bge-large-en-v1.5, e5-mistral-7b-instruct

API vs Self-Hosted

• API: No GPU management, consistent quality, per-token pricing
• Self-hosted: Fixed cost, data privacy, customizable, but requires GPU infrastructure

Common Misconceptions

• "The model with the highest MTEB score is the best choice" — MTEB averages across many tasks. Your task may weight different subtasks. Always benchmark on your own data.

• "You can switch embedding models without re-indexing" — Different models produce incompatible vector spaces. Switching models requires re-embedding your entire corpus. Plan for this in your architecture.

• "Fine-tuning an embedding model requires massive data" — Contrastive fine-tuning with as few as 1,000 query-document pairs can significantly improve domain-specific performance.

• "Embedding models and LLMs are the same thing" — Embedding models (encoder-based) produce fixed-size vectors from variable-length input. LLMs (decoder-based) generate text. Different architectures, different purposes.

• "More dimensions always means better embeddings" — Beyond a point, additional dimensions add noise. Matryoshka Representation Learning shows that the first 256 dimensions of a 3072-dim embedding capture most of the information.

How This Appears in Interviews

Embedding model selection is a practical AI engineering interview topic:

• "How would you choose an embedding model for a legal document search system?" — discuss domain specificity, benchmarking methodology, and the MTEB leaderboard. See our guides on AI engineering.
• "Your search quality dropped after switching embedding models. Why?" — the new model may not be compatible with your existing index, or may be weaker on your specific domain.
• "How do you handle embedding model updates in production?" — discuss shadow indexing, gradual migration, and quality monitoring. See our interview questions.

Related Concepts

• Vector Embeddings — What embedding models produce
• Semantic Search — Primary use case for embedding models
• RAG — Embedding models power the retrieval step
• Transformer Architecture — The architecture behind embedding models
• Fine-Tuning vs RAG — Fine-tuning embedding models for domain adaptation
• Algoroq Pricing — Practice model selection interview questions