Do You Need a Custom Embedding Model?

General embedding models are trained on broad web text. They are very good at knowing that “car” and “automobile” are close together. They are much worse at knowing that, in your support system, “ORA-01555” and “snapshot too old” are the same problem, or that in your codebase reconcile_ledger and “nightly balance job” refer to the same thing.

This is a vocabulary problem, and fine-tuning fixes it directly. Contrastive fine-tuning takes pairs of (query, correct document) from your own data and pulls their vectors closer together while pushing unrelated documents away. The model does not get smarter in general. It gets better at one thing: mapping the way your users phrase questions onto the way your documents phrase answers. That is the entire job of a retrieval embedding, which is why the lift can be large even from a small model.

A concrete scenario, and the one used as the worked example below: a support-ticket RAG system. Users write tickets in messy, abbreviated language (“db conn pool exhausted after upgrade”). The knowledge base is written in formal product language (“Connection limits in Managed Databases”). A general model sees little overlap between those two strings. A model fine-tuned on six months of resolved tickets, where each ticket is paired with the article that resolved it, sees them as near neighbors.

• Embedding: a list of numbers that represents the meaning of a piece of text. Similar text gets similar numbers.
• Knowledge base: the collection of documents, articles, or chunks of text you’re searching over. In the example below, it’s a set of support articles.
• Vector database: a database built to store embeddings and quickly find the ones closest to a given query. is a vector database built into Postgres.
• Recall@10: out of all your test queries, the share where the correct document showed up somewhere in the top 10 results. Higher is better.
• MRR (mean reciprocal rank): how close to the top of the results the correct document lands, averaged across queries. A correct answer in position 1 scores higher than one in position 5.
• Contrastive fine-tuning: training a model on pairs of “this query matches this document” so it learns to place matching pairs closer together in vector space.

Setup. Here is the comparison you should run for your own system, worked through for the support-ticket scenario above: 10,000 knowledge-base chunks, 1,000 held-out test queries (queries set aside and never shown to the model during training, used only to check its answers) with labeled correct documents, and 50,000 training pairs (each one a real support question matched with the article that resolved it) pulled from resolved tickets. A quick note on where these numbers come from.

The recall and MRR rows (defined above: how often the right answer shows up, and how close to the top it lands) are typical results other teams have reported after running this kind of fine-tune. The dimension, storage, and cost rows are just facts: they come straight from the model’s specs and each provider’s listed prices, so those hold regardless of your data. If you want a number for your own system, you’d need to run your own test, which the framework at the end of this piece walks through.

*Assumes 500 tokens per chunk at the standard $0.13/M rate. The Batch API halves this to ~$325 if you can wait up to 24 hours; the same assumptions are used in our RAG Pipeline Cost Breakdown so the two pieces stay comparable.

Three things are worth being honest about in this table.

First, the fine-tuned model wins on domain queries by a wide margin, on the order of 12 points of recall@10 against a much larger API model. That is the vocabulary gap closing, and it is consistent with what published fine-tunes on similar corpora report. Second, expect to lose a little on general-English queries. Fine-tuning trades breadth for depth. If your query mix is mostly general questions, the fine-tune does not clear the bar, and you should stop here. Third, the latency and storage wins have nothing to do with fine-tuning. They come from running a small model yourself: 768 dimensions instead of 3,072 means a quarter of the pgvector storage and index size, and no network round trip to an API at query time. You get those benefits from self-hosting bge-base even without fine-tuning it.

The training run itself is small. bge-base-en-v1.5 has 109M parameters. Fine-tuning it on 50,000 pairs takes around 40 minutes on a single H100 GPU Droplet, which bills by the hour, so the run costs about $3.40. It also fits on a cheaper card like an RTX 4000 Ada, where the same run costs under a dollar. The expensive part of this project is never the GPU. It is collecting good query-to-document pairs.

This section exists to show the run is small and reproducible, not to be a tutorial. The full walkthrough belongs in the docs; this is the reference implementation for the comparison above.

Training data is a JSONL file of positive pairs pulled from resolved tickets:

{"query": "db conn pool exhausted after upgrade", "positive": "Connection limits in Managed Databases: each plan tier has a maximum connection count..."}
{"query": "how do i rotate the ca cert", "positive": "Rotating certificates: Managed Databases uses a per-cluster CA that can be regenerated..."}

The code below is about 30 lines, using the sentence-transformers library. It uses a training method called MultipleNegativesRankingLoss, which works like this: for each query in a training batch, its correct document is the right answer, and every other document in that same batch is automatically treated as a wrong answer. That’s the whole trick, and it’s why you only need to collect the correct pairs. The wrong examples come for free from whatever else happens to be in the batch.

from datasets import load_dataset
from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
)
from sentence_transformers.losses import MultipleNegativesRankingLoss

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

dataset = load_dataset("json", data_files="pairs.jsonl", split="train")
dataset = dataset.rename_columns({"query": "anchor", "positive": "positive"})

args = SentenceTransformerTrainingArguments(
    output_dir="bge-base-support-ft",
    num_train_epochs=2,
    per_device_train_batch_size=64,   # larger batches = more in-batch negatives
    learning_rate=2e-5,
    warmup_ratio=0.1,
    bf16=True,
)

trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    loss=MultipleNegativesRankingLoss(model),
)
trainer.train()
model.save_pretrained("bge-base-support-ft/final")

Evaluation uses the InformationRetrievalEvaluator built into the same library, which reports recall@k and MRR directly against the held-out query set. Storage and search run on pgvector on Managed Postgres with an HNSW index; nothing about a fine-tuned model changes how you store or query it, so the setup is exactly what the pgvector docs describe.

The one place the model choice does show up in Postgres is size. At 768 dimensions, 10 million chunks need roughly 31 GB of vectors plus index. At 3,072 dimensions, the same corpus needs roughly 123 GB, which pushes you into a larger database plan before you have stored a single row of application data. That difference recurs every month, unlike the one-time training cost.

If you are deciding where to run this, the honest comparison looks like this.

Together AI and Fireworks both offer managed fine-tuning, but their supported-model lists are generative models: Llama, Mistral, and Qwen families for chat, vision, and reasoning. As of this writing, neither lists embedding models as fine-tunable; embeddings are inference endpoints only. That matters here because the whole point of this piece is training the embedding model, not serving it.

Baseten’s training docs describe two paths, Loops for supervised fine-tuning and Training Jobs for running your own training code, both of which produce checkpoints you can deploy straight to their inference stack. On the serving side, Baseten Embeddings Inference is a dedicated engine built for embedding, reranking, and classification models, and Baseten reports over 2x the throughput and about 10% lower latency than the next-best solution they tested. If you want a managed pipeline from training job to serving endpoint, it is a fair option to evaluate, at a platform premium over raw compute.

Modal is the closest comparison for the bring-your-own-training-job approach, and they have published this exact workflow: fine-tuning an open embedding model to beat proprietary APIs. The difference is the shape of the compute. Modal gives you serverless functions with per-second GPU billing, which is a good fit for bursty, parallel experiment sweeps.

A GPU Droplet is a plain VM: you SSH in, run the script above, and pay by the hour, with your vectors living in the same Postgres that already holds your application data. For a periodic retraining job and a production database you already operate, the VM plus Managed Postgres shape is simpler. For hundreds of parallel hyperparameter runs, Modal’s shape is better. Both are defensible; pick based on how often you retrain.

Worth naming DigitalOcean directly here too, since it’s the infrastructure behind the numbers above. GPU Droplets are raw compute, not a managed fine-tuning product: you get a VM with the GPU drivers already installed, and you run your own training script on it, which is exactly what this piece’s training run does. There’s no “submit a fine-tuning job” API the way Baseten’s Loops or Together’s fine-tuning endpoint work for chat models. What you get instead is control over the whole stack around that VM: any training script you want, Managed Postgres with pgvector for storing the resulting vectors, and a training run and a production database that can sit in the same account without going through a separate platform.

OpenRouter does not belong in this comparison. It is a routing layer over inference APIs and does not train anything, so it can serve you someone’s embedding model but cannot help you make your own.

Skip the fine-tune, without guilt, in three situations.

Low query volume. The training cost is small, but the operational cost is not: someone has to own the pairs pipeline, the eval set, and the retraining schedule. If your system serves a few hundred queries a day and retrieval is roughly working, that ownership cost never pays for itself.

Fast-moving vocabulary. If your domain terms change monthly (new product names, new error codes), your fine-tuned model is always slightly stale, and retraining cadence becomes a recurring tax. A general model plus good keyword or hybrid search often ages better here.

No labeled pairs and no way to get them. The model is only as good as the query-to-document pairs you train on. If you cannot mine them from logs, tickets, or click data, and you would be generating synthetic pairs from the documents themselves, expect a much smaller lift than the table above. Synthetic pairs teach the model your documents’ vocabulary, not your users’ vocabulary, and the gap between those two is the thing you were trying to fix.

Measure before deciding. Build a test set of 200 to 500 real queries with labeled correct documents, run your current embedding model, and compute recall@10. Then:

Fine-tune if all three hold: recall@10 on domain queries is below about 80%, you can assemble at least a few thousand real query-to-document pairs, and someone on the team can own a retraining cycle (quarterly is enough for most domains).

Stay with the API if any of these hold: your queries are mostly general English, your vocabulary churns faster than you would retrain, or you have no source of real pairs.

Self-host without fine-tuning if what you actually need is lower latency or smaller vectors. A stock open model on your own GPU gets you both, and you can add the fine-tune later once you have the pairs.

This isn’t a hard call to make. Build the eval set, spend an afternoon measuring, and you’ll know which side you’re on. That’s a lot cheaper than guessing wrong, either by locking into an API bill you didn’t need to pay or by building training infrastructure you didn’t need to build.