LangSmith to RAGAS: converting an exported dataset
I moved an eval set from LangSmith into RAGAS last month and lost half a day to two
things that never raised an error: the question column came out empty, and the reference
column silently didn't exist.
Both had the same root cause. LangSmith and RAGAS disagree about what a dataset is.
The shape problem
RAGAS stores a dataset as columnar parallel arrays — one array per field, one entry per
sample, every array the same length. LangSmith stores one example per record, with the
actual content nested inside two free-form objects.
LangSmith RAGAS [{ { "inputs": {"question": "..."}, "user_input": ["..."], "outputs": { "retrieved_contexts": [["..."]], "contexts": [...], "response": ["..."], "answer": "..." "reference": ["..."] } } }] So the conversion is a reshape, not a find-and-replace. You are unstacking rows into
columns and renaming fields, and the second half is where things go quiet.
What a LangSmith example actually contains
The Example model in the langsmith SDK has these fields:
| Field | Type | Notes |
|---|---|---|
id | UUID | example identity |
dataset_id | UUID | which dataset it belongs to |
inputs | free-form dict | whatever your pipeline logged |
outputs | free-form dict | whatever your pipeline logged |
metadata | free-form dict | arbitrary, e.g. {"source": "Wikipedia"} |
created_at / modified_at | datetime | |
source_run_id | UUID or null | if built from a trace |
Note what's missing: inputs and outputs have no schema. LangSmith does not require
inputs.question or outputs.answer. The docs' own example uses exactly those names, so
everyone copies them — but a pipeline that logged {"query": ...} produces a perfectly
valid export that no RAGAS converter can read.
The field mapping
| LangSmith | RAGAS (SingleTurnSample) | Required |
|---|---|---|
inputs.question | user_input | Yes |
outputs.contexts | retrieved_contexts | For retrieval metrics |
outputs.answer | response | For answer metrics |
outputs.reference | reference | For reference-based metrics |
Four fields in, four fields out. Everything else has nowhere to go.
The six fields that get dropped
SingleTurnSample accepts exactly these: user_input, retrieved_contexts,
reference_contexts, retrieved_context_ids, reference_context_ids, response,
multi_responses, reference, rubrics, persona_name, query_style, query_length.
There is no generic metadata field. So on the way over you lose:
iddataset_id-
created_atandmodified_at source_run_idmetadata
That last one stings. If your LangSmith examples carry {"source": "Wikipedia"} or a
document ID, RAGAS cannot store it. Your options are to keep the original export and join
back on array index (which is preserved in order), or fold the values into one of the
named fields where it makes sense.
Anything in outputs that isn't contexts, answer or reference is dropped too —
including outputs.ground_truth, if that's what your pipeline called the reference answer.
Why the question column comes out empty
This is the failure that costs the most time, because nothing warns you.
A converter reads inputs.question. If your export says inputs.query or inputs.input,
the structure is still detected as LangSmith — inputs and outputs are both present —
but every user_input entry is empty. RAGAS will load it. The metrics will run. The
scores will be meaningless.
Two fixes: rename the key in the export before converting, or rename the column after.
Always check that user_input is non-empty before you trust any number downstream:
import json data = json.load(open("converted.json")) empty = [i for i, q in enumerate(data["user_input"]) if not str(q).strip()] if empty: print("empty questions at rows:", empty) A detail about split
LangSmith's ExampleCreate and ExampleUpdate accept a split field, so you can tag
examples as train / test when you write them. The Example read model does not
expose it.
Practically: don't count on splits surviving an export-and-convert round trip. If you
need them in RAGAS, carry them yourself and slice the dataset before evaluation.
Converting by hand
If every example has the four expected keys, this is enough:
import json from ragas import EvaluationDataset, SingleTurnSample from langsmith import Client client = Client() examples = client.list_examples(dataset_name="my-eval-set") samples = [] for ex in examples: inputs, outputs = ex.inputs or {}, ex.outputs or {} samples.append( SingleTurnSample( user_input=inputs["question"], retrieved_contexts=outputs["contexts"], response=outputs["answer"], reference=outputs.get("reference"), ) ) dataset = EvaluationDataset(samples=samples) This breaks the moment one example is missing a key — which is the usual reason to run
the JSON through a converter first and read the report instead of the traceback.
If you export from the UI you get JSONL, so read it line by line:
records = [json.loads(line) for line in open("dataset.jsonl")] The UI also offers CSV and OpenAI's fine-tuning format. RAGAS wants neither.
What you can evaluate afterwards
If your LangSmith examples never logged outputs.reference, the converted dataset has no
reference column, and these cannot run: Context Recall, Context Entities Recall, Answer
Correctness, Factual Correctness, Semantic Similarity, Noise Sensitivity.
These still work: Faithfulness, Response Relevancy, Aspect Critic, and
LLMContextPrecisionWithoutReference — which scores chunks against your own generated
answer rather than a gold one.
Worth knowing before you spend an hour generating references you may not need for a first
pass.
Doing it without the hand-editing
I built a converter for this. It takes RAGAS modern / current / legacy, TruLens or
LangSmith JSON, converts to whichever target you pick, and prints every field it
renamed, flattened, nested or dropped — so the six lost fields show up in a list
instead of disappearing.
It also flags the two failures that corrupt metrics silently rather than throwing:
unequal parallel arrays, and contexts stored as a string instead of a list of strings.
ragasconverter.com — runs entirely
in the browser, no upload, no signup.
Field names above are read from langsmith/schemas.py (ExampleBase, Example,
ExampleCreate) and
src/ragas/dataset_schema.py
(SingleTurnSample). Both projects change these between releases — I checked against
langsmith 0.15 and current RAGAS main, and you should pin your own versions before
trusting the table.