An order document in MongoDB does not stay still. It is placed, then shipped, then delivered, and each step rewrites the same document in place. A Databricks table loaded from last night's export holds only one of those states, and closing that gap is what most MongoDB to Databricks pipelines are really for.

The common ways to move the data:

  • Export files. mongoexport writes JSON files to a cloud bucket, and COPY INTO or Auto Loader loads them. Freshness is hours to daily, and an updated or deleted document only arrives through a full reload or a merge you write yourself.
  • Debezium and Kafka, read with Structured Streaming. Changes arrive within seconds. In return you run Kafka, Kafka Connect, Debezium and a Databricks job that has to keep running, and the merge logic is yours.

This tutorial builds a third option. OLake Go, an open source (Apache 2.0) replication tool, does change data capture (CDC) from MongoDB change streams and commits each collection as an Apache Iceberg table into Unity Catalog. Databricks shows the result as managed Iceberg tables with one column per top-level field, and DuckDB and Spark can query the same tables through the same catalog.

I ran it against MongoDB 7 as a single-node replica set and a Databricks workspace on Google Cloud, using OLake Go v1.1.0. The first sync copied 48 documents in 23 seconds, and the second replayed 12 changes in 16 seconds. Those figures, and every other one below, come from that run.

What you need

  • MongoDB 4.0 or later running as a replica set, self-hosted or on Atlas. Mine was a database called shop with two collections, customers and orders.
  • A Databricks workspace with Unity Catalog, a catalog with managed storage, and an admin who can turn on external data access and grant EXTERNAL USE SCHEMA.
  • Docker, for OLake UI, and about 30 minutes.

Step 1: Prepare MongoDB for CDC

OLake Go reads changes through change streams, and MongoDB builds those on the replica set oplog. So the server has to run in replica set mode, and one node is enough.

On a self-hosted server, start mongod with --replSet rs0 and run this in mongosh on the primary:

// 1. A single-node replica set, if you do not have one (self-hosted only) rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] }) // 2. A user for OLake Go use admin db.createUser({ user: "olake", pwd: "<password>", roles: [{ role: "read", db: "shop" }, { role: "clusterMonitor", db: "admin" }] }) // 3. Check the oplog size; OLake Go warns when the window is below 7 days use local db.oplog.rs.stats().maxSize // bytes; raise with replSetResizeOplog if it is small 

The first copy of the collections needs read on the database, and the change stream needs the oplog access that clusterMonitor (or readAnyDatabase) grants. Confirm both pieces before you go on:

rs.status().members.map(m => m.stateStr) // one PRIMARY db.getSiblingDB("admin").getUsers().users.map(u => u.user) 

Point OLake Go at collections you already have, or create two like mine. My shop database started with 8 customers and 40 orders, with integer _id values and money stored as NumberDecimal:

use shop const now = new Date(); db.customers.insertMany([ { _id: 1, name: 'Aarav Mehta', email: 'aarav@example.com', city: 'Mumbai', tier: 'gold', created_at: now, updated_at: now }, // ... more customers with integer _id values; step 7 updates 'Liam Carter' and 'Zara Ali' ]); db.orders.insertMany([ { _id: 1, customer_id: 2, product: 'Headphones', amount: NumberDecimal('99.19'), status: 'shipped', created_at: now, updated_at: now }, // ... more orders with the same fields ]); 

On MongoDB Atlas the replica set and the oplog already exist. Create the user under Database Access with the read role on your database plus clusterMonitor, and allow OLake Go's IP under Network Access.

⚠️ The oplog is your replay window, and Atlas and most self-hosted installs keep 24 hours or less of it by default. If the saved change stream token has rolled out of the oplog when the next sync starts, OLake Go needs a fresh snapshot.

Step 2: Set up Unity Catalog in Databricks

Unity Catalog is the catalog built into every Databricks workspace, and OLake Go writes to it through its Iceberg REST (representational state transfer) endpoint. Databricks allows writes through that endpoint for managed Iceberg tables only (Databricks: access tables from Iceberg clients). Four things have to be in place, once:

Setting Where
External data access on the metastore Catalog, the gear icon, Metastore, then External data access
A catalog with managed storage Catalog, Create catalog, with a storage location backed by an external location
EXTERNAL USE SCHEMA for the writing principal On the catalog, in the SQL below
A personal access token Settings, Developer, Access tokens

Then run this in the Databricks SQL editor with your own principal:

-- The catalog OLake Go writes into. Its managed storage holds the Iceberg files. CREATE CATALOG IF NOT EXISTS lakehouse MANAGED LOCATION '<external location URL>/lakehouse'; -- What the writing principal needs: browse and create schemas, then read and write tables in them GRANT USE CATALOG, CREATE SCHEMA ON CATALOG lakehouse TO `olake@example.com`; GRANT USE SCHEMA, CREATE TABLE, SELECT, MODIFY ON CATALOG lakehouse TO `olake@example.com`; GRANT EXTERNAL USE SCHEMA ON CATALOG lakehouse TO `olake@example.com`; 

Because the grants sit on the catalog, every schema OLake Go creates later inherits them. My catalog was named datazip, and my token expired after seven days. The SQL here says lakehouse, so swap in your own catalog name.

⚠️ EXTERNAL USE SCHEMA is not part of ALL PRIVILEGES, and only the catalog owner can grant it (Databricks: Unity Catalog privileges).

Step 3: Start OLake UI and connect MongoDB

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d 

Open http://localhost:8000 and sign in with admin / password.

Open Sources, click Create Source and pick MongoDB as the connector. Under Change data capture (CDC), leave Enable CDC on.

Under Connection details, add the host as host:port, then the Database Name, Auth DB, username, password and Replica Set name. My MongoDB ran in another container, so the values were host.docker.internal:27017, shop, admin, olake and rs0, with Read Preference left on primary. For Atlas, enter one SRV host and turn Use SRV on.

OLake UI source form for MongoDB on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal:27017, database shop, auth DB admin, user olake, replica set rs0 and read preference primary

Click Set up source. It tests the connection, confirms that a change stream can be opened and reports the oplog window before anything is saved. A single-node test container keeps a short oplog, so expect a warning here that does not block the sync.

Step 4: Add the Unity Catalog destination

Open Destinations and click Create Destination. Pick Apache Iceberg as the connector, Unity as the catalog type and Personal Access Token as the authentication type.

Field Value
REST Catalog URL https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
Unity Catalog Name lakehouse, the catalog from step 2
Token the personal access token from step 2

OLake UI destination form for an Apache Iceberg destination on the Unity catalog type, showing the personal access token fields

Click Create. You never enter a bucket path, because Unity Catalog decides where the files go.

Step 5: Create the job and run the first sync

  1. Open Jobs and click Create Job. I named mine mongodb_to_databricks. The schedule sets how often the change stream is drained, not how often a full copy runs.
  2. Select the MongoDB source and the Unity destination.
  3. Under Advanced Settings, set Target Query Engines to Databricks, plus DuckDB and Apache Spark if they will read the tables too.
  4. Click Next, and on the Streams page tick customers and orders. Set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert.
  5. Click Create Job, then open the job's menu and choose Sync Now.

Item 3 is the one that matters for Databricks: under Unity Catalog it reads Iceberg v3 deletion vectors and rejects commits that carry equality deletes. With Databricks in the list, the Upsert Type on both streams was already Deletion Vector, so I had no format to choose.

OLake UI Streams Selection page with the shop database, customers and orders selected, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Deletion Vector on Iceberg format version V3

The first sync copied 8 customers and 40 orders (48 documents, 6.40 KB) in 23 seconds, container start included. It opens the change stream and keeps the resume token first, then reads each collection in parallel chunks split on _id. Whatever is written during the copy is replayed from that token afterwards, so there is no separate backfill job.

OLake UI Job Logs and History for mongodb_to_databricks showing two completed syncs, the first in 23 seconds and the second in 16 seconds

OLake Go names the schema after the job and the source database, so mine was mongodb_to_databricks_shop. The Iceberg DB field at the top of the Streams page changes that before the first sync.

Step 6: Query the collections in Databricks

There is nothing to register and nothing to refresh. The SQL editor or a notebook reads the tables by their three-part name:

SHOW TABLES IN lakehouse.mongodb_to_databricks_shop; SELECT _id, name, city, tier FROM lakehouse.mongodb_to_databricks_shop.customers ORDER BY _id; SELECT status, count(*) AS n, round(sum(CAST(amount AS DOUBLE)), 2) AS total FROM lakehouse.mongodb_to_databricks_shop.orders GROUP BY status ORDER BY status; 

Two MongoDB details show in these queries. The key column keeps its document name, _id. And amount is a STRING because the source stores it as Decimal128, which is why the roll-up casts before it sums.

A 2X-Small serverless SQL warehouse was enough for my checks. Next to your own fields, OLake Go adds _olake_id (a hash of the primary key), _op_type, _olake_timestamp, _cdc_timestamp and, for MongoDB, _cdc_resume_token.

Step 7: Change documents in MongoDB and sync again

A snapshot alone proves little, so I made 12 changes after the first sync: 7 inserts and 5 updates.

const now = new Date(); db.customers.insertMany([ { _id: 9, name: 'Maya Patel', email: 'maya@example.com', city: 'Pune', tier: 'silver', created_at: now, updated_at: now }, { _id: 10, name: 'Lucas Silva', email: 'lucas@example.com', city: 'São Paulo', tier: 'bronze', created_at: now, updated_at: now } ]); db.orders.insertMany([ { _id: 41, customer_id: 9, product: 'Laptop', amount: NumberDecimal('1299.00'), status: 'placed', created_at: now, updated_at: now }, { _id: 42, customer_id: 9, product: 'Mouse', amount: NumberDecimal('25.50'), status: 'placed', created_at: now, updated_at: now }, { _id: 43, customer_id: 10, product: 'Monitor', amount: NumberDecimal('349.99'), status: 'placed', created_at: now, updated_at: now }, { _id: 44, customer_id: 3, product: 'Keyboard', amount: NumberDecimal('89.00'), status: 'placed', created_at: now, updated_at: now }, { _id: 45, customer_id: 7, product: 'Webcam', amount: NumberDecimal('79.90'), status: 'placed', created_at: now, updated_at: now } ]); db.customers.updateOne({ name: 'Liam Carter' }, { $set: { tier: 'gold', updated_at: now } }); db.customers.updateOne({ name: 'Zara Ali' }, { $set: { city: 'Abu Dhabi', updated_at: now } }); db.orders.updateMany({ _id: { $in: [1, 4, 7] } }, { $set: { status: 'delivered', updated_at: now } }); 

Choose Sync Now again. The second sync read 12 change events (1.60 KB) from the stream and finished in 16 seconds end to end.

For customers, that produced one new data file with 4 rows: the 2 inserts plus the new versions of the 2 updated documents. One deletion vector, a small bitmap per data file that marks replaced positions, retires the 2 old versions. The commit report in the sync log shows the same thing: addedDataFiles=1, addedRecords=4, addedDVs=1 and addedPositionalDeletes=2.

OLake UI sync log filtered to the Iceberg commit reports for customers and orders after the change sync, showing addedDataFiles 1, addedDVs 1, addedRecords 4 and 8, and addedPositionalDeletes 2 and 3

The orders table has the same shape, with 8 rows and 3 deleted positions, and the equality delete count stays at 0.

Run the SELECT from step 6 again. Databricks shows the new state on the next query: Liam Carter is gold, Zara Ali has moved to Abu Dhabi, and the two new customers are listed.

Databricks SQL editor running SELECT on datazip.mongodb_to_databricks_shop.customers on the 2X-Small SQL warehouse, returning 10 rows with id, name, city and tier

Add _op_type to the query to see how each row arrived: r for snapshot rows, c for the inserts and u for the updates. The orders roll-up matched MongoDB to the cent once amount was cast. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26 and 11 shipped for 6397.25.

Schema changes. MongoDB has no schema, so a new field can appear in any document. OLake Go adds a column for a new top-level field on the next sync, and documents without the field hold NULL. A field whose type changes between documents is widened to STRING.

Read the same tables from DuckDB and Spark

No export or second copy is involved here. DuckDB attaches the Unity Catalog catalog as an Iceberg REST catalog with the same token, and Unity Catalog hands it short-lived storage credentials per table.

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; CREATE SECRET unity (TYPE iceberg, TOKEN '<personal access token>'); ATTACH 'lakehouse' AS uc ( TYPE iceberg, ENDPOINT 'https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest', SECRET unity, ACCESS_DELEGATION_MODE 'vended_credentials' ); SELECT _id, name, city, tier FROM uc.mongodb_to_databricks_shop.customers ORDER BY _id; 

DuckDB 1.5.5 returned the same 10 customers and the same orders roll-up as Databricks. DuckDB reads Iceberg v3 deletion vectors from duckdb-iceberg 1.5.3, and the principal behind the token needs EXTERNAL USE SCHEMA, the same as OLake Go.

Open source Spark reaches the catalog as a plain Iceberg REST catalog, with the catalog name as the warehouse:

spark-sql \ --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0,org.apache.iceberg:iceberg-gcp-bundle:1.10.0 \ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ --conf spark.sql.catalog.uc=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.uc.type=rest \ --conf spark.sql.catalog.uc.uri=https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest \ --conf spark.sql.catalog.uc.warehouse=lakehouse \ --conf spark.sql.catalog.uc.token=<personal access token> \ --conf spark.sql.catalog.uc.header.X-Iceberg-Access-Delegation=vended-credentials 

Spark 4.0.1 with Iceberg 1.10.0 returned the same rows as the other two engines. Swap iceberg-gcp-bundle for iceberg-aws-bundle or iceberg-azure-bundle to match the cloud your workspace runs on.

Data types to check before you query

MongoDB Databricks shows Note
int, timestamp INT the integer _id and customer_id fields arrived as INT
long BIGINT
double DOUBLE
boolean BOOLEAN
date TIMESTAMP written in Coordinated Universal Time (UTC)
decimal128 STRING cast before you sum
objectId, binData, regex, code STRING an ObjectId key arrives as its 24-character hex string
object, array STRING JSON text; parse it at query time

The Databricks column comes from DESCRIBE on the tables of my run.

Tips and limits

  • Cast Decimal128 before arithmetic. CAST(amount AS DOUBLE) worked for my roll-up, and DECIMAL(12,2) gives exact arithmetic.
  • Flattening stops at one level. Nested objects and arrays arrive as JSON text in a STRING column. If every query would start by parsing that text, a landing table built for semi-structured data may serve you better.
  • Rotate the token. An expired personal access token fails the next sync at the first catalog call. Use a service principal with OAuth machine-to-machine credentials for production jobs.
  • These are Iceberg tables, not Delta tables. If a pipeline needs native Delta Lake features on these rows, load them into a Delta table downstream.
  • Check predictive optimization. Unity Catalog runs snapshot expiration and file compaction on managed Iceberg tables when it is on (Databricks: Iceberg tables).
  • Freshness is a schedule, down to one minute, not a continuous stream.

MongoDB to Databricks, three ways, side by side

Method Freshness Updates and deletes Where the data lands What you run
mongoexport to a bucket, then COPY INTO or Auto Loader hours to daily no; full reloads or your own merge tables written by a Databricks job an export script, a bucket and a Databricks job
Debezium, Kafka and Structured Streaming seconds yes, with your own merge logic Kafka topics, then tables written by a Databricks streaming job Kafka, Kafka Connect, Debezium and a running Databricks job
OLake Go into Unity Catalog (this post) CDC on a schedule down to one minute yes, as deletion vectors Iceberg tables that Databricks, DuckDB and Spark all read one Docker Compose stack, open source; your storage and your own compute

Debezium fits when you need seconds of latency and already run Kafka. Writing open tables once pays off as soon as a second engine needs the same rows.

FAQ

Does OLake Go need a Databricks warehouse or cluster to write?

No. It runs wherever you run Docker and writes the files itself, so Databricks compute is spent only when you query. Storage is the cloud bucket behind the catalog's managed location, billed at your cloud's storage rates.

What happens when a document is deleted in MongoDB?

The next sync reads the delete from the change stream, and _op_type uses d for a delete. Like an update, it retires the old row with a deletion vector, not with an equality delete. My run only made inserts and updates, so I have no delete numbers to report.

How long can the job stay paused?

No longer than the oplog window. Each sync resumes from the token the previous one saved, and a token that has rolled out of the oplog means a fresh snapshot. Raise the window with replSetResizeOplog on a self-hosted server, or with the minimum oplog window setting on Atlas.

Where to go from here

  • Point the source at your own collections. On Atlas, the source form takes one SRV host.
  • The command line setup, the troubleshooting list and the full type mapping are in the MongoDB to Databricks guide on olake.io.

If your run prints something this post does not cover, paste the message in the comments and I will add it.