In the early days of big data, companies had to make a frustrating trade-off. You could store petabytes of data cheaply in a Data Lake (like Amazon S3), but you would sacrifice reliability, ACID transactions, and speed. Alternatively, you could get reliability and speed from a traditional Data Warehouse (like Snowflake or BigQuery), but you would face astronomical storage costs and severe vendor lock-in.
Enter the Data Lakehouse—an architectural shift that brings data-warehouse-like governance, speed, and reliability straight to your cheap, open cloud storage.
At the absolute heart of this revolution is Apache Iceberg, paired with open processing frameworks like Apache Flink, high-performance query engines like Trino, and a Universal Semantic Layer.
Here is the ultimate, end-to-end breakdown of how a modern, production-grade Open Lakehouse operates from scratch.

The 10,000-Foot Architecture Blueprint

Before zooming into the code, let’s look at the high-level flow. Data moves from left to right, transitioning from raw events into highly governed, business-ready metrics:

 [ DATA SOURCES ] ──> Operational DBs (CDC), Apps, SaaS, IoT Devices │ ▼ [ INGESTION ] ──> Stream (Flink / Kafka) OR Batch (Spark / Airflow) │ ▼ ==================== THE ICEBERG LAKEHOUSE CORE ==================== [ STORAGE ] ──> Raw Parquet Files (Amazon S3, Azure Blob, GCS, MinIO) ▲ │ (Tracks & Coordinates Snapshots) [ CATALOG ] ──> The Brain (REST Catalog, Apache Polaris, AWS Glue) ==================================================================== ▲ │ (Pushes Down Optimized SQL Queries) [ COMPUTE LAYER ]──> Trino (Fast BI) | Apache Spark (Heavy AI & ETL) ▲ │ (Exposes Certified Metrics & Joins) [ SEMANTIC LAYER ]──> Cube / dbt Semantic Layer (The Single Source of Truth) ▲ │ (Consumes Rest API, SQL, or MCP) [ CONSUMERS ] ──> Tableau, PowerBI, Python Notebooks, LLM Agents

Layer 1: Ingestion & The Streaming Muscle (Apache Flink)

Data starts its journey inside operational databases (like Postgres) or as real-time events (like web clicks or IoT sensors).
To ingest this data cleanly into our lakehouse, we use Apache Flink. Flink acts as our real-time streaming engine. If we tried to dump streaming data straight onto cloud storage file-by-file, we would create millions of tiny files—a massive performance killer known as the "small file problem."
Flink solves this elegantly by functioning as a Streaming Sink:

  1. Flink aggregates incoming streams in memory.
  2. At designated intervals (e.g., every 30 seconds during a Flink checkpoint), it flushes that bundled data down to storage.
  3. Because Flink integrates natively with Iceberg, it guarantees Exactly-Once semantics. If a pipeline crashes mid-stream, Flink uses its checkpoint history to resume writing exactly where it left off, avoiding data duplication or corruption.

Layer 2: The Core Storage & Table Format (Apache Iceberg)

Once Flink flushes data out of memory, it lands in the Iceberg Lakehouse Core. This core is broken into two distinct parts:

1. The Storage Floor (The Muscles)

Your physical data bits are stored as highly compressed, open-source file types—usually Apache Parquet or ORC—sitting in cheap cloud object storage like Amazon S3, Google Cloud Storage (GCS), or Azure Blob.

2. The Iceberg Catalog & Metadata (The Brain)

In an old-school data lake, storage was just a messy pile of folders. Apache Iceberg transforms this storage into a fully transactional SQL database. It achieves this by maintaining a strict, tree-like "card catalog" of metadata directly on top of your Parquet files.

[ Iceberg Catalog ] ──> Points to the current state of the table │ [ Metadata File ] ──> Stores table schema, partition specs, and snapshots │ [ Manifest List ] ──> Groups manifest files and tracks snapshots │ [ Manifest Files ] ──> Tracks individual data files and stats (min/max values) │ [ Data Files ] ──> The actual raw data (Parquet chunks)

Because Iceberg tracks data at the file level instead of the folder level, it unlocks enterprise-grade superpowers:
• ACID Transactions: Readers will only ever see fully committed data. You can write to a table while someone else is querying it without any interference.
• Time Travel: Every change creates a new metadata snapshot. You can query your data exactly as it looked last Tuesday for debugging or auditing.
• Hidden Partitioning: You no longer need to manually layout folders by date. Iceberg automatically handles and optimizes data layout under the hood.

Layer 3: The Multi-Engine Compute Layer (Trino & Spark)

Because Iceberg is a completely open table format, your data is decoupled from any single vendor. You own the files and the catalog. This means you can plug multiple computing engines into the exact same S3 files simultaneously, selecting the best tool for the specific job.
• For Fast Interactive SQL & BI: We connect Trino or Dremio. These are massively parallel, in-memory query engines. When an executive opens a dashboard, Trino hits the Iceberg catalog, instantly figures out which Parquet files to read, bypasses the rest, and returns sub-second results.
• For Heavy ETL & Machine Learning: We connect Apache Spark. When it's time to train an AI model or run massive nightly batch cleanups over petabytes of history, Spark scales horizontally to do the heavy lifting.

Layer 4: The Translation Layer (The Semantic Layer)

Even with super-fast tables managed by Iceberg, humans and AI systems still struggle with raw database structures. A data scientist might write an SQL query calculating "Revenue" using gross sales, while a financial analyst might build a dashboard calculating "Revenue" minus returns. This discrepancy creates chaos.
The Semantic Layer (powered by tools like Cube or the dbt Semantic Layer) sits right on top of your compute engines to act as a Universal Translation Layer.
Instead of writing complex SQL joins and metrics inside your BI tools or AI prompts, you define your business logic once in code (using YAML or JavaScript) within the semantic layer:

yaml # Example: Defining a metric once in a Semantic Layer metrics: - name: net_revenue description: "Total revenue adjusted for returns and discounts" type: measure expr: gross_amount - discount_applied - return_amount #Use code with caution.

When a user in Tableau requests "Net Revenue", or an AI Agent queries your data via the Model Context Protocol (MCP), the Semantic Layer intercepts the request, looks up the certified calculation, compiles it into optimized SQL, pushes it down to Trino, and fetches the clean answer from your Iceberg tables.

Why This Architecture Wins

By constructing your data platform with this specific end-to-end design, your organization achieves three major competitive advantages:

  1. Zero Vendor Lock-In: If a faster query engine or a cheaper storage provider hits the market tomorrow, you simply swap out that layer. You never have to undergo a painful, multi-million dollar data migration.
  2. Massive Cost Optimization: You keep petabytes of historical data on dirt-cheap cloud object storage, bypassing the premium storage markups charged by proprietary data warehouses.
  3. A Unified Source of Truth: Your streaming data, batch data, BI tools, and LLM applications all point to a single, identical set of certified Iceberg tables. The Open Lakehouse ensures your data remains fast, affordable, governed, and completely under your control.