Lakehouse pilot

problem

Problem

A parser reads raw events from Kafka and stores them as Parquet on HDFS in a CDH 6.3 cluster with Hive 2.1, Impala and Kerberos. The pilot tested whether this layer could move to a cloud lakehouse: Iceberg tables in S3-compatible object storage, Spark for ingestion and Trino for queries.

solution

Solution

The first stage ran in VK Cloud with a managed Iceberg metastore (JDBC catalog), Trino and S3. The second stage repeated the setup in Yandex Cloud with Managed Metastore (Hive Metastore), Managed Trino, Object Storage and Data Proc.

• Streaming: Spark Structured Streaming reads Kafka in micro-batches of up to 1 million messages and writes a raw Iceberg table with the payload as map<string,string>, partitioned by application, event and date
• Typed layer: the raw table is split into one table per application and event pair, with type inference and schema evolution. By the end of September 2025 this layer had 1486 tables for 25 applications
• History: about 14 TB of 2025 Parquet files were copied from HDFS to S3 in 20 parallel streams with success markers, then registered in Iceberg with add_files, without rewriting the data
• Table maintenance through Trino: expire_snapshots and remove_orphan_files
• Prototypes for small loads: writing PyArrow batches through pyiceberg and reading with pyarrow.dataset
• Data was copied between the clouds with rclone, and a daily export from HDFS kept the second stage up to date

stack

Stack

Apache Iceberg 1.7 and 1.8, Spark 3.5, Trino 468 and 476, Kafka, PyArrow 20, pyiceberg 0.9, Hive Metastore, S3-compatible object storage, rclone, GNU parallel.

role

My role

Lead engineer: designed the pilot architecture in both clouds, built the ingestion from Kafka and the migration of the history, and tracked the issues found during the pilot.

result

Outcome

The pilot ran on real data: the event stream from Kafka and the 2025 history, about 14 TB. It also showed the limits of the setup: Trino write limits (1 GB of memory per node, 100 writers), metadata growth on bulk inserts and outages of managed services in the first stage. The move was postponed, and the raw layer stays on CDH.