Airflow 3 platform

problem

Problem

The data warehouse ran on Airflow 1.10 with about 190 DAGs and more than 500 connections. DAGs were deployed by a single CI job that ran git pull over SSH on five VMs. Nothing was checked before merge, so an import error showed up only on the servers. Airflow 1.10 reached end of life in 2021.

solution

Solution

A new cluster on Airflow 3.1.5: two control-plane hosts with the scheduler, DAG processor, API server, PostgreSQL and RabbitMQ, and three Celery workers. Components run in Podman containers as systemd units, the cluster is rolled out with Ansible, and images are built with kaniko in GitLab CI.

• DAG bundles: two Git repositories are attached as GitDagBundle, Airflow tracks the main branch and refreshes every 30 seconds. The git provider version is pinned: an early release removed .git from bundle versions, and every task cloned the repository again
• CI for DAG repositories: ruff with a GitLab Code Quality report, a format check and an import check that loads the DagBag and serializes every DAG in the production Airflow image. Import errors and duplicate dag_id values block the merge. pre-commit runs ruff on commit and rejects direct commits to the main branch
• dbt through astronomer-cosmos: dbt run, test and source commands run in separate containers on the worker hosts, with Kerberos authentication for Hive and Impala. One metric is one task group: dbt models, a transfer from Hive to PostgreSQL and aggregates. Gate groups check source freshness and the presence of data for the date before the calculation starts
• Safe releases: images from merge requests get their own tags, and only the main branch publishes the tag that the cluster updates from
• A shared Python package for DAG code with its own CI: lint, tests, a wheel build and versioned releases

stack

Stack

Airflow 3.1.5 with CeleryExecutor, RabbitMQ, PostgreSQL 17 with WAL-G, Podman, Ansible, kaniko, GitLab CI, ruff, pre-commit, dbt-core 1.7 with Hive, Impala and PostgreSQL adapters, astronomer-cosmos 1.14.

role

My role

Lead engineer: designed the cluster, the delivery of DAGs and the CI checks, set up dbt runs through Cosmos, and now lead the move of DAGs from Airflow 1.10.

result

Outcome

Airflow picks up merged DAG changes from Git every 30 seconds, without a separate deploy. Import and style errors are caught in the merge request instead of on the servers. The dbt project has about 200 models and 95 tests. The move of DAGs from Airflow 1.10 continues.