DWH on-call monitoring
Dashboards and alerts for the engineer on duty: failed and stuck DAG runs, DAGs that run longer than usual, and stale metrics in product dashboards.
attributes
- role
- lead engineer
- source
- work project, code is private
Problem
A data warehouse on Airflow 1.10 runs about 190 DAGs that feed product dashboards for 25 projects. The engineer on duty needs to see failed and stuck runs, DAGs that run longer than usual and metrics that did not arrive on time, and to learn about failures from the team chat.
Solution
Monitoring consists of Grafana dashboards, failure alerts from Airflow to a corporate messenger and scheduled checks that run as DAGs.
- Dashboards: an on-call overview with 17 panels refreshed every 5 minutes and a dashboard of DAGs that run longer than usual with 41 panels refreshed every minute. Data comes from the Airflow metadata database, a product metrics table in PostgreSQL and statsd metrics in VictoriaMetrics
- DAG duration: statsd_exporter quantiles were NaN in 1329 of 1362 samples, so duration is calculated from the _sum and _count series
- Failure alerts: an on_failure_callback in 118 DAGs posts a message with a link to the run, and the thread under it mentions the engineer on duty and contains the traceback
- Hourly check: failed runs, tasks left in the running state from previous dates, and a warning when fewer than three workers are active
- Freshness check: every 15 minutes during the day, the metrics of 25 projects are compared with thresholds that account for the expected delay of each metric and project. The report is a message with a thread per project and a progress bar
- The on-call schedule is kept in Airflow, and alerts take the person to mention from it
Stack
Airflow 1.10 (callbacks, PythonOperator, SQLAlchemy models of the metadata database), PostgreSQL, statsd_exporter, VictoriaMetrics, Grafana, the messenger Bot API, Python with requests, pandas and Jinja2.
My role
Lead engineer: designed the on-call monitoring, from the dashboards to the alert format and the scheduled checks.
Outcome
Failures of 118 DAGs reach the team chat with the engineer on duty mentioned and the traceback in the thread. Stale metrics are reported per project during the day, stuck tasks and a shortage of workers every hour.