Course · Streaming & orchestration
Airflow
Airflow schedules and orchestrates pipelines as DAGs. Learn scheduling, task dependencies, retries and idempotent task design.
- Lessons
- 8
- Interview questions
- 2
- Projects & case studies
- 7
- Reading time
- ~3 h
Your progress
Saved in this browser onlyCourse structure
- 2 lessonsBeginnerCore concepts you will use every day.
- 5 lessonsIntermediatePatterns used in production pipelines.
- 1 lessonAdvancedPerformance, internals and edge cases.
Practise
- InterviewAirflow interview questionsThe full list with difficulty, type and a box to tick off each one.
- Cheat sheetAirflow Cheat SheetA quick Airflow reference: TaskFlow DAGs, schedules and data intervals, retries, templating, sensors, trigger rules and the Airflow 3 CLI commands you use most.
- InterviewAll interview questionsEvery question across all topics in one filterable list.
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Beginner
Core concepts you will use every day.
- Airflow DAG Fundamentals, TaskFlow and Dynamic DAGsBuild Airflow 3 DAGs from first principles: tasks and dependencies, the TaskFlow API, params and Jinja templating, dynamic DAGs and dynamic task mapping.
- Airflow Operators, Hooks, Providers, Branching and Trigger RulesUse BashOperator and PythonOperator well, write custom operators and hooks, pick provider packages, run pods, and control flow with branching and trigger rules.
Intermediate
Patterns used in production pipelines.
- Airflow Sensors and Deferrable OperatorsWait for files, other DAGs and external jobs without wasting workers: sensor modes, timeouts, ExternalTaskSensor, FileSensor, triggers and what replaced Smart Sensors.
- Airflow Scheduling: Intervals, Logical Dates, Catchup, Timetables and AssetsHow the Airflow 3 scheduler creates runs: cron and presets, logical dates and data intervals, catchup pitfalls, timetables, asset-aware scheduling and deadline alerts.
- Airflow XCom, Variables, Connections and SecretsMove small data between tasks with XCom, keep large data out with custom backends, manage config and credentials with Variables, Connections and secrets backends.
- Operating Airflow: Backfills, Retries, Alerting, Logging and the REST APIRun Airflow day to day: backfill history, clear tasks safely, set retries and alerts, configure logs and StatsD metrics, and use the REST API v2.
- Airflow Best Practices: Idempotent Tasks, Testing and CI/CDWrite Airflow DAGs that are safe to rerun and cheap to parse, then test them with DagBag integrity tests, unit tests and dag.test(), and ship them through CI/CD.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
System design case studies
- AdvancedDesign a Backfill and Late-Data Handling SystemDesign the platform capabilities that let a data team handle late-arriving data automatically in daily and hourly pipelines, and run large backfills (after bug fixes, new columns or new pipelines) across months of history and dozens of dependent tables, safely, cheaply and without disturbing production loads or consumers.
- AdvancedDesign a Data Catalog and Lineage SystemAnalysts at a large company spend days finding the right table, nobody knows who owns half the datasets, and engineers cannot tell what will break if they change a column. Design a data catalog and lineage system that automatically harvests metadata from warehouses, lakehouses, pipelines and BI tools, makes data discoverable and trustworthy, shows table- and column-level lineage, and supports governance workflows such as ownership, classification and access requests.
- AdvancedDesign a Data Observability SystemA data platform runs 3,000 tables across a warehouse and a lakehouse, fed by Airflow, dbt, Spark and streaming jobs. Problems are usually found by business users hours later. Design a data observability system that monitors pipelines and data automatically, detects freshness, volume, schema and distribution anomalies, finds the likely root cause through lineage, and drives incidents to resolution against defined SLAs.
- AdvancedDesign a Data SLA and Freshness Monitoring SystemDesign a system that tells a company, for each of its important datasets, whether the data is fresh, complete and correct enough to use right now; alerts the right owner before consumers notice a problem; shows which downstream dashboards and models are affected; and reports SLA attainment over time.
- AdvancedDesign an Idempotent Reprocessing SystemA bug in the revenue logic went unnoticed for three weeks; a source re-sent a month of corrected data; a new metric needs two years of history. Today each of these takes a week of manual work and risks duplicates. Design a reprocessing system that lets engineers re-run any pipeline over any range of history, safely and repeatably, while daily runs continue, and that propagates corrections downstream.
- IntermediateDesign a Batch Ingestion FrameworkA data team writes a new pipeline by hand for every source, and now runs 150 slightly different jobs pulling from databases, SFTP drops, object storage and REST APIs. Design a reusable, metadata-driven batch ingestion framework that onboards a new source through configuration, lands data reliably and idempotently in the lakehouse, and is easy to operate, backfill and monitor.
Resources
Cheat sheets
Related courses
- PythonPython glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.
- KafkaKafka is a distributed log used for streaming data. Learn topics, partitions, consumer groups and delivery semantics before building streaming pipelines.
- Apache SparkHow Spark works inside: driver and executors, RDDs, jobs and stages, shuffles and skew, caching, Catalyst, AQE, memory tuning, Kubernetes and the History Server.