Workflow Orchestration

Data engineering

Pipelines, sketches, approximate aggregations, object storage, and stream processing with event time, windows, and exactly-once sinks.

• Workflow Orchestration

  6 studies
  1. 1.Workflow Orchestration - Scheduled DAGs vs Durable Execution (Airflow, Temporal & Friends)Concept hub: two families of orchestrators, scheduled batch DAGs (Airflow, Dagster, Prefect, Argo) vs durable execution (Temporal, Cadence, Step Functions, Durable Functions, Restate, Inngest); the shared kernel (durable state, scheduler, queue, workers, retries, heartbeats/leases, at-least-once units so idempotency matters); comparison tables; runnable task-level vs step-journal crash demo and lease/heartbeat kernel; decision chart.
  2. 2.DAG Fundamentals & Airflow Architecture - Topological Scheduling, the Scheduler Loop, Executors, Task States & PoolsWhy DAGs and what topological order buys (waves, critical path, cycle detection); Airflow 3.x architecture (Dag processor, Dag bundles, scheduler, API server and Task Execution API, triggerer, metadata DB); task-instance states and trigger rules; pools and concurrency limits; executors compared (Local, Celery, Kubernetes, Edge, multiple executors); HA schedulers via row locks; runnable mini scheduler and wave/mapping simulation; executor decision chart.
  3. 3.Writing Correct Pipelines - Data Intervals, Catchup & Backfill, Idempotent Tasks, Deferrable Sensors & AssetsLogical date and data intervals, catchup and backfill in Airflow 3 (catchup off by default, logical_date None for asset/API runs), idempotent tasks with partition overwrite (runnable sqlite demo: append vs delete+insert), DST and time zones (runnable America/Chicago demo), XCom limits, sensors vs deferrable operators vs assets and AssetWatcher, dynamic task mapping, Deadline Alerts replacing SLAs, failure modes; Dagster/Prefect comparisons; waiting decision chart.
  4. 4.Durable Execution & the Temporal Model - Workflows vs Activities, Event History, Replay & DeterminismWorkflows vs activities, event history and replay, determinism rules (SDK time/random, no I/O in workflow code) with a runnable replay engine showing a wall-clock non-determinism bug and the fix; timers, signals, queries and updates (runnable race demo); task queues and workers; retry policy defaults and the four activity timeouts; what exactly-once means (and does not) with idempotency keys; history and payload limits; workflow-vs-activity decision chart.
  5. 5.Temporal Patterns in Production - Sagas, Child Workflows, Continue-As-New, Versioning & Worker ScalingSagas with compensations (register compensation first, non-retryable errors, runnable trip saga), child workflows and parent close policy, continue-as-new and history limits (runnable), versioning with patching vs Worker Versioning (runnable patched-marker and unsafe-change demo), human-in-the-loop with signals/updates and timers, idempotent activities, worker scaling and sticky queues, persistence and visibility stores and history shards; shipping-a-change decision chart.
  6. 6.Choosing & Operating Workflow Orchestrators - Airflow vs Dagster vs Prefect vs Temporal vs Step Functions vs Argo, Testing & MigrationAirflow vs Dagster vs Prefect vs Argo vs Temporal vs Step Functions vs cron+queue on run model, graph, state model, latency, duration, dynamic-ness, backfills, human waits, ops burden and cost drivers; what goes wrong if you pick otherwise; observability signals; testing (runnable DAG integrity test and workflow replay-test gate); migrations (cron to Airflow, Airflow 2 to 3, Airflow to Dagster, to durable engines); interview Q&A; decision chart.