Data Orchestration: How It Works and How to Retry Safely
Data orchestration coordinates when data tasks run, which dependencies they require, and how retries and timeouts are handled, as represented in Apache Airflow’s workflow model. The model controls execution while leaving the processing logic inside each task. Airflow task boundaries

Data orchestration, pipelines, and ETL
ETL means extract, transform, and load: retrieving source data, preparing it, and loading it into a destination; ELT changes the sequence by loading data before transforming it, as explained in AWS’s ETL documentation. These terms describe the processing sequence. The orchestration question is how that work is scheduled, coordinated, and recovered.
A data pipeline groups activities into a workflow; Azure Data Factory defines a pipeline as a logical collection of activities managed together, with datasets identifying their inputs and outputs (Microsoft’s pipeline documentation).
Processing and orchestration can share the same platform: Azure Data Factory includes data movement, transformation, and control activities within pipelines. Microsoft’s activity model The practical distinction is between the operations performed on data and the rules governing their execution.
How data orchestration works
A workflow definition connects units of work through dependencies; a directed acyclic graph, or DAG, represents ordered connections without cycles inside the graph. Airflow’s DAG documentation Building an executable workflow then involves three responsibilities:
- Define dependencies. Tasks have upstream and downstream relationships; Airflow’s default behavior runs a task after all its upstream tasks succeed, with configurable control flow for other conditions. Airflow tasks Specify actual prerequisites rather than relying on elapsed time between separate scheduled jobs.
- Choose the starting condition. Work can begin through a direct invocation, an ad hoc deployment run, a schedule, or an event automation; Prefect documents these ways of creating a flow run. Prefect flows Select the trigger according to the required input availability and processing cadence.
- Track execution. A task definition is instantiated for a particular run, and Airflow records states including queued, running, success, failed, and up for retry. Task instances Use this history to identify the affected execution and the work requiring attention.
Dependencies and datasets describe different relationships: Azure’s pipeline model distinguishes activity control from the datasets used as inputs and outputs. Pipeline and dataset relationships Document both the required execution order and the data each step consumes or produces.
When orchestration is useful
Recurring workflows, historical processing, and recovery are concrete orchestration uses: Airflow supports scheduled batch workflows, manual triggers, log inspection, task monitoring, backfills, and rerunning failed tasks (Airflow overview).
A practical reason to introduce an orchestrator is the need to manage those controls explicitly across dependent work. Evaluate the dependency graph, required execution history, and recovery scope before adding a platform. For bounded workloads, Airflow’s documentation identifies workflows with a clear start and end as its intended batch orchestration model. Airflow workload guidance Match the execution model to the workload as well as to its trigger.
Run scope: retries, reruns, and backfills
The time work executes and the period of data it processes are separate concepts: Airflow assigns a data interval to a run, and its manual-run interval depends on the timetable and trigger path. Airflow data intervals Record the intended period explicitly when launching historical work.
- Retry: attempt a failed task again under its configured retry behavior; Airflow marks a failed task with attempts remaining as
up_for_retry(Task retry state). - Rerun: explicitly repeat selected work; Airflow supports clearing task instances and choosing scopes such as failed, upstream, or downstream tasks (Task reruns).
- Backfill: execute work for a specified historical period, using a selected date range and reprocessing behavior (Airflow backfills).
Choose the required scope before restarting work. A task retry, a downstream refresh, and historical reprocessing require different selections even when they use the same workflow definition.
Make retries safe with idempotent operations
An idempotent operation can be repeated without additional side effects; this is the property AWS describes as the basis for safe API retries (Making retries safe with idempotent APIs).
A missing response can leave completion uncertain, so repeating a request requires an idempotency contract or reconciliation of what already happened. AWS on retries and side effects Treat retry timing and repeatable operation behavior as separate implementation decisions.
For data tasks, Apache Airflow recommends three practices:
- Use explicit partitions. Read and write a specific partition; reading whichever data is latest can change the input between attempts (Partition guidance).
- Use repeatable writes. Airflow recommends UPSERT in place of INSERT during reruns because repeated inserts can duplicate database rows. Write guidance Choose the destination’s operation according to the records the task is intended to update.
- Produce complete results. Airflow advises treating tasks like database transactions, avoiding incomplete outputs, and avoiding the current wall-clock time in critical computation. Task design guidance Define the input selection and completion condition before enabling automatic retries.
For external calls, AWS describes using a caller-provided request identifier to recognize repeated requests, with atomic handling of the identifier and the associated mutation. Idempotent API design Reuse an identifier for the same intended operation only under the receiving service’s documented contract; confirm that repeated calls are handled consistently before making them automatic.
Bound retries and define the failure path
AWS Step Functions lets retry rules match named errors and configure attempt limits, initial delay, backoff, maximum delay, and randomized jitter. Step Functions retry controls Define which errors qualify for another attempt and set a finite recovery policy rather than relying on repetition alone.
When a state has both Retry and Catch, Step Functions evaluates applicable retries first and applies the matching catch transition if retries fail to resolve the error. Retry and catch behavior Design that transition to retain failure details and route the unresolved work for repair. Keep the requirements for releasing downstream output explicit in the workflow.
Validate outputs and record lineage
Data quality requires specified checks: Dagster’s asset checks can test non-null values, schema requirements, and whether data needs refreshing. Dagster asset checks Define the properties required for an output to be usable, and make those properties part of the workflow’s acceptance criteria.
In Dagster, a failed asset check does not block downstream materialization by default; setting blocking=True changes that behavior. Blocking downstream materialization Configure the check’s execution behavior as well as its validation logic when downstream work must wait for acceptable data.
Lineage records a different dimension of the workflow: OpenLineage distinguishes a job, an individual run, and its input and output datasets, with metadata facets for details such as source code version and data partitions (OpenLineage object model).
For recovery, retain the run identifier, code version, input and output dataset identifiers, and relevant partition information available through that model. Run and dataset metadata Use those records to identify the data involved in an execution and investigate downstream impact; assess data quality through the checks defined for the output.
Data orchestration tools and their workflow models
Compare platforms by their documented execution model and the controls required by the workload:
| Tool | Documented model |
|---|---|
| Apache Airflow | Python-defined workflows organized into tasks and dependencies, with scheduling and execution controls represented by a Dag. Airflow Dags |
| Dagster | Asset definitions identify a table or file, its upstream assets, and the Python function that computes and stores its contents. Defining assets |
| Prefect | Decorated Python functions compose flows with parameters, tracked execution state, configurable retries, and timeouts. Prefect flows |
| AWS Step Functions | State-based workflows use explicit error matching, retry policies, and catch transitions to control failure handling. Step Functions errors |
| Azure Data Factory | Pipelines group activities for data movement, transformation, and control, with datasets identifying inputs and outputs. Pipelines and activities |
Evaluate the required integrations, where processing runs, how historical work is selected, and how output checks and lineage are recorded. Document the retry contract for each write or external action alongside the workflow definition.