Make Data Orchestration Safe to Retry
The daily orders report is due, but yesterday’s figures have not appeared. The extraction finished; a transformation is still running; the report refresh has already started against the previous output. Someone restarts the transformation, and now the question changes: will it replace yesterday’s figures or append another copy? This is an illustrative situation, but it captures the job that sends many readers looking for data orchestration. They need dependent data work to run in the right order and to recover without making the result less trustworthy.
Data orchestration coordinates that work across tasks, runs, and datasets. A schedule may start the process, but the useful design says which task must finish before another begins, what counts as completion, and what should happen when an execution fails. Apache Airflow’s DAG documentation describes a directed acyclic graph as tasks connected by dependencies. The graph makes order explicit. It does not, by itself, establish that an output contains the right records.
That distinction should shape the whole design. An orchestrator can tell you that a task finished under its platform rules. It cannot infer from a green task state that a report uses the intended period, that a repeated write did not duplicate records, or that a failed input left no partial output. Those are decisions the workflow author must make at the task and dataset boundaries. The strongest starting point is therefore a small, understandable path from a named input period to a named output, with a clear answer for what a rerun will do.
A schedule starts the work; dependencies control it
Imagine the illustrative orders report has four steps: collect the orders for a day, prepare a cleaned daily dataset, calculate the report’s figures, and publish the report. Running all four at the same clock time would leave the later steps exposed to unfinished inputs. Running them in a single long script would preserve order, but it would give the operator little help in identifying the failed step or choosing what to rerun. A dependency graph expresses the actual requirement: the daily dataset follows collection, the calculation follows the dataset, and publication follows the calculation.
The point of the graph is not to draw an impressive pipeline. It is to encode the conditions that must hold before work proceeds. If two independent calculations consume the same completed daily dataset, they can sit on separate branches; neither needs to wait for the other merely because both appear in the same report. If publication requires both calculations, it should depend on both. This is a design judgment about the work, expressed through the task relationships described in Airflow’s DAG model.
A directed acyclic graph cannot circle back to an earlier task within the same run. That is useful for explaining a finite sequence of work, yet it also creates a common misunderstanding: a business process may repeat every day, while the graph for one daily execution remains acyclic. The repetition belongs to successive runs, not to an arrow looping inside one run. Keeping those ideas separate makes it easier to discuss a failed Tuesday run without confusing it with Wednesday’s run.
An orchestration task is a chosen unit of work, not a natural law. Airflow’s task documentation treats tasks as units inside a workflow, and the boundary is up to the author. If one task collects, cleans, calculates, and publishes, a failure near the end may force all of that work to be repeated. Splitting every minor operation into its own task has its own cost: the graph becomes harder to read, and a run gains more boundaries to manage. Separate tasks where a result can be named, reused, or recovered independently. Keep tightly coupled operations together when a separate task would add no useful decision point.
Give each run a specific input and output
The words “run the orders pipeline again” are incomplete. Which day of orders should it read? Which version of the prepared data should it replace? Which report should it update? Without those answers, the scheduler may correctly execute every task while the data moves to the wrong place. A dependable run needs a defined input boundary and a defined output boundary before retry settings or dashboards become useful.
In the illustrative report, a daily run can be assigned to a particular reporting day. The collection task reads that day’s orders; the preparation task writes that day’s cleaned dataset; the calculation consumes that same day’s prepared data. If a late correction requires rerunning Tuesday, Tuesday’s run should still address Tuesday’s inputs and outputs, even if the operator starts it on Friday. This is a design prescription, not a claim that a tool can identify the correct business period automatically.
Airflow’s best practices recommend reading and writing a specific partition rather than taking whatever data happens to be latest when a task starts. That recommendation matters because “latest” can change between attempts. A first attempt may read one set of rows; a later attempt may read an updated set, and both may claim to be the same daily run. Giving the task a period-specific input makes the intended repeat more predictable. It does not settle what to do when the source itself corrects a past period; that policy must be stated separately.
The output boundary matters just as much. A task that writes to a named daily partition gives the operator a place to inspect and, where the storage operation permits it, replace the result for that day. A task that writes into a general table with no run or period key can leave the operator unsure which records came from which attempt. The preferred design is to make the period visible in the input selection and output destination. Its cost is that someone must define how period corrections, overlapping runs, and downstream refreshes should be handled. Concealing those choices inside “process latest” does not remove the work; it postpones it until recovery is urgent.
Make a retry repeat the result, not the side effect
An orchestrator’s retry control answers when to try a failed step again. It cannot make the step safe to repeat. AWS Step Functions documents retry routing for selected errors; that capability is valuable for failures that might clear on another attempt. Yet a task can fail after writing data and before recording success. A second attempt may then repeat the write. The workflow is mechanically recoverable only if the underlying operation can tolerate that repetition.
For the illustrative daily report, an append that inserts Tuesday’s calculated rows on every attempt is a poor default. If the first attempt inserted the rows and later failed, a retry could insert them again. Airflow’s best practices recommend repeatable task behavior for reruns. An update keyed to the intended record, or a controlled replacement of the day’s partition, is easier to reason about than an unconditional append. The exact write method depends on the destination, but the required behavior is stable: after a successful retry, Tuesday’s output should represent one intended result, not the accumulated effects of several attempts.
The same principle applies when a task causes an external action rather than writing a dataset. Consider an illustrative publication step that sends a report notification. If the report was sent but the task failed while recording completion, blindly rerunning it may send another notification. The orchestration graph knows the step’s recorded state; it may not know whether the recipient saw the first message. Put a separate safeguard around such effects, or make the operator decide whether to retry them. This is why repeatable data writes and repeatable side effects need distinct treatment.
There is also a trade-off in how much work a retry repeats. A large task that collects and publishes in one attempt has fewer visible steps, but any failure may rerun work whose effects already happened. A sequence of tasks can isolate the failed portion, provided each boundary has a durable output that the next task can use. The useful size of a task is therefore partly a recovery choice. Split at the point where the result is meaningful and can be repeated safely; avoid a split that leaves an ambiguous, half-written handoff.
Decide what a failure may stop or redirect
Once the task boundaries are clear, decide what the workflow should do with a failure. In the illustrative orders report, a failed collection step should block calculations that require its output. Allowing the report to publish simply because an older prepared dataset is present would make the run appear complete while showing yesterday’s numbers. A failure path must preserve the meaning of the published result, not merely move the workflow to a terminal state.
Some failures justify another attempt: the task may be able to complete without changing the requested input period or duplicating its output. Others require a person or a repair step because the input is missing or the output was only partly written. Step Functions’ error handling documentation describes both retry rules and catch paths that send a failed state to another state. A catch path is a way to choose the next action. It does not repair the data that the failed step was meant to produce.
That distinction changes what an appropriate fallback looks like. A path that records the failed reporting day and stops publication preserves the question an operator must answer. A path that publishes the previous day’s report under today’s label does not. If the business genuinely permits an older report, the publication step should make that age explicit to its readers. Otherwise the safer call is to hold publication until the intended day’s result exists. This is a judgment about the report’s promise, expressed in workflow behavior.
Task states help locate where execution stopped. Airflow’s task model records state for a particular task execution under the platform’s rules. It can tell an operator whether an attempt is waiting, running, or complete. It does not explain why an output is acceptable for the business decision the report supports. A completed task is an execution fact; the acceptance rule for the resulting data must be designed alongside it.
Connect the run to the dataset it changed
A dependency graph answers which task was supposed to run after which other task. Recovery raises a different question: which dataset did the failed or repeated run actually read and write? If the daily calculation was rerun for Tuesday, an operator needs to know whether the report consumed Tuesday’s refreshed dataset or an earlier output. That question crosses the boundary between workflow history and data history.
OpenLineage’s object model separates a job definition, an individual run, and the datasets involved in that run. That separation is practical. “Calculate daily orders” names the job; “the attempt for Tuesday” identifies an occurrence; “Tuesday’s prepared orders” and the report dataset identify inputs and outputs. Recording these relationships can show what depends on a particular execution. It can also help determine which outputs may need to be refreshed after a corrected input is processed.
Lineage is only as useful as the systems that provide it and the detail they record. A missing connection can leave part of the path invisible. More importantly, a relationship between a run and a dataset is not a statement that the values inside the dataset are correct. Treat lineage as a map of possible impact and of the path data took. Judge correctness using the output’s own requirements: the intended period, the expected records, and the publication condition for this report. The map helps find where to look; it cannot make the decision for the operator.
This is where task names, period choices, and output names pay for themselves. A run called only “daily” and an output called only “latest” provide little context when the team needs to recover an earlier date. Clear identities let a reader connect the failed step to its intended data slice and see what downstream work should be considered for another run. The naming is modest work up front, but it reduces guesswork at exactly the moment a retry is most tempting.
Choose the orchestration design before choosing a tool
Airflow and Step Functions illustrate useful mechanisms, but neither name is a complete answer to “do we need data orchestration?” Airflow’s DAG and task models describe ordered work and individual task executions; Step Functions’ error handling describes configured retry and catch behavior. Airflow’s DAG documentation and AWS’s error handling documentation show those mechanisms in their respective systems. The essential question comes first: are there dependent data tasks whose order, repeat behavior, and failure path need to be explicit?
For a single isolated job with one clear input and output, a larger graph may add ceremony without improving the result. Once a report depends on several upstream datasets, a scheduled trigger alone becomes thin: it does not express which output each calculation is waiting for or which steps must be reconsidered after a correction. The case for orchestration grows when a failed middle step, a repeated period, or a downstream publication needs a deliberate answer. This is an architectural judgment drawn from the task and dependency model, not a claim that every team needs a particular platform.
Start the design on paper with one real path, such as the illustrative daily orders report. Name the source period, the prepared dataset, the calculation output, and the published report. Put an arrow only where the next step genuinely requires the prior result. Then describe what happens if each step fails after doing some work. If an answer requires “just rerun everything,” identify where that could duplicate a write or publish stale data. This short exercise usually exposes the task boundaries that matter more than a feature comparison.
The design has a cost. Defined periods require a policy for corrections. Repeatable writes may require a key or a replaceable partition. Separate tasks require clear handoffs. Run and dataset records require coverage across the systems in use. Those are real obligations, but they are also the parts of the workflow that determine whether recovery is predictable. A tool can execute the choices; it cannot supply the missing business meaning of a Tuesday report.
The test is a corrected past run
A useful way to assess the finished design is to ask how it handles a past day whose input has changed. Can the operator identify the day to reprocess? Will the task read that day’s intended input rather than the current “latest” data? Will the write replace or reconcile the day’s output without doubling it? Will publication wait for the refreshed calculation? Can someone trace which dataset the new run wrote and which downstream result uses it? Each question follows from the same central requirement: a run must have a clear meaning before its completion state can be trusted.
Data orchestration works when it turns those answers into ordinary operating behavior. The graph orders dependent tasks; the run identifies the occasion and period; repeatable task design makes recovery safe; and run-linked dataset information shows where the result went. If those pieces are in place, rerunning Tuesday is a controlled correction. If any piece is missing, a green workflow may still leave the report’s reader with the wrong day, a duplicate, or an unexplained number.