What Is a Data Pipeline? Design the Flow from Marketing Events to Decisions

A data pipeline is a repeatable sequence of steps that collects data from one or more sources, validates and moves it, optionally transforms it, and delivers it somewhere people or software can use it. For a marketing team, the useful unit is not a connector or dashboard. It is the whole governed path from a defined event to a decision, including identity, storage, modeling, quality checks, delivery, ownership, and recovery.

A data pipeline in plain language#

Amazon Web Services gives the broad category definition: a data pipeline is a series of processing steps that prepares source data for analysis. Sources can push or be pulled; processing can sort, reformat, deduplicate, verify, filter, or summarize; and the destination can be storage, an analysis surface, or another system.

There is no defining data-pipeline formula. A pipeline is an architecture and operating process, not a calculated metric. You can measure one with freshness, end-to-end latency, throughput, error rate, completeness, and correctness, but no single equation proves that the flow is useful. A diagram with arrows is not enough either: a working pipeline needs executable steps, explicit contracts, tests, ownership, and a repair path.

The most common terminology confusion is about scope:

• ETL is one type of data pipeline: extract the data, transform it, then load it into its destination.
• ELT extracts and loads first, then transforms inside the destination.
• Data pipeline is the wider category. A pipeline can use ETL, use ELT, mix the two, stream events continuously, or move data without transforming it at all.
• Tracking plan is an input contract for a behavioral-data pipeline. It defines which events and properties should be collected and why; it is not the complete source-to-decision system.

Batch and streaming describe when the flow processes data. A batch pipeline processes a bounded group on a schedule or trigger. A streaming pipeline handles an ongoing event sequence. Neither is inherently more mature. The right choice follows the latest time at which a consumer can still make the intended decision.

Most marketing pipelines can be understood through six stages:

Here is an illustrative marketing example, not company data. A team wants to decide which acquisition sources produce accounts that reach a defined activation state. A website records campaign context and an anonymous identifier; the product records account creation and the activation event; a revenue system contributes the account’s commercial state. The pipeline lands those records, links them under documented identity rules, builds one activation row per account and acquisition cohort, tests the result, and publishes it to a weekly planning view. The view is the final surface, not the pipeline itself.

Start with a decision contract, not a tool#

The first design step is to write what someone will do differently when the data arrives. Twilio’s tracking-plan guidance starts in the same place: define the business question, identify the measures required to answer it, and only then derive events and properties.

For the illustrative activation pipeline, the decision contract could look like this:

This contract prevents two expensive forms of ambiguity. First, the data team knows when the pipeline is complete: when the named consumer receives decision-ready data by the agreed time. Second, the marketing team cannot silently redefine “activation,” population, or source treatment inside a chart.

Write the event contract before instrumentation#

An event should describe an observable occurrence, not a dashboard label. Twilio Segment’s Track specification provides one concrete envelope: an event records an action, properties add detail, user or anonymous IDs connect it to an identity, and timestamps and context describe when and where it occurred.

The portable lesson is to make every collected field part of a contract. For each event, record:

Avoid event names tied to interface details such as a button color. Interfaces change while business events should remain interpretable. account_created describes a durable occurrence; blue_button_clicked does not explain whether an account exists. Interface interactions can still matter, but collect them only when a defined question requires that evidence.

Schema versioning matters because producers and consumers do not always change together. A new property may be backward-compatible. Renaming an event, changing a type, or redefining its trigger may not be. Treat those changes like interface changes: document them, test consumers, provide a migration window when needed, and preserve the old interpretation long enough to reconcile history.

Set identity, time, and privacy rules explicitly#

A stream of correct events can still yield a wrong account-level answer when identity rules are implicit. Decide which identifiers represent a browser, session, person, account, campaign, and source record; which system owns each identifier; and which joins are permitted.

Twilio Segment’s Identify specification illustrates the distinction between anonymous and known identity. It permits an anonymous ID before a person is known and recommends a stable database ID rather than a mutable email address or username for the durable user key.

That is a product-specific model, not a universal merge rule. Your identity contract still needs to answer hard cases: one person using several devices, several people in one buying account, a person changing employers, a shared email address, a deleted profile, and an anonymous event that never becomes identified. Never merge merely because two records look similar unless the approved rule says that evidence is sufficient. Preserve the contributing source IDs so a merge can be explained or reversed.

Time needs the same precision. Keep at least the concepts of event time (when the business action occurred), receipt time (when collection received it), and processing time (when a model incorporated it). Then define how late events, clock skew, time zones, and backfills affect a cohort. Without that contract, the same account can move between periods merely because a job ran late.

Marketing events can contain personal data. Where the GDPR applies, the European Commission’s summary of processing principles includes purpose limitation, data minimization, accuracy, storage limitation, integrity, confidentiality, and accountability.

This is not a complete compliance design. It is enough to reject “collect everything in case it becomes useful” as a pipeline requirement. Record the purpose, lawful handling basis where applicable, consent or preference dependency, allowed consumers, access controls, retention or deletion trigger, and qualified legal owner before routing personal data to additional destinations.

Choose batch or streaming from the decision deadline#

Streaming is justified when the action loses material value if it waits: fraud response, time-sensitive product behavior, or an operational message that must react to a current state. A weekly planning decision usually does not require a continuously running path. A scheduled batch or micro-batch is often easier to reconcile, replay, and operate.

AWS distinguishes batch and streaming pipelines by their execution pattern: batches process bounded volumes at intervals, while streams continuously process events and must tolerate conditions such as delayed or out-of-order arrival.

Many marketing systems need both. A time-sensitive event can enter through a stream while a scheduled process reconciles it with ad-platform, CRM, billing, or product records. The architecture should expose the two freshness levels rather than letting consumers assume that every field is equally current.

Whatever the movement pattern, preserve a raw or minimally altered landing layer when the risk and storage policy allow it. Include source record IDs, schema versions, event and receipt times, ingestion metadata, and rejection reasons. That layer gives you evidence for replay and reconciliation. It should not become an unrestricted duplicate of sensitive data; access, retention, and deletion requirements still apply.

Design replay before the first incident. A repairable pipeline can reprocess an event or bounded period without multiplying outcomes, isolate malformed records instead of silently discarding them, and distinguish a retry from a genuinely new business event. The precise technique depends on the system, but the acceptance criterion is plain: after a controlled replay, the consumer model represents the same real-world occurrences once.

Separate source evidence from business meaning#

Raw events report what producers emitted. Decision models state what the organization means. Keep those responsibilities in separate layers so a source repair does not require rewriting every dashboard and a metric change does not destroy the original evidence.

A practical model stack is:

1. Raw source records: append source evidence with receipt metadata and minimal handling.
2. Staging models: normalize names and types, expose source keys, and quarantine invalid records without changing business meaning.
3. Core entity models: represent governed people, accounts, campaigns, activities, and their relationships.
4. Decision models: apply population, grain, state, time-window, and metric rules for a named use case.
5. Serving surfaces: expose the approved model to reports or operational consumers without reimplementing its core logic.

For every decision model, document the grain, eligible population, included states, time semantics, joins, exclusions, attribution convention, owner, and version. A chart title such as “activated accounts by source” is not a metric contract. It leaves unanswered whether the row represents a person or account, which event counts as activation, which source wins, how late data behaves, and whether internal or test records are excluded.

Turn every boundary into an acceptance test#

Pipeline testing should follow the data path, not stop at whether a scheduled job returned success. dbt’s data-test documentation describes assertions on models and sources, including reusable checks for uniqueness, non-null values, accepted values, and relationships, plus custom SQL assertions for business-specific rules.

Map tests to failure modes at each boundary:

Add an end-to-end trace because local tests cannot prove that all boundaries agree. Use a controlled record with known inputs, follow its event IDs and entity keys through every layer, and confirm the final model’s state and exclusions. Repeat the trace for a duplicate event, a late event, a missing identifier, an invalid property, and a schema-version change. Those cases test the actual promises your design makes.

Monitor the consumer-visible result#

A pipeline can have healthy servers and successful jobs while delivering stale or wrong data. Google Cloud’s infrastructure reliability guide identifies throughput and latency as important data-pipeline indicators because they affect freshness, and it treats correctness as an essential reliability indicator for most applications.

There is no cross-industry threshold that makes a marketing pipeline “good.” Set targets from the consumer contract:

• Freshness: Is the latest usable decision model recent enough for the named review or action?
• End-to-end latency: How long passes between the relevant source event and consumer-ready state?
• Coverage: Did the expected sources, periods, partitions, and eligible records arrive?
• Validity and uniqueness: Do values match their contracts, and does the declared grain hold?
• Correctness: Do controlled traces and reconciliations produce the expected business state?
• Recovery: Can the team repair a missed or corrupt period within the decision’s useful window?

Measure at the consumer-visible endpoint as well as within jobs. If the model refreshed but the report cache did not, the decision surface is stale. If a job completed after reading only part of a source period, execution succeeded but coverage did not. Alert the owner who can act, include the affected dataset and period, and show consumers when data is stale or under repair.

Lineage makes that response faster. OpenLineage models metadata about datasets, jobs, and runs so teams can trace how data was produced, investigate a root cause, and assess which downstream outputs a change may affect.

Lineage is a map, not a quality guarantee. Pair it with run history, schema versions, code or configuration versions, test results, ownership, and a backfill log. When recovery completes, reconcile the repaired output against source evidence and record which consumers were republished.

Prove one vertical slice before expanding the pipeline#

Build the smallest path that supports one real decision. The dependencies are deliberate:

Do not add the next source merely because a connector exists. Add it when the decision contract names evidence the current path lacks. Do not switch to streaming merely because the platform supports it. Switch when the consumer’s action has a verified latency need. Do not call the pipeline complete when the dashboard renders. Call it complete when the decision-ready output can be explained, tested, monitored, repaired, and owned.

The practical takeaway is a one-page source-to-decision map. If one row is blank, the design is not ready to implement: