Data Lineage:
What It Is and How to Trace a Metric

Data lineage records where data comes from, how it is transformed, and where it is used. It connects a reported metric to the source data and processing steps behind it. The NIST glossary defines lineage through a data element’s processing history, including transfers and the actions performed on it.

Lineage supports investigation in two directions. Upstream tracing follows an output back to its inputs; downstream tracing follows an input toward the outputs that depend on it. Microsoft’s lineage overview identifies troubleshooting, root cause investigation, and change impact analysis as uses of these relationships.

For a metric, the practical goal is to identify the source fields, transformations, and reporting calculations that produce the displayed value. A useful trace also identifies the processing execution being examined and the consumers affected by a change.

What a data lineage map contains#

A lineage map represents dependencies between data assets and processing steps. In Microsoft Purview’s entity-level model, source and target assets are connected through the process that moves or transforms the data. The map can extend from raw data through prepared datasets into visualization.

Read each connection as an input, an operation, and an output. The operation deserves as much attention as the asset names: record whether it copies data, derives a field, joins datasets, filters records, or aggregates values. Keep the transformation code or configuration accessible alongside the relationship.

OpenLineage’s object model distinguishes three objects:

• Dataset: an identifiable collection of data.
• Job: a process that consumes or produces datasets.
• Run: one occurrence of a job.

Record fully qualified asset identifiers and the relevant run identifier with the trace.

Table-level, column-level, and execution detail#

Choose the detail needed to answer the question. Google’s lineage visualization documentation distinguishes dependencies between tables from dependencies between individual columns. It identifies column-level tracing as useful for verifying the source fields used to calculate a metric.

In OpenLineage, design events have no associated run; runtime events describe an execution. Preserve that distinction when examining historical results.

Column lineage also needs to account for fields that influence a result without being copied into it. The OpenLineage column lineage specification distinguishes direct derivations from indirect dependencies, including join conditions, grouping, and filters. For a metric trace, inspect those dependencies alongside the fields being counted or summed.

How data lineage is collected#

Lineage collection combines metadata from the systems that store, transform, and consume data. Three collection methods provide different kinds of evidence:

• Code and metadata analysis. Extract dependencies from SQL statements or declared processing metadata. The OpenLineage integration documentation describes its SQL parser, which extracts lineage from SQL statements. Label declared dependencies separately from execution observations.
• Runtime collection. Capture processing observations as jobs execute. Microsoft’s collection guide describes ETL tools reporting lineage at execution time. Check which details the integration actually emits.
• Manual or custom documentation. Add verified relationships that existing integrations do not capture. Microsoft’s classic Purview user guide supports manual lineage for sources where automation is unavailable and custom entries where automatically generated lineage is incomplete.

Keep the collection method visible for each relationship. A manual entry should identify its supporting code, configuration, or other evidence and the date it was checked. An automatically collected edge should identify its producing integration and relevant execution or observation time. Leave unsupported connections marked as gaps until their evidence is available.

How to trace a metric across systems#

The following procedure combines dependency mapping with inspection of the calculation and execution records. Start with one named output and follow its documented relationships.

1. Identify the exact metric and reporting context#

Record the report or dashboard, metric name, displayed value, reporting period, time zone, filters, and refresh time. Locate the definition and the expression that implements it.

Capture the aggregation, input expression, inclusion rules, and time field. dbt’s simple metric documentation illustrates why these details belong in a definition: its specification distinguishes the aggregation function, column expression, dimension filters, and aggregation time dimension. For a ratio, also record the numerator and denominator definitions; for a distinct count, record the identifier being counted.

Keep the definition version with the trace. Use that version consistently when comparing source data, intermediate results, and the displayed output.

2. Follow the reporting layer back to its inputs#

Inspect the report’s own calculations and filters, then identify the dataset or query that supplies it. Continue through prepared datasets and their producing jobs until the source assets are identified.

Include the reporting layer in this inspection. The Purview user guide documents reporting lineage that includes datasets, reports, and dashboards. At each boundary, record the source identifier, destination identifier, and process linking them. Mark an unidentified input as a gap requiring investigation.

3. Inspect the transformations that affect the result#

For every step, read the calculation or configuration and record the relevant field mappings. Include joins, grouping, filters, deduplication rules, null handling, and unit conversions wherever they occur in the implemented path.

Use the direct and indirect dependency distinction to guide the inspection. Follow fields used to derive the output and fields that control which records contribute to it. Compare intermediate outputs against the recorded metric definition, using the same period and population throughout the investigation.

4. Connect the dependency path to a specific execution#

Collect the run identifier, execution timestamps, completion status, and applicable code or configuration version. Inspect the input partitions and dataset versions available for that run.

OpenLineage’s object model supports metadata for code versions, dataset versions, and run-specific partitions. Use the fields actually collected. Supplement missing evidence with available job logs or query history, and document what could not be established.

5. Trace downstream uses and retain the evidence#

Follow the recorded dependencies forward to consuming datasets, reports, and exports. Identify the consumers that require inspection before a source field or calculation changes. Confirm their input mappings and any additional transformations against the current configuration.

Retain a compact record for each handoff:

• Source and destination asset identifiers, with relevant fields.
• Transformation code or configuration reference and version.
• Collection method and supporting evidence.
• Applicable run or reporting period, plus the observation date.
• Responsible owner and unresolved gaps.

Check the trace again when a source schema, transformation, metric definition, or consuming report changes. Keep the earlier execution evidence available for questions about historical results.

Data lineage, catalogs, provenance, and observability#

These concepts overlap, but their documented roles differ.

A data catalog organizes metadata about assets. The AWS Glue Data Catalog is a centralized metadata repository that indexes data locations, schemas, and runtime metrics. Catalog metadata can help identify an asset; tracing a metric also requires the relationships and transformations between assets.

Data provenance can describe more of the context behind a result. The W3C PROV overview includes the entities, activities, and people involved in producing data or other things. Lineage’s emphasis on processing history fits within that context; terminology alone does not establish which details a particular system records.

Data observability concerns data health across systems and pipelines. IBM’s overview discusses freshness, distribution, volume, schema, and lineage together. Use health signals and quality checks alongside the dependency path to investigate whether inputs and outputs meet their requirements.

Coverage and limits of a lineage trace#

A map’s coverage depends on what its integrations capture. The Purview user guide explicitly acknowledges missing or incomplete automated lineage. Assess coverage by checking the named systems, operations, fields, and downstream consumers included in the trace.

Table-level coverage also needs separate verification from field-level coverage. Google’s documented lineage limitations include missing column-level capture for BigQuery load jobs and routines. These are limits of that service; check the corresponding documentation for each implementation being used.

Lineage metadata may also omit the underlying values needed to reproduce a calculation. Google’s lineage documentation states that its service captures movement metadata rather than the data itself and documents limits on retained history. Verify access to the relevant source data, transformation version, and execution evidence separately.

For a metric investigation, document the trace’s boundary: the output examined, the definition and period used, the relationships supported by evidence, and the remaining gaps. Validate input quality and calculation correctness alongside that record. This gives the metric an inspectable processing history and makes the next unresolved check explicit.