Data lineage
Instruction and application
CompleteThe phrase “farm to plate” tracks how food is produced and transported.Data lineage does the same for information: where it originated, how it was transformed and where it flows inside your systems.
What is data lineage?
Data lineage is the complete lifecycle of data—creation, movement, joins, feature engineering and consumption. For ML, lineage underpins integrity, traceability, debugging and compliance. Models depend not only on row-level quality but onevery upstream choice that shaped the training view.
Poor tracking leads to silent drift, unfair outcomes and audit failures. Strong lineage helps you explain which data produced which decision.
Benefits of tracking lineage
Model reliability:
verify that training views reflect current, approved sources.
Faster debugging:
when predictions shift, trace whether a feed, join or feature code changed.
Compliance:
demonstrate lawful processing and retention paths in regulated sectors.
Trust:
stakeholders can follow data from source to score.
Technical foundations
Data provenance
Provenance records origin and history: who created data, when, why and what changed. Metadata is often the first place to look—timestamps, owners, purpose and transform logs reduce the risk of training on stale, biased or mislabelled data.Example: The Environmental Data Initiative requires rich metadata so users can trace ecological datasets; see their provenance documentation.
Lineage representation
Lineage is often shown as graphs or flow diagrams: tables and files as nodes, pipelines as edges. Visual lineage helps engineers and auditors spot bottlenecks, missing documentation and risky shortcuts.Example: Collibra Data Lineage illustrates graph-style flows between assets.
Lineage tracking tools (examples)
- Apache Atlas — metadata governance and lineage.
- DataHub — cataloguing and lineage at scale.
- OpenLineage — open standard for pipeline events.
- Great Expectations — data quality tests and documentation tied to datasets.
Action item: Pause and think
Does your organisation catalogue lineage for core ML features? If a tool exists, open a flow and follow one production feature from raw source to model input.
Practical applications
Impact analysis: credit scoring and economic data
A bank adds macroeconomic indicators to a credit model. Without lineage, analysts might miss that the new feed over-represents wealthier postcodes. Approval rates drop for disadvantaged groups; issues surface months later under regulatory pressure.How lineage helps: impact analysis ties model shifts tospecific ingest steps and features so teams can review representativenessbefore broad rollout.
Debugging: predictive maintenance drift
Sensor firmware changes temperature readings by ~10%. A failure-prediction model suddenly over-alerts. With lineage, engineers trace the jump to the sensor software version feeding the training path, recalibrate inputs and retrain—days instead of weeks of blind tuning.
Reproducibility and auditability
For high-stakes domains, you must reconstruct exactly which data and transforms produced a decision. Lineage plus versioned datasets and code is the practical backbone of reproducibility.
Ethical dimensions
Transparency
Users and regulators need to see how data was collected, joined and filtered—otherwise systems feel like black boxes.
Hiring example: candidates should be able to understand which fields influenced screening and whether automated steps applied.
Explainability
Clinicians need to see which vitals and history drove a risk score. Lineage connects inputs and transforms to the explanation layer.
Accountability
When harm occurs, lineage shows whether biased proxies (for example postcode), bad joins or stale snapshots were involved—supporting remediation and policy fixes.
Further reading: CFPB guidance on adverse action with complex models.