Metadata management
Instruction and application
CompleteMetadata is data about data: who created a dataset, what columns mean, how it was transformed, who may access it and how long it is kept. For ML, metadata is theconnective tissue between raw tables, governed features and explainable outcomes.
Without governance, teams mislabel fields, lose provenance and struggle to prove compliance. This lesson covers types of metadata, standards, tooling and quality practices.
What is metadata?
Metadata can be embedded in files or stored in catalogues. It powers search, access control, retention, interoperability and preservation—especially when datasets feed automated decisions.
How metadata is used
Organisation and cataloguing
— consistent naming, domains and ownership.
Discovery
— find approved datasets quickly instead of copying ad hoc extracts.
Governance
— lifecycle rules, sensitivity labels and audit trails.
Interoperability
— integrate APIs, warehouses and notebooks safely.
Preservation
— document lineage and technical characteristics over years.
Types of metadata
1. Descriptive metadata
What is this dataset? Names, descriptions, keywords, business definitions and glossary links.Examples: catalogue cards; CSV headers; data dictionaries. See NYC Open Data — Wi-Fi hotspots for a public dictionary pattern.
2. Structural metadata
How is it organised? Schemas, keys, relationships and validation rules.Examples: ERD diagrams; JSON schemas; XSD for transactions. See a sample Sakila schema diagram.
3. Administrative metadata
Who owns it and under what rules? ACLs, retention classes, compliance tags.Examples: UK Data Service study metadata showing permitted uses (study 8473).
4. Provenance metadata
Where did values come from and how were they changed? Sources, transforms, versions and approvals.Example: Environmental Information Data Centre provenance sections.
5. Usage metadata
How is it consumed? Query logs, training job references, download counts, API traffic.Example: Kaggle dataset pages show popularity and reuse signals.
Why metadata matters for AI operations
Explainability and bias mitigation:
document training sources, feature rationale and preprocessing so reviewers can test for proxy discrimination.
Regulatory alignment:
GDPR-class regimes expect traceability for personal and sensitive processing.
Collaboration:
analysts spend less time reconciling “shadow” spreadsheets when the catalogue is authoritative.
Standards and frameworks (selected)
[Dublin Core](https://www.dublincore.org/)
— widely used descriptive metadata for discovery and archives.
[ISO/IEC 11179](https://www.iso.org/standard/78914.html)
— metadata registries and harmonised definitions across enterprises.
[FAIR principles](https://www.go-fair.org/fair-principles/)
— Findable, Accessible, Interoperable, Reusable datasets for research and regulated industries.
Catalogue platforms
— Apache Atlas, OpenMetadata, CKAN for publishing and governance patterns.
Tip
On your laptop, open Get Info (macOS) orProperties → Details (Windows) on a file—those panels are everyday descriptive metadata.
Tools and techniques
Extraction and tagging
Automated taggers infer file types, owners, keywords and sensitivity from content. Cloud drives often auto-label documents (for example “Contract”, “PII”) to enforce access rules.
Quality control and validation
Profiling detects missing timestamps, schema drift and ownership gaps. Financial reporting stacks often combine automated validation,timestamp checks andownership verification before regulatory submissions ship.Vendor examples (not exhaustive): Alation, Collibra, Erwin Data Modeler, Informatica data governance.
Repositories and catalogues
Central repositories anchor search, lineage and policy enforcement—for example finding the latest approved customer mart instead of a stale export.
Key point
Metadata schemas are not bureaucracy—they reduce rework, make fairness reviews possible and shrink incident blast radius when something breaks.