Skip to main content

Metadata management

Instruction and application
Complete

Metadata is data about data: who created a dataset, what columns mean, how it was transformed, who may access it and how long it is kept. For ML, metadata is theconnective tissue between raw tables, governed features and explainable outcomes.

Without governance, teams mislabel fields, lose provenance and struggle to prove compliance. This lesson covers types of metadata, standards, tooling and quality practices.

What is metadata?​

Metadata can be embedded in files or stored in catalogues. It powers search, access control, retention, interoperability and preservation—especially when datasets feed automated decisions.

How metadata is used​

  • Organisation and cataloguing — consistent naming, domains and ownership.
  • Discovery — find approved datasets quickly instead of copying ad hoc extracts.
  • Governance — lifecycle rules, sensitivity labels and audit trails.
  • Interoperability — integrate APIs, warehouses and notebooks safely.
  • Preservation — document lineage and technical characteristics over years.

Types of metadata​

1. Descriptive metadata​

What is this dataset? Names, descriptions, keywords, business definitions and glossary links.Examples: catalogue cards; CSV headers; data dictionaries. See NYC Open Data — Wi-Fi hotspots for a public dictionary pattern.

2. Structural metadata​

How is it organised? Schemas, keys, relationships and validation rules.Examples: ERD diagrams; JSON schemas; XSD for transactions. See a sample Sakila schema diagram.

3. Administrative metadata​

Who owns it and under what rules? ACLs, retention classes, compliance tags.Examples: UK Data Service study metadata showing permitted uses (study 8473).

4. Provenance metadata​

Where did values come from and how were they changed? Sources, transforms, versions and approvals.Example: Environmental Information Data Centre provenance sections.

5. Usage metadata​

How is it consumed? Query logs, training job references, download counts, API traffic.Example: Kaggle dataset pages show popularity and reuse signals.

Why metadata matters for AI operations​

  • Explainability and bias mitigation: document training sources, feature rationale and preprocessing so reviewers can test for proxy discrimination.
  • Regulatory alignment: GDPR-class regimes expect traceability for personal and sensitive processing.
  • Collaboration: analysts spend less time reconciling “shadow” spreadsheets when the catalogue is authoritative.

Standards and frameworks (selected)​

  • Dublin Core — widely used descriptive metadata for discovery and archives.
  • ISO/IEC 11179 — metadata registries and harmonised definitions across enterprises.
  • FAIR principles — Findable, Accessible, Interoperable, Reusable datasets for research and regulated industries.
  • Catalogue platforms — Apache Atlas, OpenMetadata, CKAN for publishing and governance patterns.

Tip​

On your laptop, open Get Info (macOS) orProperties → Details (Windows) on a file—those panels are everyday descriptive metadata.

Tools and techniques​

Extraction and tagging​

Automated taggers infer file types, owners, keywords and sensitivity from content. Cloud drives often auto-label documents (for example “Contract”, “PII”) to enforce access rules.

Quality control and validation​

Profiling detects missing timestamps, schema drift and ownership gaps. Financial reporting stacks often combine automated validation,timestamp checks andownership verification before regulatory submissions ship.Vendor examples (not exhaustive): Alation, Collibra, Erwin Data Modeler, Informatica data governance.

Repositories and catalogues​

Central repositories anchor search, lineage and policy enforcement—for example finding the latest approved customer mart instead of a stale export.

Key point​

Metadata schemas are not bureaucracy—they reduce rework, make fairness reviews possible and shrink incident blast radius when something breaks.

Action item: Quiz​

Question 1 of 2
Which metadata category best captures table keys, relationships and validation rules?
  • A. Structural metadata
  • B. Usage metadata only
  • C. Network throughput metadata
  • D. Hyperparameter metadata
Correct Answer: A

Feedback: Structure describes how fields relate; usage tracks consumption patterns separately.

The FAIR principles primarily promote datasets that are:
  • A. Fully anonymous
  • B. Immutable forever
  • C. Encrypted with a single vendor algorithm
  • D. Findable, Accessible, Interoperable and Reusable
Correct Answer: D

Feedback: FAIR is a metadata and publishing practice set for trustworthy reuse.