Responsible data collection and usage
Instruction and application
CompleteYou have already explored legislation, privacy controls and ethical frameworks. This lesson focuses on how those ideas apply when you collect and use data for AI and ML.
Why data ethics matters
AI models are only as good as the data they learn from. Biased, incomplete or irresponsibly collected data produces systems that skew decisions, discriminate and erode trust with users and regulators.
Ethical collection is not about gathering as much data as possible. It is about ensuring data is lawful, proportionate, representative and high quality for the problem you are solving.
Key principles for ethical data collection
Legitimate interests (UK GDPR)
Under UK GDPR, Legitimate Interests can be a lawful basis for processing when the processing is necessary and abalancing test shows that your interests are not overridden by individuals' rights.
- You must still be transparent: explain what you collect, why and how AI uses it.
- Provide this in plain language (for example a concise privacy notice linked at the point of collection).
- Users retain rights including the right to object to Legitimate Interests processing. Your notice should say how to exercise that right.Example: A health app processes aggregated biometric signals for service improvement. It should state the purpose, why processing is necessary, and how users can object, even when data is aggregated or pseudonymised.
Diversity and representativeness
Datasets should reflect the real world across the dimensions that matter for the use case:
- Demographics such as age, ethnicity, geography and socioeconomic context where relevant.
- Temporal bias: training on an unusual period (for example heavy lockdown travel data) can misrepresent “normal” behaviour.
- Geography: models trained mainly on urban data may fail or act unfairly in rural contexts.
- Features: missing or underrepresented variables produce incomplete models.
Representation is not one-off: monitor datasets and model performance as populations and behaviours change.
Proportionality (data minimisation)
Collect only what you need for the stated purpose. Over-collection increases privacy risk and breach impact.
Example: A fitness tracker should not harvest contact lists or browsing history if those are irrelevant to its purpose.
The risks of poor data collection
Car insurance and the ethnicity penalty
Postcodes and similar proxies can correlate with protected characteristics. If pricing models lean on such proxies without careful fairness analysis, people in some areas can face higher quotes despite similar risk profiles.
Health care algorithm favouring white patients
Using past health care costs as a proxy for need can encode systemic access inequalities. Lower historical spend does not always mean lower clinical need.
Google Translate gender bias
Web-scale training text reflects societal stereotypes. Without debiasing and evaluation, models can map occupations to gendered language in harmful ways.
Unity ad targeting error
Faulty customer demographic and behaviour data flowed into ML pipelines, reportedly causing major revenue impact and costly recovery work.
Key point
In each case, harm or loss came from data that was a poor proxy, unrepresentative or low quality. You are responsible for ensuring training data is fit for purpose, collected lawfully and assessed for bias.
Strategies for identifying and mitigating bias
Dataset audits
A dataset audit reviews quality, completeness, consistency and compliance. Organisations should audit training data for disparities across race, gender, geography and socioeconomic groups. On a new project, start with exploratory analysis and escalate gaps you find.Example: If a résumé screener rejects one group at higher rates, audits can reveal biased labels or skewed sampling.
Reweighting and resampling
Reweighting adjusts sample weights so underrepresented groups contribute more during training (for example weight 9.0 on a 10% class). Watch for overcorrection harming the majority.Resampling balances classes by oversampling the minority or undersampling the majority. Duplicates, synthetic data or dropped rows each carry reliability risks.
Transparency in data labelling
Human labels drive many models. Reduce label bias with:
- Clear guidelines and a stable schema.
- Diverse, trained annotators and inter-rater checks.
- Periodic audits of labelled data against the schema.
Data quality and integrity in ethical ML
Strong practice includes:
- Accuracy from the start: validated types, correct columns, reliable labels.
- Representation across the population and contexts the model will serve.
- Source documentation: provenance, licences and ethical sourcing.
- Fair collection methods: avoid skewed convenience samples where they create bias.
- Processing transparency: documented cleaning, joins and feature engineering.
- Versioning and traceability: which dataset version trained which model.
- Auditability: lineage and reproducibility so third parties can verify fairness claims.
Case study: discriminatory diagnostics
A widely used skin cancer model underperformed on darker skin because training images skewed toward lighter skin tones. Underrepresentation led to misdiagnosis and delayed care. See NCBI for further discussion.
Why this matters
High-integrity, representative data is the foundation of fair, reliable models. Without it, technical sophistication cannot undo bad inputs.
Storage, retention and disposal
AI systems often hold sensitive financial, health, biometric and communications data. Weak storage becomes a breach magnet.
Key point
Use defence in depth: encryption, access controls, least privilege, logging and anonymisation or pseudonymisation where appropriate.
Protecting data from unauthorised access
Role-based access, MFA, separation of duties and tight controls on exports (including email and removable media) reduce insider and external risk.
Secure storage and backups
Maintain encrypted backups in resilient environments. Many organisations blend on-premises and cloud storage while meeting regulatory requirements.
Encryption
Common patterns include AES-256 for data at rest andTLS for data in transit. Email is often a weak channel; avoid sending sensitive training data as attachments when secure transfer is available.
Example: 2017 Equifax breach
A known vulnerability left systems exposed, affecting tens of millions of people. The case underscores patching, segmentation, monitoring and incident readiness—not only “AI model” risk but platform hygiene.
Purpose-driven retention
Retain data only as long as needed for the stated purpose and legal requirements. Excess retention increases breach blast radius.
- Periodic data audits check accuracy, relevance, security and compliance.
- Automated retention schedules delete or archive data on policy.
- User-controlled deletion: for example Google auto-delete controls.
Example: 2019 Facebook / FTC order
Long retention and weak controls around third-party access contributed to a landmark penalty and new restrictions. Define retention periods and enforce them with automation where possible.
Ethical disposal when data is no longer needed
Poor disposal enables recovery and misuse. Common approaches:
| Method | Role | Caveat |
|---|---|---|
| Soft delete | Reversible tombstone for operations | Not sufficient for full erasure requests |
| Cryptographic erasure | Destroy keys so ciphertext is unreadable | Needs strong crypto and key governance |
| Secure overwrite | Replace data with random patterns | Can be slow; SSD wear-levelling complicates guarantees |
| Physical destruction | Shred drives for highest sensitivity | Costly; impractical for pure cloud abstractions |
Example: 2020 Morgan Stanley OCC matter
Inadequate erasure before decommissioning hardware reportedly allowed recovery of customer data, illustrating why secure destruction processes must be operational, not aspirational.
Action item: Quiz
- A. Using too many features in a model
- B. Letting data from an unusual time period dominate training so it no longer reflects typical behaviour
- C. Encrypting time-series data incorrectly
- D. Collecting data only from mobile devices
Feedback: If a narrow window (for example a crisis period) is not representative of normal operations, models can learn the wrong “baseline”.
- A. Obtain explicit consent for every processing activity
- B. Publish model weights publicly
- C. Perform a balancing test and document why interests are not overridden by individuals' rights
- D. Avoid providing a privacy notice if data is anonymised
Feedback: Legitimate Interests requires necessity and a documented balancing test, plus transparency and objection rights.