AI & ML

Responsible and Reliable ML

Part of Evaluation & Trust, level 9. About 11 hours of study across the topics below.

Fairness Privacy

  1. Dataset BiasRepresentation, measurement, and history: how datasets encode their world.1–2 h
  2. Fairness ConceptsParity notions and their impossibility: what fairness math can and cannot promise.1–2 h
  3. PrivacyMemorization, minimization, and differential privacy as a promise shape.1–2 h
  4. Fairness Criteria and Their IncompatibilityDemographic parity, equalised odds, and calibration cannot all hold when base rates differ.1–2 h

Safety

  1. Adversarial ExamplesEvasion and poisoning: small perturbations, flipped decisions.1–2 h
  2. HallucinationUngrounded generation: detection, measurement, and mitigation that actually helps.1–2 h
  3. Specification Gaming and Reward HackingOptimisers exploit gaps between the stated objective and the intended one.1–2 h

Oversight

  1. Interpretability: Goals and LimitsAttribution, probes, and the limits of explanations.1–2 h
  2. Human OversightReview, abstention, escalation: autonomy levels with honest handoffs.1–2 h
  3. Documentation, Governance, and Risk ManagementModel cards, datasheets, and risk-management frameworks that make ML systems accountable.1–2 h
  4. Scalable Oversight and Dangerous-Capability EvaluationSupervising systems whose outputs humans struggle to check, and evaluating for high-risk capabilities before deployment.1–2 h