Worked Example — Algorithmic Impact Assessment of an AI Hiring System

1 · Overview

Goal, scope & deliverable

This is the kind of expert-report the individual-work component (40%) asks for: state the system, the affected rights and principles, the applicable law, and a reasoned recommendation.

The brief. A mid-size European software firm wants to deploy TalentRank, an AI system that ingests submitted CVs, parses them against a job description, scores each candidate 0–100 and ranks a shortlist for recruiters. It is trained on ten years of the company's own past hiring decisions. Before launch, the firm commissions an Algorithmic Impact Assessment (AIA): is the system lawful, is it ethical, and what must change before it can go live?

This document works that assessment end-to-end. It is deliberately analysis and policy only — no code — and it follows the course's own method: pick the right ethical lens, attach the right legal instrument, and produce an actionable recommendation rather than a gesture at "ethics in general".

Goal

Decide whether TalentRank may be deployed, and on what conditions, by assessing legality (AI Act + GDPR), ethics and risk.

Sessions exercised

S2 principles & frameworks · S4 AI Act · S5 GDPR & DPIA · S6 bias & explainability · S7 accountability.

Method

Classify → audit → analyse ethically → register risks → mitigate → recommend. An expert-report structure.

Deliverable

This AIA plus a one-page policy memo with a go / no-go recommendation and conditions.

Why this case. CV-screening is the textbook EU AI Act high-risk use (employment & worker management, Annex III) and it processes personal data, so it sits exactly at the intersection of the two regimes the course drills hardest — the AI Act (Sessions 3–4) and the GDPR (Session 5).

2 · The system & stakeholders

What TalentRank does, and to whom

Before any legal or ethical judgement, map the system's data flow, its actors and who carries the risk.

How it works

Inputs. Candidate-submitted CVs (free text + PDF), the structured job description, and — critically — 10 years of historical hiring outcomes as training labels (who was interviewed, who was hired, who was promoted). Processing. The model extracts features (skills, education, employer prestige, gaps in employment, keywords) and outputs a 0–100 match score and a ranked shortlist. Output & use. Recruiters see the ranking and, in the firm's first design, auto-reject everyone below a score of 40 without human review.

Two red flags in the design itself. (1) Training on past human decisions means the model learns the firm's historical preferences — including any past discrimination. (2) Auto-rejecting below a threshold with no human looking at it is an automated decision producing legal/significant effects on the candidate — squarely within GDPR Art. 22.

Stakeholders & what is at stake for each

Candidates (data subjects)

Bear the harm: unfair rejection, opaque scoring, no real chance to contest. The party with most at stake and least power.

The firm (deployer)

Wants speed and cost savings; carries AI Act "deployer" duties and GDPR "controller" liability and reputational risk.

The vendor (provider)

Built the model; under the AI Act carries the heaviest high-risk "provider" obligations (conformity, documentation).

Recruiters (users)

Risk "automation bias" — over-trusting the score — and de-skilling of their own judgement.

Regulators (AEPD, market surveillance)

Enforce GDPR and the AI Act; can fine, audit or ban deployment.

Society

If many firms adopt similar tools, historical bias scales into a structural barrier to employment.

3 · Legal classification

EU AI Act tier & GDPR mapping

Two instruments apply at once. The AI Act governs the system; the GDPR governs the personal data it processes.

EU AI Act — risk tier ↳ see the pyramid

Classification: HIGH-RISK. Under the EU AI Act, AI systems used for recruitment and selection — in particular to screen or filter applications and evaluate candidates — are listed as high-risk (Annex III, area "employment, workers management and access to self-employment"). It is not an unacceptable-risk practice (it is not banned social scoring), but it sits one tier below, carrying the heaviest set of compliance obligations short of prohibition.

Obligations that follow (provider & deployer checklist):

Risk-management system across the lifecycle — identify, evaluate and mitigate foreseeable risks to health, safety and fundamental rights. AI Act Art. 9
Data & data governance — training/validation/test data must be relevant, representative and examined for bias. Art. 10
Technical documentation demonstrating conformity, kept up to date. Art. 11 + Annex IV
Automatic record-keeping (logging) for traceability over the system's lifetime. Art. 12
Transparency & instructions for use so the deployer can interpret output correctly. Art. 13
Human oversight — designed so a person can understand, monitor and override the system. Art. 14
Accuracy, robustness & cybersecurity appropriate to the use. Art. 15
Conformity assessment + CE marking + EU database registration before placing on the market. Arts. 43, 48–49
Deployer duties — use per instructions, ensure human oversight, monitor, and inform affected workers/candidates. Art. 26
Fundamental Rights Impact Assessment by the deployer before putting the high-risk system into use. Art. 27

GDPR — principles & data-subject rights mapping ↳ see GDPR

GDPR principle (Art. 5)	Applied to TalentRank	Status
Lawfulness, fairness & transparency	Need a lawful basis (likely legitimate interest, balanced against candidate rights). "Fairness" is breached if scoring discriminates. Candidates must be told the logic exists.	At risk
Purpose limitation	Historical hiring data was collected to make past decisions, not to train a model. Reusing it is a new purpose needing a compatibility test.	Needs test
Data minimisation	Features like "employment gaps" or address proxy little legitimate signal but high discrimination risk; should be dropped.	Excessive
Accuracy	A 0–100 score implies a precision the model does not have; mis-parsed CVs produce wrong scores.	Weak
Storage limitation	Rejected-candidate data and scores must have a defined, short retention period.	Undefined
Integrity & confidentiality	Scores and CVs are sensitive; require access control and security.	Manageable
Accountability	Controller must demonstrate compliance — i.e. a documented DPIA, not just good intentions.	Required

The decisive GDPR hook: Article 22. Auto-rejecting candidates below a threshold with no human involvement is a solely automated decision with legal or similarly significant effects. That is restricted by default and triggers extra safeguards: the right to human intervention, to express a view and to contest the decision, plus meaningful information about the logic involved (Arts. 13–15, 22). Because the processing is high-risk, a DPIA (Art. 35) is mandatory.

4 · Ethical analysis

Two lenses, then the gaps

Legality is the floor, not the ceiling. The course's method: argue the case from both foundational lenses, then name the specific principle each gap violates. ↳ Six principles

Consequentialist lens

"Which action produces the best overall outcomes?"

For deployment: faster screening, lower cost, consistent criteria applied to every CV, and recruiters freed for higher-value interviews — a real efficiency gain.
Against: if the model encodes historical bias, the aggregate harm is large — qualified candidates filtered out at scale, eroded trust, reputational and legal cost, and a chilling effect on applicants.
Verdict: the expected-value calculation flips negative unless bias is measured and controlled. The efficiency is real but conditional on fairness safeguards.

Deontological lens

"Which action respects duties and rights, whatever the outcome?"

Duty of respect: candidates are ends, not data points; being silently auto-rejected by a black box treats them as means and denies their dignity and autonomy.
Right to explanation & contest: a decision that cannot be justified to the person it affects violates a duty regardless of how "efficient" it is.
Verdict: even a perfectly accurate model would still owe candidates transparency, a human in the loop and a route to contest — these are non-negotiable duties, not trade-offs.

Where the lenses converge. Consequentialism says "fix the bias or the harms outweigh the gains"; deontology says "respect the candidate's rights whatever the gains". Both point to the same design: measured fairness, transparency, and meaningful human control — which is also exactly what the AI Act and GDPR require. Ethics and law line up.

Bias & accountability gaps ↳ AI challenges

1 · Historical bias laundering

Trained on past human decisions, the model reproduces and legitimises any prior discrimination — now wrapped in a veneer of mathematical objectivity ("the algorithm said so").

Mitigate: audit training data and outputs for disparate impact across protected groups; remove proxy features; re-weight or re-label.

2 · Proxy discrimination

Even with gender/ethnicity removed, features like postcode, name, employment gaps or university stand in as proxies, so bias persists indirectly.

Mitigate: proxy-feature analysis; drop high-risk/low-signal features; test fairness metrics, not just accuracy.

3 · Opacity / explainability gap

A 0–100 score with no reasons cannot be contested by a candidate or justified by a recruiter — the "black box" problem.

Mitigate: per-decision explanations (top factors), documented model logic, and the GDPR "meaningful information about the logic".

4 · Accountability gap

With a vendor, a deployer and recruiters all in the chain, "no one is responsible" becomes the default when a candidate is wrongly rejected.

Mitigate: contractually allocate provider vs deployer duties; name an accountable owner; log every decision for traceability.

5 · Automation bias

Recruiters tend to defer to the score even where they are nominally "in the loop", hollowing out the human oversight that the law requires.

Mitigate: present score as one input among several, withhold the rank until after a human read, train recruiters on override.

6 · No recourse

Candidates cannot see, question or appeal their score — failing both the deontological duty and GDPR Art. 22 safeguards.

Mitigate: notice that AI is used, a human-review channel, and a documented contest/appeal process.

5 · Risk register

Risks, severity & mitigations

A structured register linking each risk to the principle/law it threatens, a severity rating and a concrete control.

#	Risk	Threatens	Severity	Mitigation	Owner
`R1`	Discriminatory shortlisting from biased training data	Equality; GDPR fairness; AI Act Art. 10	High	Disparate-impact audit across protected groups before & after deployment; bias-corrected retraining; drop proxy features	Provider + DPO
`R2`	Unlawful solely-automated rejection (auto-reject < 40)	GDPR Art. 22; autonomy	High	Remove hard auto-reject; require human review before any rejection; provide intervention/contest rights	Deployer
`R3`	Opaque scoring candidates cannot contest	Transparency; explainability; Art. 13–15	High	Per-decision explanations; candidate notice that AI is used; documented appeal route	Deployer + Provider
`R4`	Purpose-creep reusing hiring records as training data	GDPR purpose limitation; lawful basis	Medium	Compatibility test; document lawful basis; DPIA; consider anonymisation/aggregation for training	DPO
`R5`	Automation bias recruiters rubber-stamp the rank	AI Act human oversight, Art. 14	Medium	Oversight by design: blind first read, score as one factor, override training, log overrides	Deployer
`R6`	Diffuse accountability across vendor/firm/recruiter	Accountability; AI Act Arts. 16, 26	Medium	Contractual duty split; named accountable owner; full decision logging (Art. 12)	Provider + Deployer
`R7`	Over-retention of scores and rejected CVs	GDPR storage limitation	Low	Defined short retention; auto-deletion; minimise stored features	DPO
`R8`	Security breach of CV / score data	Integrity & confidentiality; AI Act Art. 15	Low	Access control, encryption, breach-notification procedure (72h)	Provider

Reading the register. The three High-severity risks (R1–R3) are blockers: each one alone makes deployment unlawful and unethical. R4–R6 are conditions to satisfy before launch; R7–R8 are standard controls. None is unfixable — which is why the recommendation below is "conditional go", not "no".

6 · Policy memo

One-page recommendation

The deliverable a decision-maker actually reads: a verdict, the reasons, and the conditions.

Policy Memorandum — Deployment of "TalentRank" AI CV-Screening System

To: Chief Operating Officer & Head of People From: AI Governance / Data Protection Office Re: Algorithmic Impact Assessment — deployment decision Classification: EU AI Act — HIGH-RISK · GDPR — high-risk processing (DPIA required)

Recommendation

CONDITIONAL GO. Do not deploy in the current design. Deployment is permissible only after the three high-severity blockers (R1–R3) are remediated and the conditions below are met and documented.

Why

TalentRank delivers genuine efficiency, but as designed it is unlawful (the hard auto-reject breaches GDPR Art. 22; no DPIA exists; high-risk AI Act obligations are unmet) and unethical (it launders historical bias and denies candidates transparency and recourse). Both the consequentialist and the deontological analysis reach the same conclusion: the system is acceptable only with fairness safeguards, transparency and meaningful human control.

Conditions for deployment

Remove the automated auto-reject. No candidate is rejected solely by the score; a recruiter reviews before any rejection (closes R2, satisfies Art. 22).
Run and pass a bias audit. Disparate-impact testing across protected groups, proxy-feature removal, and corrected retraining; repeat on a schedule (closes R1).
Make it explainable and contestable. Per-decision reasons, candidate notice that AI is used, and a documented appeal channel (closes R3).
Complete a DPIA and a Fundamental Rights Impact Assessment before go-live (GDPR Art. 35; AI Act Art. 27).
Engineer human oversight against automation bias, and allocate accountability contractually between vendor and firm with full decision logging (Arts. 12, 14, 26).
Minimise & time-limit data: drop low-signal/high-risk features, set short retention, secure storage (R4, R7, R8).

If the conditions are not met

Then the recommendation is NO-GO: keep recruiters making the decision with the tool used, at most, as a non-binding sorting aid — never as an automated gate. Reassess once the controls are in place.

7 · Mapping to learning outcomes

What this exercise demonstrates

How the worked example evidences each of the course's five learning objectives. ↳ see objectives

LO1 — Identify & comprehend the ethical, social and regulatory issues: the bias, accountability and recourse problems are named explicitly in §4. ↳ S1–S2
LO2 — Analyse, reason & debate a key dilemma (privacy, equality, accountability) through two opposing ethical lenses in §4. ↳ S2, S6
LO3 — Become familiar with the regulations: the EU AI Act high-risk regime and the GDPR principles & Art. 22 are applied in §3. ↳ S3–S5
LO4 — Apply principles to a real situation and bake them into design from the first step: the risk register and memo turn law into engineering conditions (§5–§6). ↳ S5, S13
LO5 — Carry out an expert report / opinion: the whole document is an AIA with a reasoned, defensible recommendation. ↳ Individual work · 40%

8 · References

Instruments & sources

The legal instruments and course texts this assessment is grounded in.

Regulation (EU) 2024/1689 — Artificial Intelligence Act. Risk-based classification; Annex III high-risk uses (employment); provider & deployer obligations (Arts. 9–15, 26–27, 43). Sessions 3–4
Regulation (EU) 2016/679 — GDPR. Art. 5 principles; Arts. 13–15 transparency; Art. 22 automated decision-making; Art. 35 DPIA. Session 5
EU Charter of Fundamental Rights. Non-discrimination (Art. 21), data protection (Art. 8), human dignity (Art. 1). Session 2
OECD AI Principles (2019) & UNESCO Recommendation on the Ethics of AI (2021). Human-centred, fairness, transparency, accountability. Session 2
Council of Europe — Unboxing AI: 10 steps to protect Human Rights. Practical human-rights checklist applied to the mitigations. Sessions 6–7
Stephanie Hare, Technology Is Not Neutral. Design choices embed values — the framing for treating the auto-reject default as an ethical choice. Sessions 1–2
Carissa Véliz, Privacy Is Power. Personal data as power; the candidate-vs-firm asymmetry in §2. Session 5

Note on scope & AI use. "TalentRank" is a hypothetical system built for teaching. Article numbers reflect the EU AI Act and GDPR as a study aid and should be checked against the consolidated texts for any real assessment — per the course's own AI policy: fact-check, cite multiple sources, and keep your own voice.