
explain how bias enters health algorithms and devices using documented cases, and argue a position on the responsibilities of AI creators while fairly stating the positions I do not hold.
Instruction
Human-centered AI means designing systems around the needs, rights and wellbeing of the people they affect, and keeping people meaningfully in control. It sounds obvious. The cases below show how hard it is.
Case 1: a cost label standing in for need. In 2019 Ziad Obermeyer and colleagues published a study in Science of a widely used commercial algorithm that helped U.S. health systems decide which patients to enroll in extra care programs. The algorithm predicted future health-care costs and used them as a stand-in for health needs. Because less money had historically been spent on Black patients with the same level of illness, the algorithm scored Black patients as healthier than equally sick white patients. The researchers reported that changing what the algorithm predicted greatly reduced the bias, and they worked with the developer on that change. The lesson: a model can be accurate at predicting its label and still be unfair, if the label is a poor stand-in for what we care about. No one had to intend harm for harm to happen.
Case 2: a device that measures some bodies less well. Pulse oximeters clip onto a finger and estimate blood oxygen by shining light through the skin. In 2021 the FDA issued a safety communication that pulse oximeters can be less accurate in people with darker skin pigmentation, and it has since worked on updated testing expectations. This is not AI, but it matters for AI: if device readings used to train or feed a model are less accurate for some patients, the model inherits the error. Bias can live in the data before any learning begins.
Where bias enters. Use these checkpoints when you examine any system: who is in the training data (and who is missing); what the label really measures; how the inputs were collected; who tested the system and on whom; and who can question or override its output.
Positions in the debate. Reasonable people disagree about who is responsible. Some argue developers bear primary responsibility, because they choose data and labels. Some argue hospitals and clinicians who deploy a tool must validate it on their own patients. Others emphasize regulators, who set testing requirements, or say patients and communities must be involved in design from the start. Some argue that imperfect AI may still reduce harm compared with the status quo, since human judgment is also biased. Each position has evidence behind it.
How this lab runs. This is a structured written debate, and you argue every side. Write one short turn each for three stakeholders: developer, hospital, clinician, patient advocate or regulator. Before you write each new turn, restate the previous turn in one sentence its holder would accept as fair. Evidence must come from the cases or from a source you name. If a family member or other adult is willing, they may read one turn aloud from your notes and you answer it aloud; this is optional. You will finish with a 250-word position statement that names the strongest argument against your view and answers it.
Why an arts course? The images and stories we make about AI shape what the public expects of it. Media artists who understand bias can make work that shows it clearly, which is often the first step in fixing it.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
In the 2019 Science study, what was the main source of the bias?
A model can predict its training label accurately and still produce unfair results.
An AI model is trained on blood-oxygen readings from pulse oximeters that read less accurately for patients with darker skin. What will the model most likely do?
Write your 250-word position statement on who should bear the most responsibility for bias in a medical AI tool. Restate the strongest opposing position fairly, then answer it with evidence from the cases.
How confident are you that you can explain how bias enters health AI using real cases, and argue a position while stating others fairly?
Practice
Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.
Printable version: this unit’s practice workbook (PDF)
Fluency
Build speed and accuracy with the core skill.
- Define human-centered AI, proxy label and stakeholder.
Check your answer
Answer: Human-centered AI is designing AI around the needs, rights and wellbeing of the people it affects, with people kept meaningfully in control. A proxy label is a measurable stand-in for what we really care about, such as cost standing in for health need. A stakeholder is a person or group affected by, or with influence over, a system.The Obermeyer case is the lesson’s example of a proxy label. - List the five checkpoints the lesson gives for where bias enters a system.
Check your answer
Answer: Who is in the training data (and who is missing); what the label really measures; how the inputs were collected; who tested the system and on whom; who can question or override its output.Use these checkpoints on any system you examine. - What did the FDA’s 2021 safety communication say about pulse oximeters, and why does it matter for AI even though an oximeter is not AI?
Check your answer
Answer: Pulse oximeters can be less accurate in people with darker skin pigmentation. If those readings train or feed a model, the model inherits the error: bias can live in the data before any learning begins.The device estimates blood oxygen by shining light through the skin; its error carries into anything built on its readings.
Application
Use the skill in context. Show your reasoning.
- Explain the 2019 Obermeyer case in 3 sentences: what the algorithm predicted, why that produced bias, and what reduced the bias.
Check your answer
Answer: The algorithm predicted future health-care costs and used them as a stand-in for health needs when choosing patients for extra care programs. Because less money had historically been spent on Black patients with the same level of illness, it scored Black patients as healthier than equally sick white patients. Changing what the algorithm predicted greatly reduced the bias.The lesson’s point: a model can be accurate at predicting its label and still be unfair if the label is a poor stand-in. No one had to intend harm. - Write one sentence restating each of these positions so that someone who holds it would accept it as fair: (a) developers bear primary responsibility; (b) imperfect AI may still reduce harm.
Check your answer
Answer: (a) ‘Developers choose the data and the labels, so they are best placed to prevent bias and should answer for it.’ (b) ‘Human judgment is also biased, so an imperfect tool can still do less harm than the status quo, and rejecting it has costs too.’The debate rule is to restate the previous turn fairly before answering. A strong restatement includes the position’s best reason. - A hospital buys a skin-image classifier tested only on patients at one clinic. Using two of the five bias checkpoints, name two questions the hospital should ask before using it.
Check your answer
Answer: Who is in the training data, and who is missing? Who tested the system, and on whom? (Other valid checkpoints: how inputs were collected; who can question or override the output.)The position that hospitals must validate a tool on their own patients follows from these checkpoints. - Your 250-word position statement argues that regulators bear the most responsibility. Write the two sentences in which you name the strongest argument against your view and answer it.
Check your answer
Answer: Model: ‘The strongest objection is that developers choose the data and labels, so only they can prevent bias at its source, as the Obermeyer case showed. But that case was fixed only after outside researchers found it, which is why regulators must set testing requirements, as the FDA did after its 2021 pulse oximeter communication, rather than trusting each developer to catch its own errors.’A strong answer states the opposing view with its evidence, then answers with evidence from the cases. Any position is acceptable if it is argued this way.
Review
Keep earlier skills sharp.
- Review 1.2: In the cell-card lab, a model trained only on red sickle cards might learn color. How is that like the pulse oximeter problem?
Check your answer
Answer: In both cases the problem is in the data before learning: the model learns whatever the data carries, whether a color habit or a measurement error, and reproduces it.Training data carries the habits of whoever made it, and the limits of the devices that collected it.
Two Documented Bias Cases and Three Kinds of Bias: Obermeyer 2019, the FDA on Pulse Oximeters, and NIST
Z. Obermeyer et al., Science 366(6464):447-453 (Oct. 25, 2019), doi 10.1126/science.aax2342, PubMed 31649194 (abstract paraphrased, one clause quoted); FDA, Pulse Oximeter Accuracy and Limitations: FDA Safety Communication (Feb. 19, 2021) and FDA press announcement of Jan. 6, 2025; NIST AI 100-1 (Jan. 2023) sec. 3.7; retrieved Sept. 26, 2026 · Public domain (FDA and NIST works), with SLS original summaries, positions and model paragraph; brief quotation from the Science abstract
Why this reading. Your 250-word position statement asks you to answer an opposing position “with evidence from the cases,” and to restate a position you do not hold. The lesson describes the cases in a paragraph each; this reading gives the published wording and the numbers for them, NIST’s categories of bias, four positions on who is responsible (each with its best argument), and a model paragraph. What to look for: in each case, what was measured, what was meant to be measured, and who could have noticed the gap.
Case 1. A commercial algorithm that used cost to stand in for need
Source. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 366 (6464), 25 October 2019, pp. 447–453, doi 10.1126/science.aax2342 (PubMed 31649194; abstract read 26 September 2026). The paper itself is under a publisher license; I paraphrase its abstract and quote one clause.
What the authors report. Health systems use commercial prediction algorithms to identify patients with complex health needs so they can be offered extra help. They studied a widely used one that affects millions of patients. At a given risk score, Black patients were considerably sicker than white patients, as shown by signs of uncontrolled illness. If the disparity were remedied, the share of Black patients receiving additional help would rise from 17.7% to 46.5%. The cause they identify is that the algorithm predicts health care costs rather than illness, and because access to care is unequal, less money is spent caring for Black patients than white patients. Cost looked like a good stand-in for health by some measures of predictive accuracy, yet “large racial biases arise.” They suggest that the choice of a convenient proxy for the truth can be an important source of algorithmic bias.
Worked with the checkpoints. Who is in the training data? Patients whose costs were recorded, which reflects who got care. What does the label measure? Money spent, not need. How were the inputs collected? From billing and claims history. Who decided what “success” means? Whoever chose cost as the target. Notice that nothing in the description requires anyone to intend harm.
Case 2. A device that measures some bodies less accurately
Source. U.S. Food and Drug Administration, “Pulse Oximeter Accuracy and Limitations: FDA Safety Communication,” 19 February 2021 (the page itself was not retrievable on 26 September 2026; the date and content are from the FDA’s own later pages that cite it), and FDA, “FDA Proposes Updated Recommendations to Help Improve Performance of Pulse Oximeters Across Skin Tones,” 6 January 2025. Public domain (U.S. government works).
What the FDA says. The 2021 communication said pulse oximeters have limitations and a risk of inaccuracy, and that published reports suggest they may be less accurate in people with darker skin pigmentation. In January 2025 the agency wrote that “current scientific evidence suggests there are some accuracy differences in pulse oximeter performance between individuals with lighter and darker skin pigmentation,” and proposed that manufacturers gather clinical data across the full range of skin pigmentations, enroll more participants, use both a subjective skin-tone scale (the Monk Skin Tone Scale) and an objective measurement, and include prominent labeling. Note the phrase “draft guidance”: it is a proposal, not a rule.
Why it belongs in an AI lab. A pulse oximeter is not AI. But readings from devices like it are what a model would be trained on or fed. If the input is less accurate for some patients, the model inherits the error and may make it worse without anyone noticing.
Three kinds of bias, from NIST
The U.S. National Institute of Standards and Technology (AI Risk Management Framework 1.0, NIST AI 100-1, January 2023, section 3.7; public domain) says bias is broader than demographic balance and names three major categories, each of which “can occur in the absence of prejudice, partiality, or discriminatory intent”: systemic bias, which can be present in datasets, organizational practices and the broader society that uses AI; computational and statistical bias, which often stems from systematic errors due to non-representative samples; and human-cognitive bias, which relates to how people perceive AI output or think about what a system is for. It adds that systems whose harmful biases are mitigated are “not necessarily fair.” Sort each case: the cost proxy is mostly systemic (unequal spending) reaching the model through a label choice; the oximeter is a measurement problem that produces statistical bias downstream.
Four positions on responsibility (SLS original summaries)
| Position | Its best argument | Its weakest point |
|---|---|---|
| The developer carries the most responsibility | The developer chooses the target, the data and the tests, and is the only party who can inspect them before release. | Developers often cannot see how a health system will use a score, or what the data leave out. |
| The health system that deploys it | It decides which patients the score is used on and whether anyone checks outcomes by group after it starts. | Systems buy tools because they lack staff to build or audit them. |
| The regulator or standards body | Only a common rule (such as the FDA’s proposed testing across skin tones) makes fair testing the price of entry for everyone. | Rules follow harm; they are slow, and many tools fall outside them. |
| Shared, with named duties | Each party controls a different link (data, use, audit, rule); saying “everyone” without duties spreads blame until no one has it. | “Shared” can become an excuse unless each duty is named and someone checks. |
The structured written debate, in steps
You argue every side yourself, in writing. Choose three of the stakeholders (developer, deploying health system, clinician, patient advocate, regulator) and write one turn for each, in order.
- Each turn names the case and the checkpoint it uses (who is in the data / what the label measures / how inputs were collected / who defines success).
- Before writing the next turn, restate the previous turn in one sentence its holder would accept as fair. If you cannot, the previous turn is not yet clear; rewrite it.
- Every claim about a case must point to a fact in this reading (a number, a quotation or a category).
- Optional: a family member or other adult reads one turn aloud from your notes, and you answer it aloud before writing your reply.
A model paragraph (SLS original; a constructed exemplar, not any person’s view)
“Some argue that the developer bears most of the responsibility, because the developer picks the target and can inspect the data before anyone else can. That is strong for the cost-proxy case: the authors found the label itself, cost, was the problem, and a label is a design choice. It is weaker for the pulse oximeter, where the FDA’s 2025 proposal shows a regulator, not the manufacturers, asking for testing across skin tones. I hold that responsibility should be assigned by which party controls the link that failed. In Case 1 that is the developer and, after release, the system that chose to rely on the score without checking results by group. I therefore rank the developer first for what is built and the deploying system first for what is checked.”
Notice the moves: the opposing view is stated in its strongest form, evidence from both cases is used, and the answer is a position with a rule, not a slogan.
Sources: Obermeyer et al. (2019), Dissecting racial bias in an algorithm used to manage the health of populations, Science 366 (doi.org); FDA, FDA Proposes Updated Recommendations to Help Improve Performance of Pulse Oximeters Across Skin Tones (Jan. 6, 2025) (fda.gov); FDA, Pulse Oximeters (fda.gov); NIST AI Risk Management Framework 1.0 (NIST AI 100-1) (doi.org)