THE VR SCHOOL
  • Schools & Districts
    • License the academy
    • Free program for your students
    • Request a demo
  • Catalog
    • AI & Data
    • Cybersecurity & Web
    • Programming
    • Health Science
    • All Guilds
    • All courses
  • Families
  • Accreditation
  • Log In
  • Request a demo
  • Home
  • Courses

RLP AI Design A — Machine Learning, Media and the Human Body

Curriculum

  • 4 Sections
  • 20 Lessons
  • Lifetime
Expand all sectionsCollapse all sections
  • Unit 1: How Machines Learn: Making AI Visible
    5
    • 1.1
      Explaining the Invisible: Designers Who Made Data Visible
      50 mins
    • 1.2
      Sorting by Features: Training a Classifier by Hand
      100 mins
    • 1.3
      Critiquing AI Explainers: Accuracy, Clarity and Hype
      50 mins
    • 1.4
      Designing the Explainer: Hierarchy, Sequence and Storyboard
      100 mins
    • 1.5
      Portfolio Task — How a Machine Learns: Explainer and Storyboard
      200 mins
  • Unit 2: Human-Centered AI: History, Ethics and Authorship
    5
    • 2.1
      Drawing Time: Timeline Design and the History of AI
      50 mins
    • 2.2
      Verified Event Cards: Building a Physical Timeline Storyboard
      100 mins
    • 2.3
      Who Is Responsible? Bias and Human-Centered AI in Medicine
      100 mins
    • 2.4
      Authorship and AI Imagery: Making the Presentation Yours
      150 mins
    • 2.5
      Portfolio Task — The Evolution of a Healthcare AI Innovation
      200 mins
  • Unit 3: Drawing the Body: The Eleven Organ Systems
    5
    • 3.1
      Art and Anatomy: Five Centuries of Seeing Inside the Body
      50 mins
    • 3.2
      Sculpting a System: Modeling Structure and Function
      100 mins
    • 3.3
      Clinician and Patient: Role-Play and the Critique of Health Communication
      100 mins
    • 3.4
      Illustrating a Condition: Panels for One Body System
      150 mins
    • 3.5
      Portfolio Task — The Body Atlas Exhibition
      200 mins
  • Unit 4: Signals and Diagnosis: Visualizing Vital Signs and Medical Data
    5
    • 4.1
      Data Portraits: Du Bois, Einthoven and the Art of the Signal
      50 mins
    • 4.2
      Pulse to Pixels: Coding a Heart-Rate Visualization
      100 mins
    • 4.3
      Who Sees Your Data? Medical Records, Genetics and HIPAA
      50 mins
    • 4.4
      Imaging the Invisible: Designing a Patient Brochure
      150 mins
    • 4.5
      Portfolio Task — Reading the Body’s Signals
      200 mins

Who Is Responsible? Bias and Human-Centered AI in Medicine

Unit 2  ·  Critique & Discourse Lab  ·  Lesson 8 of 20

Who Is Responsible? Bias and Human-Centered AI in Medicine

MA:Cn11A.IIRLP-AIDA
Hands, a pulse oximeter, and a sorting machine show how biased data can change medical care.
By the end of this lesson I can…

explain how bias enters health algorithms and devices using documented cases, and argue a position on the responsibilities of AI creators while fairly stating the positions I do not hold.

Instruction

Human-centered AI means designing systems around the needs, rights and wellbeing of the people they affect, and keeping people meaningfully in control. It sounds obvious. The cases below show how hard it is.

Case 1: a cost label standing in for need. In 2019 Ziad Obermeyer and colleagues published a study in Science of a widely used commercial algorithm that helped U.S. health systems decide which patients to enroll in extra care programs. The algorithm predicted future health-care costs and used them as a stand-in for health needs. Because less money had historically been spent on Black patients with the same level of illness, the algorithm scored Black patients as healthier than equally sick white patients. The researchers reported that changing what the algorithm predicted greatly reduced the bias, and they worked with the developer on that change. The lesson: a model can be accurate at predicting its label and still be unfair, if the label is a poor stand-in for what we care about. No one had to intend harm for harm to happen.

Case 2: a device that measures some bodies less well. Pulse oximeters clip onto a finger and estimate blood oxygen by shining light through the skin. In 2021 the FDA issued a safety communication that pulse oximeters can be less accurate in people with darker skin pigmentation, and it has since worked on updated testing expectations. This is not AI, but it matters for AI: if device readings used to train or feed a model are less accurate for some patients, the model inherits the error. Bias can live in the data before any learning begins.

Where bias enters. Use these checkpoints when you examine any system: who is in the training data (and who is missing); what the label really measures; how the inputs were collected; who tested the system and on whom; and who can question or override its output.

Positions in the debate. Reasonable people disagree about who is responsible. Some argue developers bear primary responsibility, because they choose data and labels. Some argue hospitals and clinicians who deploy a tool must validate it on their own patients. Others emphasize regulators, who set testing requirements, or say patients and communities must be involved in design from the start. Some argue that imperfect AI may still reduce harm compared with the status quo, since human judgment is also biased. Each position has evidence behind it.

How this lab runs. This is a structured written debate, and you argue every side. Write one short turn each for three stakeholders: developer, hospital, clinician, patient advocate or regulator. Before you write each new turn, restate the previous turn in one sentence its holder would accept as fair. Evidence must come from the cases or from a source you name. If a family member or other adult is willing, they may read one turn aloud from your notes and you answer it aloud; this is optional. You will finish with a 250-word position statement that names the strongest argument against your view and answers it.

Why an arts course? The images and stories we make about AI shape what the public expects of it. Media artists who understand bias can make work that shows it clearly, which is often the first step in fixing it.

📖 Read for this lesson

The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.

  • Two Documented Bias Cases and Three Kinds of Bias: Obermeyer 2019, the FDA on Pulse Oximeters, and NISTZ. Obermeyer et al., Science 366(6464):447-453 (Oct. 25, 2019), doi 10.1126/science.aax2342, PubMed 31649194 (abstract paraphrased, one clause quoted); FDA, Pulse Oximeter Accuracy and Limitations: FDA Safety Communication (Feb. 19, 2021) and FDA press announcement of Jan. 6, 2025; NIST AI 100-1 (Jan. 2023) sec. 3.7; retrieved Sept. 26, 2026 · about 6 min

    The published numbers and wording for both cases, NIST’s three kinds of bias, four positions on responsibility with their best arguments, the steps of the structured written debate in which you argue each side, and a model paragraph, so your statement can cite evidence from the cases.

Vocabulary in context

  • human-centered AI — Designing AI around the needs, rights and wellbeing of the people it affects, with people kept meaningfully in control.
  • algorithmic bias — Systematic error in a model’s outputs that disadvantages some groups, often inherited from data, labels or collection methods.
  • proxy label — A measurable stand-in for what we really care about, such as cost standing in for health need.
  • stakeholder — A person or group affected by, or with influence over, a system.

Formative check

Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.

+50 XP

In the 2019 Science study, what was the main source of the bias?

The label (cost) was a poor proxy for need.
Myth or Fact?

A model can predict its training label accurately and still produce unfair results.

If the label is a biased stand-in for what matters, accurate prediction of it reproduces the bias.
Make a prediction

An AI model is trained on blood-oxygen readings from pulse oximeters that read less accurately for patients with darker skin. What will the model most likely do?

It will most likely inherit the error. Bias can be in the data before any learning happens, which is why the FDA’s 2021 safety communication matters for AI too.

Write your 250-word position statement on who should bear the most responsibility for bias in a medical AI tool. Restate the strongest opposing position fairly, then answer it with evidence from the cases.

0 words
Quick self-check

How confident are you that you can explain how bias enters health AI using real cases, and argue a position while stating others fairly?

Not yetVery confident

Practice

Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.

Printable version: this unit’s practice workbook (PDF)

Fluency

Build speed and accuracy with the core skill.

  1. Define human-centered AI, proxy label and stakeholder.
    Check your answer
    Answer: Human-centered AI is designing AI around the needs, rights and wellbeing of the people it affects, with people kept meaningfully in control. A proxy label is a measurable stand-in for what we really care about, such as cost standing in for health need. A stakeholder is a person or group affected by, or with influence over, a system.
    The Obermeyer case is the lesson’s example of a proxy label.
  2. List the five checkpoints the lesson gives for where bias enters a system.
    Check your answer
    Answer: Who is in the training data (and who is missing); what the label really measures; how the inputs were collected; who tested the system and on whom; who can question or override its output.
    Use these checkpoints on any system you examine.
  3. What did the FDA’s 2021 safety communication say about pulse oximeters, and why does it matter for AI even though an oximeter is not AI?
    Check your answer
    Answer: Pulse oximeters can be less accurate in people with darker skin pigmentation. If those readings train or feed a model, the model inherits the error: bias can live in the data before any learning begins.
    The device estimates blood oxygen by shining light through the skin; its error carries into anything built on its readings.

Application

Use the skill in context. Show your reasoning.

  1. Explain the 2019 Obermeyer case in 3 sentences: what the algorithm predicted, why that produced bias, and what reduced the bias.
    Check your answer
    Answer: The algorithm predicted future health-care costs and used them as a stand-in for health needs when choosing patients for extra care programs. Because less money had historically been spent on Black patients with the same level of illness, it scored Black patients as healthier than equally sick white patients. Changing what the algorithm predicted greatly reduced the bias.
    The lesson’s point: a model can be accurate at predicting its label and still be unfair if the label is a poor stand-in. No one had to intend harm.
  2. Write one sentence restating each of these positions so that someone who holds it would accept it as fair: (a) developers bear primary responsibility; (b) imperfect AI may still reduce harm.
    Check your answer
    Answer: (a) ‘Developers choose the data and the labels, so they are best placed to prevent bias and should answer for it.’ (b) ‘Human judgment is also biased, so an imperfect tool can still do less harm than the status quo, and rejecting it has costs too.’
    The debate rule is to restate the previous turn fairly before answering. A strong restatement includes the position’s best reason.
  3. A hospital buys a skin-image classifier tested only on patients at one clinic. Using two of the five bias checkpoints, name two questions the hospital should ask before using it.
    Check your answer
    Answer: Who is in the training data, and who is missing? Who tested the system, and on whom? (Other valid checkpoints: how inputs were collected; who can question or override the output.)
    The position that hospitals must validate a tool on their own patients follows from these checkpoints.
  4. Your 250-word position statement argues that regulators bear the most responsibility. Write the two sentences in which you name the strongest argument against your view and answer it.
    Check your answer
    Answer: Model: ‘The strongest objection is that developers choose the data and labels, so only they can prevent bias at its source, as the Obermeyer case showed. But that case was fixed only after outside researchers found it, which is why regulators must set testing requirements, as the FDA did after its 2021 pulse oximeter communication, rather than trusting each developer to catch its own errors.’
    A strong answer states the opposing view with its evidence, then answers with evidence from the cases. Any position is acceptable if it is argued this way.

Review

Keep earlier skills sharp.

  1. Review 1.2: In the cell-card lab, a model trained only on red sickle cards might learn color. How is that like the pulse oximeter problem?
    Check your answer
    Answer: In both cases the problem is in the data before learning: the model learns whatever the data carries, whether a color habit or a measurement error, and reproduces it.
    Training data carries the habits of whoever made it, and the limits of the devices that collected it.

Missouri Learning Standards — Media Arts addressed: MA:Cn11A.II, MA:Re8A.II, SL.11-12.1, SL.11-12.3, RST.11-12.8

Instructional focus: Evidence-based argument about the social context of technology and media

Reading

Two Documented Bias Cases and Three Kinds of Bias: Obermeyer 2019, the FDA on Pulse Oximeters, and NIST

Z. Obermeyer et al., Science 366(6464):447-453 (Oct. 25, 2019), doi 10.1126/science.aax2342, PubMed 31649194 (abstract paraphrased, one clause quoted); FDA, Pulse Oximeter Accuracy and Limitations: FDA Safety Communication (Feb. 19, 2021) and FDA press announcement of Jan. 6, 2025; NIST AI 100-1 (Jan. 2023) sec. 3.7; retrieved Sept. 26, 2026 · Public domain (FDA and NIST works), with SLS original summaries, positions and model paragraph; brief quotation from the Science abstract

Why this reading. Your 250-word position statement asks you to answer an opposing position “with evidence from the cases,” and to restate a position you do not hold. The lesson describes the cases in a paragraph each; this reading gives the published wording and the numbers for them, NIST’s categories of bias, four positions on who is responsible (each with its best argument), and a model paragraph. What to look for: in each case, what was measured, what was meant to be measured, and who could have noticed the gap.

Case 1. A commercial algorithm that used cost to stand in for need

Source. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 366 (6464), 25 October 2019, pp. 447–453, doi 10.1126/science.aax2342 (PubMed 31649194; abstract read 26 September 2026). The paper itself is under a publisher license; I paraphrase its abstract and quote one clause.

What the authors report. Health systems use commercial prediction algorithms to identify patients with complex health needs so they can be offered extra help. They studied a widely used one that affects millions of patients. At a given risk score, Black patients were considerably sicker than white patients, as shown by signs of uncontrolled illness. If the disparity were remedied, the share of Black patients receiving additional help would rise from 17.7% to 46.5%. The cause they identify is that the algorithm predicts health care costs rather than illness, and because access to care is unequal, less money is spent caring for Black patients than white patients. Cost looked like a good stand-in for health by some measures of predictive accuracy, yet “large racial biases arise.” They suggest that the choice of a convenient proxy for the truth can be an important source of algorithmic bias.

Worked with the checkpoints. Who is in the training data? Patients whose costs were recorded, which reflects who got care. What does the label measure? Money spent, not need. How were the inputs collected? From billing and claims history. Who decided what “success” means? Whoever chose cost as the target. Notice that nothing in the description requires anyone to intend harm.

Case 2. A device that measures some bodies less accurately

Source. U.S. Food and Drug Administration, “Pulse Oximeter Accuracy and Limitations: FDA Safety Communication,” 19 February 2021 (the page itself was not retrievable on 26 September 2026; the date and content are from the FDA’s own later pages that cite it), and FDA, “FDA Proposes Updated Recommendations to Help Improve Performance of Pulse Oximeters Across Skin Tones,” 6 January 2025. Public domain (U.S. government works).

What the FDA says. The 2021 communication said pulse oximeters have limitations and a risk of inaccuracy, and that published reports suggest they may be less accurate in people with darker skin pigmentation. In January 2025 the agency wrote that “current scientific evidence suggests there are some accuracy differences in pulse oximeter performance between individuals with lighter and darker skin pigmentation,” and proposed that manufacturers gather clinical data across the full range of skin pigmentations, enroll more participants, use both a subjective skin-tone scale (the Monk Skin Tone Scale) and an objective measurement, and include prominent labeling. Note the phrase “draft guidance”: it is a proposal, not a rule.

Why it belongs in an AI lab. A pulse oximeter is not AI. But readings from devices like it are what a model would be trained on or fed. If the input is less accurate for some patients, the model inherits the error and may make it worse without anyone noticing.

Three kinds of bias, from NIST

The U.S. National Institute of Standards and Technology (AI Risk Management Framework 1.0, NIST AI 100-1, January 2023, section 3.7; public domain) says bias is broader than demographic balance and names three major categories, each of which “can occur in the absence of prejudice, partiality, or discriminatory intent”: systemic bias, which can be present in datasets, organizational practices and the broader society that uses AI; computational and statistical bias, which often stems from systematic errors due to non-representative samples; and human-cognitive bias, which relates to how people perceive AI output or think about what a system is for. It adds that systems whose harmful biases are mitigated are “not necessarily fair.” Sort each case: the cost proxy is mostly systemic (unequal spending) reaching the model through a label choice; the oximeter is a measurement problem that produces statistical bias downstream.

Four positions on responsibility (SLS original summaries)

PositionIts best argumentIts weakest point
The developer carries the most responsibilityThe developer chooses the target, the data and the tests, and is the only party who can inspect them before release.Developers often cannot see how a health system will use a score, or what the data leave out.
The health system that deploys itIt decides which patients the score is used on and whether anyone checks outcomes by group after it starts.Systems buy tools because they lack staff to build or audit them.
The regulator or standards bodyOnly a common rule (such as the FDA’s proposed testing across skin tones) makes fair testing the price of entry for everyone.Rules follow harm; they are slow, and many tools fall outside them.
Shared, with named dutiesEach party controls a different link (data, use, audit, rule); saying “everyone” without duties spreads blame until no one has it.“Shared” can become an excuse unless each duty is named and someone checks.

The structured written debate, in steps

You argue every side yourself, in writing. Choose three of the stakeholders (developer, deploying health system, clinician, patient advocate, regulator) and write one turn for each, in order.

  1. Each turn names the case and the checkpoint it uses (who is in the data / what the label measures / how inputs were collected / who defines success).
  2. Before writing the next turn, restate the previous turn in one sentence its holder would accept as fair. If you cannot, the previous turn is not yet clear; rewrite it.
  3. Every claim about a case must point to a fact in this reading (a number, a quotation or a category).
  4. Optional: a family member or other adult reads one turn aloud from your notes, and you answer it aloud before writing your reply.

A model paragraph (SLS original; a constructed exemplar, not any person’s view)

“Some argue that the developer bears most of the responsibility, because the developer picks the target and can inspect the data before anyone else can. That is strong for the cost-proxy case: the authors found the label itself, cost, was the problem, and a label is a design choice. It is weaker for the pulse oximeter, where the FDA’s 2025 proposal shows a regulator, not the manufacturers, asking for testing across skin tones. I hold that responsibility should be assigned by which party controls the link that failed. In Case 1 that is the developer and, after release, the system that chose to rely on the score without checking results by group. I therefore rank the developer first for what is built and the deploying system first for what is checked.”

Notice the moves: the opposing view is stated in its strongest form, evidence from both cases is used, and the answer is a position with a rule, not a slogan.

Sources: Obermeyer et al. (2019), Dissecting racial bias in an algorithm used to manage the health of populations, Science 366 (doi.org); FDA, FDA Proposes Updated Recommendations to Help Improve Performance of Pulse Oximeters Across Skin Tones (Jan. 6, 2025) (fda.gov); FDA, Pulse Oximeters (fda.gov); NIST AI Risk Management Framework 1.0 (NIST AI 100-1) (doi.org)

Up NextComplete & Continue →

Leave a Reply Cancel reply

Verified Event Cards: Building a Physical Timeline Storyboard
Prev
Authorship and AI Imagery: Making the Presentation Yours
Next

© 2026 The VR School | Career content by Wiley | Powered by Sejal Learning Systems.

Join free · Accreditation · Privacy Policy · Terms of Service

Modal title

Main Content