
analyze how persuasive health media shapes public expectations, and compare the evidence behind a documented AI success with the evidence behind a documented AI failure.
Instruction
Every health technology reaches the public through media, and media is designed to persuade. This unit asks you to tell true stories about AI in medicine, so start by studying how persuasion works and how to test it against evidence.
WPA health posters (1936-1943). During the Great Depression, artists employed by the Works Progress Administration (later the Work Projects Administration) designed thousands of silkscreen posters, many promoting public health: tuberculosis testing, syphilis treatment, nutrition, clean milk and safety at work. The Library of Congress holds a large collection. Their design is direct: flat areas of bold color, one image, a short command in large type. They were made to change behavior quickly, often for people with limited schooling. Look at one and ask: what does it want you to feel, and what does it want you to do?
Today’s version. AI health products are presented with a different visual language: clean white interfaces, glowing overlays on scans, confident percentages. Like the posters, these are designed to create trust. Unlike the posters, they often promote a product. Your job is to check whether the evidence supports the feeling.
A documented success, with a twist. Diabetic retinopathy is damage to the retina caused by diabetes and can lead to blindness if not caught. In 2016 Google researchers published in JAMA a deep-learning system that graded retinal photographs for referable diabetic retinopathy with high sensitivity and specificity compared with ophthalmologists. In 2018 DeepMind (also owned by Google’s parent company) and Moorfields Eye Hospital published in Nature Medicine a system that recommended referrals for many eye conditions from OCT scans. Also in 2018 the FDA authorized IDx-DR, from a different company, to give a screening result without a clinician interpreting the image. Then came the twist. In 2020 Google researchers published a human-centered study (Beede and colleagues) of their system in clinics in Thailand. Nurses found that many images taken in real conditions, with poor lighting, were rejected by the system as ungradable, and slow internet connections delayed results. The model worked; the deployment struggled. The lesson is that a laboratory result is not the same as a clinical success.
A documented failure. IBM’s Watson for Oncology was promoted in the 2010s as a tool to recommend cancer treatments. Published reporting, including internal IBM documents obtained by the health news outlet STAT in 2018, described recommendations that doctors judged unsafe or incorrect, and training that relied partly on hypothetical cases rather than real patient data. A separate project at the University of Texas MD Anderson Cancer Center was halted after a 2017 university audit. IBM sold its Watson Health data and analytics business in 2022. The gap between the marketing and the evidence became the story.
Reading evidence. For every case, separate three kinds of claim: what the developer says, what an independent study found, and what happened in real deployment. Note who did each study. A study by the developer’s own researchers is useful but not independent. Keep a two-column chart: claim and evidence. It will become the spine of your talk.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
What did the 2020 human-centered study of the retinopathy system in Thai clinics mainly show?
The IDx-DR system authorized by the FDA in 2018 and Google’s 2016 JAMA system were made by the same company.
Developer claim
Tap to revealWhat the maker says the system does
Independent study
Tap to revealTesting by researchers who did not build the system
Real deployment
Tap to revealWhat happened with real users, equipment and patients
WPA health posters
Tap to reveal1936-1943 silkscreens: bold color, one image, one command
Compare one WPA health poster with one current AI health product page. For each, name the feeling it creates and the action it asks for. Then state one claim on the product page and say what kind of evidence you would need to check it.
How confident are you that you can analyze persuasive health media and separate developer claims, independent evidence and deployment results in a case?
Practice
Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.
Printable version: this unit’s practice workbook (PDF)
Fluency
Build speed and accuracy with the core skill.
- Describe the design of the WPA health posters (1936-1943) in three features, and say what they were made to do.
Check your answer
Answer: Flat areas of bold color, one image, and a short command in large type. They were made to change behavior quickly, often for people with limited schooling, on topics such as tuberculosis testing, nutrition, clean milk and safety at work.Artists employed by the Works Progress Administration made thousands of silkscreen posters; the Library of Congress holds a large collection. - Define independent validation, deployment and persuasive design.
Check your answer
Answer: Independent validation: testing a model on new data by researchers who did not build it. Deployment: putting a system into real use, with real users, equipment and conditions. Persuasive design: design choices intended to change what a viewer feels, believes or does.All three are needed to tell a true story: the design creates a feeling; validation and deployment test whether it is deserved. - Name the three kinds of claim the lesson says to separate for every case.
Check your answer
Answer: What the developer says; what an independent study found; what happened in real deployment.Also note who did each study; a study by the developer’s own researchers is useful but not independent.
Application
Use the skill in context. Show your reasoning.
- Sort these retinopathy items into developer claim, independent study, or real deployment, and explain the twist: (a) Google’s 2016 JAMA paper on grading retinal photographs; (b) the 2020 Beede study of Google’s system in Thai clinics.
Check your answer
Answer: (a) A study by the developer’s own researchers (a developer claim, not independent). (b) Real deployment, also by Google researchers. The twist: in clinics, many images taken in poor lighting were rejected as ungradable and slow internet delayed results. The model worked; the deployment struggled.A laboratory result is not the same as a clinical success. - Build a two-column claim-and-evidence chart with three rows for IBM Watson for Oncology, using only facts from the lesson.
Check your answer
Answer: Claim: promoted in the 2010s as a tool to recommend cancer treatments. Evidence: internal IBM documents obtained by STAT in 2018 described recommendations doctors judged unsafe or incorrect. Claim: trained to give good recommendations. Evidence: training relied partly on hypothetical cases rather than real patient data. Claim: a partner project at MD Anderson. Evidence: halted after a 2017 university audit; IBM sold its Watson Health data and analytics business in 2022.The gap between the marketing and the evidence became the story. - Compare the visual language of a WPA health poster with the visual language the lesson describes for AI health products today. Name one similarity and one difference.
Check your answer
Answer: Similarity: both are designed to create trust and change what people do. Difference: posters use bold flat color, one image and a command, often to promote a public-health behavior; AI products use clean white interfaces, glowing overlays on scans and confident percentages, and often promote a product.Your job is to check whether the evidence supports the feeling. - An invented AI eye-screening product page says: ‘97% accurate. Trusted by clinics.’ Write two questions you would ask to check it, using the lesson’s three kinds of claim.
Check your answer
Answer: Model: ‘Who measured the 97%: the developer’s own researchers or an independent team, and on what data?’ ‘What happened in real deployment, for example how many images were rejected as ungradable in real clinic lighting?’A strong answer asks about independence and about deployment, not only about the number.
Review
Keep earlier skills sharp.
- Review (Semester A, Unit 2): The FDA authorized IDx-DR in 2018. What did the FDA describe as first about it, and is it the same system as Google’s 2016 JAMA system?
Check your answer
Answer: The first device authorized to give a screening decision for diabetic retinopathy without a clinician interpreting the image. No: it came from a different company.Keep cases distinct in your research.
Fight Tuberculosis, RetinaLite and Two Real Cases: A Poster, a Product Page and the Evidence Behind Them
Poster record: Library of Congress LCCN 98513584 (retrieved 2026-09-26); FDA news release 2018-04-11; Abramoff et al., npj Digit Med 2018 (CC BY 4.0); Beede et al., CHI 2020; Wong et al., JAMA Intern Med 2021 (abstract); RetinaLite page and chart are SLS original and illustrative · Public domain (Library of Congress record, FDA release); CC BY 4.0 (Abramoff et al. figures cited); facts from published studies cited with attribution; SLS original (constructed product page and chart)
Why this reading. The lesson asks you to compare a WPA health poster with an AI product page, then compare the evidence behind a documented success with the evidence behind a documented failure. Everything you need is here: the poster, written out so you can analyze it without the picture; a product page, written out the same way; and the primary evidence for two real cases. Nothing needs a link, a teacher or a search.
A. The poster, on the record
| Field | Record |
|---|---|
| Title | Fight tuberculosis – obey the rules of health |
| Maker | WPA Federal Art Project, District 4, New York (Works Progress Administration) |
| Date | Between 1936 and 1941; a date stamp on the back reads May 14, 1941 |
| Medium and size | Color silkscreen print on board, 55.8 x 35.5 cm (about 22 x 14 inches) |
| Held by | Library of Congress, Prints and Photographs Division. Call number POS – WPA – NY .01 .F533, no. 1 (B size). Control number 98513584. Digital file LC-DIG-ppmsca-38342. |
| Rights | Library of Congress rights advisory: “No known restrictions on publication.” A federal government project of the 1930s; treated here as public domain. |
| Catalog summary | “Poster promoting better health care through the prevention of tuberculosis by better eating and sleeping habits, and more exposure to sunshine.” (Library of Congress) |
What the poster looks like. Portrait format. The palette is five flat colors plus the off-white paper: dark red-brown, mustard yellow, tan, black and white. At the top, the words FIGHT TUBERCULOSIS fill two lines in heavy, blocky red-brown capitals that run nearly edge to edge. At the bottom right, OBEY is set large, and THE RULES OF HEALTH runs beneath it in the same letters. Between the words, a woman’s face fills the center, tilted sideways, eyes closed with long lashes, lips red-brown, drawn in flat tan with almost no shading. Her wavy yellow hair curls around her, and a red-brown and yellow lattice band crosses beneath her chin like a blanket or scarf. She lies against a solid black shape whose edge is jagged, like torn or spiked paper. Behind her head, at left, is a tall glass filled with yellow liquid, drawn with white lines. At upper right are a pear, an apple and a bunch of grapes. At upper left is a yellow sun with white rings, and straight yellow rays run diagonally across the top of the sheet. In the lower left margin, in tiny letters running upward, is the credit WPA FEDERAL ART PROJECT. There is no number, no address, no clinic name and no source anywhere on the sheet.
How to read it. The three pictures (sun, food, a resting person) stand for the three habits in the catalog summary: sunshine, better eating, better sleep. The command, “obey the rules of health,” is general. The feeling is calm and warmth, not fear. The poster claims that following these habits will fight the disease, and it offers no evidence for that claim on the sheet itself. That last observation is the one to carry into the product page below.
B. A product page, written out (constructed for this lesson)
SLS original / illustrative. RetinaLite is an invented product. Nothing here describes any real company, and the figure shown is invented.
The page has a deep-blue header with a thin white logo and the words RetinaLite AI. Below it, a photograph of a smiling nurse beside a laptop. The laptop shows a retinal image with a bright green ring drawn around a small patch, and the label “Finding detected.” Beside it, in the largest type on the page, is 97% in white on the blue, and under it the words “Accurate. Fast. Trusted by clinics everywhere.” A row of gray hospital logos follows. A green button reads “Book a demo.” At the very bottom, in small light-gray type: “Based on internal testing.”
| Question | WPA poster | RetinaLite page (invented) |
|---|---|---|
| Feeling it creates | Calm, warmth, care | Confidence, speed, being modern |
| Action it asks for | Follow healthy habits | Book a demo |
| Design tools | Bold flat color, one central image, short command in huge type | Clean white space, glowing overlay on a scan, one giant number, logos as proof |
| Evidence on the sheet | None | One number, no measure named, footnote “internal testing” |
| Who benefits if you believe it | Public health agencies | The company |
C. Case 1: a documented success with a twist (diabetic retinopathy screening)
The regulator’s account. On April 11, 2018 the FDA announced that it had permitted marketing of IDx-DR, which it called the first medical device to use artificial intelligence to detect more than mild diabetic retinopathy in adults with diabetes. It said the device was reviewed under the De Novo pathway, and is the first authorized to give a screening decision “without the need for a clinician to also interpret the image or results.” The FDA evaluated a study of 900 patients with diabetes at 10 primary care sites. In that study IDx-DR correctly identified more than mild retinopathy 87.4 percent of the time and correctly identified patients without it 89.5 percent of the time. The FDA also listed people who should not be screened (for example, those with a history of laser treatment or eye surgery, and pregnant patients) and said patients still need a complete eye exam. The release quotes an FDA official noting that about 50 percent of people with diabetes do not see an eye doctor yearly. (Source: FDA news release, April 11, 2018; the device appears on the FDA AI-enabled device list as DEN180001, final decision 04/11/2018.)
The published trial (open access, CC BY 4.0). Abramoff and colleagues, npj Digital Medicine 1:39 (2018), report that 900 people were enrolled at 10 sites, 819 could be fully analyzed, and 198 of the 819 had more than mild retinopathy by the reading-center standard. The paper states the system’s sensitivity as 87.2% (95% confidence interval 81.8–91.2%), specificity as 90.7% (88.3–92.7%) and imageability (the share of patients for whom the system could give a result) as 96.1%. Notice three things a careful reader checks. The FDA release and the paper give slightly different percentages for the same study (87.4 and 89.5 versus 87.2 and 90.7). The paper’s printed prevalence of 23.8% does not equal 198 divided by 819, which is 24.2%. And the reference standard was a reading center, not the device maker. Note the gaps and say what you would ask the authors.
The deployment study. Beede and colleagues (Google Research) published a human-centered study at the ACM CHI conference in 2020 of a different deep-learning retinopathy system used in eleven clinics in Thailand. Their abstract reports “tensions between the model’s thresholds for data quality, and the quality of data that arise from an imperfect, resource-constrained environment.” In the body of the paper they report that of 1,838 images put through the system in its first six months, 393 (21%) did not meet the system’s standard for grading, and that in the clinics they studied, images sometimes took 60 to 90 seconds to upload. In one clinic an internet outage lasting two hours cut the number of patients screened from 200 to 100. Compute: 393 divided by 1,838 is 21.4%.
D. Case 2: a documented failure of independent validation (sepsis prediction)
Wong and colleagues at Michigan Medicine tested the Epic Sepsis Model, a proprietary model already used at hundreds of U.S. hospitals, on their own patients: 38,455 hospitalizations of 27,697 patients between December 6, 2018 and October 20, 2019. Sepsis occurred in 2,552 hospitalizations (7%). The abstract reports an area under the ROC curve of 0.63 (95% CI 0.62–0.64), that the model “did not identify 1709 patients with sepsis (67%),” and that it generated alerts for 6,971 of the 38,455 hospitalizations (18%). Their conclusion: the model “has poor discrimination and calibration.” (Source: JAMA Internal Medicine 2021;181(8):1065-1070, doi 10.1001/jamainternmed.2021.2626, abstract retrieved from PubMed.) Compute: 2,552 minus 1,709 is 843 caught; 843 divided by 2,552 is 33.0%, or about a third of sepsis cases.
E. The two-column case evidence chart
Build a chart with one column for a documented success or challenge and one for a documented failure. Each column needs four rows: what the maker or reporter says the system does; what independent or regulatory evidence shows; what happened in real use; and what is still not known. Here are the first two rows of the success column filled in as a model. Complete the rest, and the whole failure column, from sections C and D.
| Row | Case 1: retinopathy screening | Case 2: sepsis prediction |
|---|---|---|
| Says what | Detects more than mild diabetic retinopathy in a primary care office without a specialist reading the image (FDA release, Apr. 11, 2018) | (yours) |
| Independent or regulatory evidence | 900 patients, 10 sites; 87.4% and 89.5% (FDA); reading-center standard (npj Digit Med) | (yours) |
| In real use | (yours: use the 1,838, 393 and 21% figures) | (yours) |
| Not known | (yours) | (yours) |
Not hosted here. The lesson’s background also mentions IBM Watson for Oncology. This reading does not host any claim about it, because no primary source for those claims could be checked while this reading was prepared. Use the sepsis case as the documented failure.
Sources: Library of Congress, Fight tuberculosis – obey the rules of health (LCCN 98513584) (loc.gov); FDA news release, April 11, 2018: FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems (fda.gov); Abramoff et al., Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices, npj Digital Medicine 1:39 (2018) (nature.com); Beede et al., A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy, CHI 2020 (doi.org); Wong et al., External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients, JAMA Intern Med 2021 (doi.org)