THE VR SCHOOL
  • Schools & Districts
    • License the academy
    • Free program for your students
    • Request a demo
  • Catalog
    • AI & Data
    • Cybersecurity & Web
    • Programming
    • Health Science
    • All Guilds
    • All courses
  • Families
  • Accreditation
  • Log In
  • Request a demo
  • Home
  • Courses

RLP AI Design B — Case Studies, Prototypes, Patents and Capstone

Curriculum

  • 4 Sections
  • 20 Lessons
  • Lifetime
Expand all sectionsCollapse all sections
  • Unit 1: AI in the Clinic: Case Studies from Past to Present
    5
    • 1.1
      Posters, Press Releases and Proof: How Health Technology Is Sold
      50 mins
    • 1.2
      Storyboarding a Talk: Sticky Notes, Props and the Stage
      100 mins
    • 1.3
      Talking to Machines: A Structured Written Debate on Mental Health Chatbots
      100 mins
    • 1.4
      Radiology, Prediction and Telemedicine: Writing the Talk
      150 mins
    • 1.5
      Portfolio Task — Case Talk: Success, Challenge or Failure
      200 mins
  • Unit 2: Building Small Models: Testing AI as Designers
    5
    • 2.1
      Artists and Machines: From AARON to Trained Collaborators
      50 mins
    • 2.2
      Train, Test, Tally: A Controlled Experiment on a Classifier
      100 mins
    • 2.3
      Ninety Percent Accurate? Base Rates and Honest Accuracy Graphics
      50 mins
    • 2.4
      Sequences and Anomalies: Coding a Heart-Rate Flag
      150 mins
    • 2.5
      Portfolio Task — Tested Prototype Exhibit
      200 mins
  • Unit 3: Engineering Design and the Patent Drawing
    5
    • 3.1
      The Art of the Patent Drawing and the Inventor’s Process
      50 mins
    • 3.2
      Drawing to Patent Standard: Views, Numerals and Line
      100 mins
    • 3.3
      Reading a Real Patent: Claims, Drawings and What Can Be Owned
      100 mins
    • 3.4
      From Sketch to Prototype: Engineering a Biomedical Design
      150 mins
    • 3.5
      Portfolio Task — Engineering Design Portfolio and Hypothetical Patent
      200 mins
  • Unit 4: Capstone: A Biomedical AI Innovation from Concept to Exhibition
    5
    • 4.1
      Showing the Future: Exhibitions, Pitches and the Capstone Launch
      50 mins
    • 4.2
      Paper Prototypes and Think-Aloud Testing
      100 mins
    • 4.3
      Money, Rules and Ethics: A Review Panel on the Record
      100 mins
    • 4.4
      Resume, Portfolio and Recorded Mock Interview
      150 mins
    • 4.5
      Portfolio Task — Capstone Exhibition: A Biomedical AI Innovation
      250 mins

Ninety Percent Accurate? Base Rates and Honest Accuracy Graphics

Unit 2  ·  Critique & Discourse Lab  ·  Lesson 8 of 20

Ninety Percent Accurate? Base Rates and Honest Accuracy Graphics

MA:Re9A.IIIRLP-AIDB
Two identical screening stations show mostly false alarms in a rare-condition group and mostly true positives in a common-condition group.
By the end of this lesson I can…

explain why positive predictive value depends on how common a condition is, and evaluate how accuracy claims about AI tools are presented in media.

Instruction

“Our AI detects the disease with 90% accuracy.” You will see claims like this in advertising, news and even research summaries. After this lesson you will know the questions to ask, and how to design a graphic that tells the truth.

The base-rate problem, worked through. Suppose a screening tool has 90% sensitivity and 90% specificity, and it is used on 1,000 people in a population where 2% have the condition. These numbers are illustrative. Work through it step by step. Two percent of 1,000 is 20 people with the condition, and 980 without. With 90% sensitivity, the tool flags 18 of the 20 (true positives) and misses 2 (false negatives). With 90% specificity, it correctly clears 882 of the 980 (true negatives) and wrongly flags the other 98 (false positives). So 18 + 98 = 116 people are flagged, and only 18 of them have the condition. The positive predictive value is 18 ÷ 116 ≈ 15.5%. A person who gets a positive result from this “90% accurate” tool has only about a one-in-six chance of having the condition.

What changed? Nothing about the tool. The same tool used where half the people have the condition would have a much higher PPV. The base rate, how common the condition is in the group being tested, drives the meaning of a positive result. That is why screening tools are followed by confirmatory tests, and why a tool validated in a specialist clinic (where the condition is common) can behave very differently in general screening.

Questions to ask of any accuracy claim. Accuracy of what: sensitivity, specificity, overall accuracy, or area under a curve? Tested on whom, and by whom? How common was the condition in the test group, compared with where the tool will be used? Was it tested on new data from a different site? What happens after a positive result?

Designing honest graphics. Research on risk communication has found that many people understand these ideas better as natural frequencies (18 out of 116) than as percentages, and better still in an icon array: a grid of 1,000 small figures, colored by outcome. Your 1,000-figure grid would show 18 true positives in one color, 98 false positives in another, 2 missed cases in a third, and 882 correctly cleared in a neutral tone. Seen together, the 98 false alarms dwarf the 18 true cases. This is Neurath’s Isotype principle from Semester A, applied to AI.

This critique. You will examine two of the accuracy claims printed in this lesson’s reading, taken from product pages, news stories and published studies. For each, identify which accuracy measure is reported (if any), what is missing, and whether the visual presentation (a big percentage, a gauge, a green tick) creates more confidence than the evidence supports. Then redesign one as an icon array or natural-frequency statement. Then write both sides of one question: when a company reports only its best number, is that misleading or just marketing? Write the strongest case for the view you do not hold, in two or three sentences, before you give your own answer.

📖 Read for this lesson

The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.

  • Ninety Percent Accurate? Worked Tables, a Real Tool and Three Claims to CritiqueWorked tables are SLS original and illustrative; IDx-DR figures from FDA news release 2018-04-11 and Abramoff et al., npj Digit Med 2018 (retrieved 2026-09-26); the reconstructed confusion table is an SLS calculation · about 5 min

    The lesson’s confusion table worked out, the same tool at four base rates, a real tool’s published numbers turned into a table, three accuracy claims with their fine print, and the exact layout of the icon array.

Vocabulary in context

  • base rate — How common a condition is in the group being tested; also called prevalence.
  • natural frequencies — Risk expressed as counts of people, such as 18 out of 116, instead of percentages.
  • icon array — A grid of small figures colored by outcome, used to show risks and test results.
  • confirmatory test — A follow-up test used to check a positive screening result.

Formative check

Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.

+50 XP

In the worked example (1,000 people, 2% base rate, 90% sensitivity, 90% specificity), how many people are flagged positive in total?

18 true positives + 98 false positives = 116.
Make a prediction

The same tool is used on 1,000 people where 50% have the condition. Will the positive predictive value be higher or lower than 15.5%?

Much higher. 500 with the condition: 450 true positives. 500 without: 50 false positives. PPV = 450 / 500 = 90%. The base rate drives the meaning of a positive result.
Myth or Fact?

A single large ‘90%’ on a product page gives viewers enough information to judge what a positive result means for them.

It omits which measure, the test population, the base rate and what happens after a positive result.

Critique one accuracy claim: which measure is reported, what is missing, and how the visual presentation affects confidence. Then describe your icon-array redesign in two or three sentences.

0 words
Quick self-check

How confident are you that you can explain base rates and positive predictive value, and critique and redesign accuracy graphics?

Not yetVery confident

Practice

Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.

Printable version: this unit’s practice workbook (PDF)

Fluency

Build speed and accuracy with the core skill.

  1. Define base rate, natural frequencies, icon array and confirmatory test.
    Check your answer
    Answer: Base rate: how common a condition is in the group being tested (prevalence). Natural frequencies: risk expressed as counts of people, such as 18 out of 116. Icon array: a grid of small figures colored by outcome. Confirmatory test: a follow-up test used to check a positive screening result.
    Many people understand natural frequencies and icon arrays better than percentages.
  2. List the five questions the lesson says to ask of any accuracy claim.
    Check your answer
    Answer: Accuracy of what (sensitivity, specificity, overall accuracy or area under a curve)? Tested on whom, and by whom? How common was the condition in the test group compared with where the tool will be used? Was it tested on new data from a different site? What happens after a positive result?
    These questions apply to advertising, news and research summaries.
  3. In the lesson’s worked example (1,000 people, 2% base rate, 90% sensitivity and specificity), give the four counts and the PPV.
    Check your answer
    Answer: TP 18, FN 2, FP 98, TN 882; PPV about 15.5%.
    2% of 1,000 = 20; 90% of 20 = 18 flagged, 2 missed; 980 without; 90% of 980 = 882 cleared, 98 false alarms; 18 / (18 + 98) = 18 / 116 = 15.5%.

Application

Use the skill in context. Show your reasoning.

  1. The same tool (90% sensitivity, 90% specificity) is used on 1,000 invented people where 10% have the condition. Find TP, FN, FP, TN and the PPV.
    Check your answer
    Answer: TP 90, FN 10, FP 90, TN 810; PPV 50%.
    10% of 1,000 = 100 with; 90% of 100 = 90 TP, 10 FN. 900 without; 90% of 900 = 810 TN, 90 FP. PPV = 90 / (90 + 90) = 50%.
  2. Now 1% of 1,000 invented people have the condition. Find TP, FP and the PPV to one decimal place. What pattern do you see across 1%, 2%, 10% and 50% base rates?
    Check your answer
    Answer: TP 9, FP 99, PPV about 8.3%. As the base rate rises (1%, 2%, 10%, 50%), PPV rises (8.3%, 15.5%, 50%, 90%), though the tool never changes.
    1% of 1,000 = 10; 90% of 10 = 9 TP. 990 without; 10% of 990 = 99 FP. 9 / 108 = 8.3%. The 50% figure is from the lesson: 450 / 500 = 90%.
  3. Describe the icon array for the 2% example: how many figures in each color, and what a viewer sees first.
    Check your answer
    Answer: 1,000 figures: 18 true positives in one color, 98 false positives in a second, 2 missed cases in a third, and 882 correctly cleared in a neutral tone. The viewer sees that the 98 false alarms dwarf the 18 true cases.
    18 + 98 + 2 + 882 = 1,000. This is Neurath’s Isotype principle applied to AI.
  4. An invented product page shows a big green gauge reading ‘90%’ and the words ‘Clinically proven.’ Write a critique using two of the five questions, then a natural-frequency redesign for a 2% base rate.
    Check your answer
    Answer: Model critique: It does not say which measure 90% is, or on whom it was tested and how common the condition was there; the green gauge and ‘clinically proven’ create more confidence than the evidence shown supports. Redesign: ‘If 1,000 people are screened and 20 have the condition, the tool flags 116 people; 18 of them have it.’
    A strong answer names missing information and links the visual (gauge, green tick) to overconfidence, then uses counts.
  5. Write the strongest case, in 2-3 sentences, for the view that reporting only a company’s best number is just marketing, not misleading.
    Check your answer
    Answer: Model: ‘Every advertisement leads with its strongest point, and buyers such as hospitals are expected to read the full study before they purchase. The best number is usually true, and full results are often published elsewhere. Calling it misleading treats marketing as if it were the whole evidence base.’
    The lesson asks you to write the strongest case for the view you do not hold before giving your own answer. Either view can be argued.

Review

Keep earlier skills sharp.

  1. Review 2.2: From the lesson-2.2 worked example (TP 42, FP 15), what is the PPV? Why is it higher than the 15.5% in this lesson’s example?
    Check your answer
    Answer: About 73.7%. The 2.2 example had a much higher base rate (50 of 200, or 25%), so true positives were a larger share of all positives.
    42 / 57 = 73.7%; 50 / 200 = 25%, compared with 2%.

Missouri Learning Standards — Media Arts addressed: MA:Re9A.III, MA:Re7A.II, MA:Cn11A.II, SL.11-12.3, RST.11-12.8

Instructional focus: Critique of quantitative claims and their visual presentation

Reading

Ninety Percent Accurate? Worked Tables, a Real Tool and Three Claims to Critique

Worked tables are SLS original and illustrative; IDx-DR figures from FDA news release 2018-04-11 and Abramoff et al., npj Digit Med 2018 (retrieved 2026-09-26); the reconstructed confusion table is an SLS calculation · SLS original (constructed data, computations, claims A and C), with public-domain figures from the U.S. FDA and CC BY 4.0 figures from Abramoff et al.

Why this reading. The lesson asks you to explain why positive predictive value depends on how common a condition is, to critique two accuracy claims and to redesign one as an icon array. This reading supplies the worked confusion table, the same tool at four different base rates, a real tool’s published numbers turned into a confusion table, three claims to critique with their fine print, and the exact layout of a 1,000-icon array. Every number is computed on the page so you can check it.

1. The lesson’s example, as a confusion table

SLS original / illustrative. A screening tool with sensitivity 90% and specificity 90%, used on 1,000 people of whom 2% have the condition.

Has the conditionDoes notTotal
Test positiveTP = 18FP = 98116
Test negativeFN = 2TN = 882884
Total209801,000

Derivation: 2% of 1,000 = 20; 980 do not have it. Sensitivity 0.90 x 20 = 18 true positives, so 2 missed. Specificity 0.90 x 980 = 882 true negatives, so 98 false positives. Positive predictive value = 18 / 116 = 15.5%. Negative predictive value = 882 / 884 = 99.8%. Overall accuracy = (18 + 882) / 1,000 = 90.0%.

2. The same tool, four base rates (10,000 people each)

Base rateHave itTPFNFPTNPPVAccuracy
0.5%504559958,9554.3%90.0%
2%200180209808,82015.5%90.0%
10%1,0009001009008,10050.0%90.0%
50%5,0004,5005005004,50090.0%90.0%

Look at the last two columns. Overall accuracy stays at about 90% at every base rate, while the meaning of a positive result runs from 4 chances in 100 to 9 in 10. That is why “90% accurate” alone does not tell a person what their result means.

3. A real tool’s published numbers, turned into a table

The FDA’s April 11, 2018 release on IDx-DR reports that in a study of retinal images from 900 patients with diabetes at 10 primary care sites, the device correctly identified more than mild diabetic retinopathy 87.4 percent of the time and correctly identified patients without it 89.5 percent of the time. The trial report (Abramoff et al., npj Digital Medicine 2018, CC BY 4.0) says 819 participants could be fully analyzed and 198 of them had more than mild retinopathy. From these, SLS reconstructs a table by choosing the whole-number counts that reproduce the FDA’s percentages: 173 / 198 = 87.4% and 556 / 621 = 89.5%. This is a reconstruction for practice; the paper’s own table may differ, since the paper states 87.2% and 90.7%.

DiseaseNo diseaseTotal
Test positiveTP = 173FP = 65238
Test negativeFN = 25TN = 556581
Total198621819

Sensitivity 173 / 198 = 87.4%. Specificity 556 / 621 = 89.5%. Prevalence in this study 198 / 819 = 24.2%. PPV = 173 / 238 = 72.7%. NPV = 556 / 581 = 95.7%. In this study group, about 3 in 4 positive results were true. Now a thought experiment (the device is intended for adults with diabetes, so this is not a claim about the device): keep sensitivity and specificity but change the base rate, per 1,000 people.

Base rateTrue positives (per 1,000)False positives (per 1,000)PPV
1.0%8.7103.67.8%
5.0%43.799.430.5%
10.0%87.494.248.1%
24.2%211.379.472.7%
50.0%436.952.389.3%

4. Three claims to critique

Claim A (SLS original, invented; no real product). A page for an invented skin-cancer app says “97% accurate” in large type. Fine print: “Accuracy is correct results divided by all results on our test set of 1,000 photos, of which 30 showed cancer.” Check: a program that answered “no cancer” for every photo would be right on 970 of 1,000, which is 97%, with a sensitivity of 0. The page reports neither sensitivity nor specificity.

Claim B (real, from the FDA release above). “87.4 percent” and “89.5 percent,” with the words “correctly identify” and the study size given. Here the measure is named, the group is named (900 patients at 10 primary care sites), and the base rate is not stated in the release.

Claim C (SLS original, invented headline). “New AI beats doctors, catching 95% of cases.” Fine print: tested on 200 patients all known to have the disease; the AI flagged 190. That is a sensitivity of 190 / 200 = 95%. Specificity cannot be computed because nobody in the test was disease-free.

Question to ask of each claimABC
Which measure is reported?Accuracy onlySensitivity and specificitySensitivity only
Tested on whom?1,000 photos, 3% with cancer900 patients, 10 sites200 known cases
Base rate reported?Yes, hidden in fine printNo100% by design
What is missing?Sensitivity; the do-nothing baseline of 97%Base rate; what happens after a positiveSpecificity; any healthy people

5. The icon array: exact layout for the lesson’s example

Use a grid of 40 rows by 25 columns (1,000 icons), filled left to right, top to bottom, in this order: 18 true positives (dark red), 2 missed cases (hollow red), 98 false positives (orange), 882 correctly cleared (light gray). Row 1 holds the 18 dark red, the 2 hollow and 5 orange icons. Rows 2 to 4 are all orange (75 icons); row 5 has 18 orange and 7 gray; rows 6 to 40 are all gray (35 rows x 25 = 875 icons). Check: 5 + 75 + 18 = 98 orange; 7 + 875 = 882 gray. Label it “Of 1,000 people tested, 20 have the condition” and put the count 18 out of 116 positives beside it.

Sources: FDA news release, April 11, 2018 (IDx-DR) (fda.gov); Abramoff et al., npj Digital Medicine 1:39 (2018) (nature.com)

Up NextComplete & Continue →

Leave a Reply Cancel reply

Train, Test, Tally: A Controlled Experiment on a Classifier
Prev
Sequences and Anomalies: Coding a Heart-Rate Flag
Next

© 2026 The VR School | Career content by Wiley | Powered by Sejal Learning Systems.

Join free · Accreditation · Privacy Policy · Terms of Service

Modal title

Main Content