THE VR SCHOOL
  • Schools & Districts
    • License the academy
    • Free program for your students
    • Request a demo
  • Catalog
    • AI & Data
    • Cybersecurity & Web
    • Programming
    • Health Science
    • All Guilds
    • All courses
  • Families
  • Accreditation
  • Log In
  • Request a demo
  • Home
  • Courses

RLP Statistics A — Exploring Data, Study Design and Probability

Curriculum

  • 4 Sections
  • 20 Lessons
  • 10 Weeks
Expand all sectionsCollapse all sections
  • Unit 1: Exploring One-Variable Data
    5
    • 1.1
      Distributions and the Statistical Process
      50 mins
    • 1.2
      Measuring Center and Spread
      50 mins
    • 1.3
      Comparing Distributions Across Missouri
      50 mins
    • 1.4
      Standardizing, Transforming and Reporting
      50 mins
    • 1.5
      Performance Task — Climate Station Profile
      150 mins
  • Unit 2: Two-Variable Data and Regression
    5
    • 2.1
      Scatterplots and Correlation
      50 mins
    • 2.2
      Least-Squares Regression
      50 mins
    • 2.3
      Residuals, Influence and Causation
      50 mins
    • 2.4
      Exponential Models and the Limits of Extrapolation
      50 mins
    • 2.5
      Performance Task — Bivariate Investigation Report
      150 mins
  • Unit 3: Designing Studies: Samples and Experiments
    5
    • 3.1
      Populations, Samples and Bias
      50 mins
    • 3.2
      Simulating Sampling Variability and the Margin of Error
      50 mins
    • 3.3
      Experiments, Observational Studies and Confounding
      50 mins
    • 3.4
      Comparing Treatments with a Randomization Test
      50 mins
    • 3.5
      Performance Task — Design, Run and Report a Study
      150 mins
  • Unit 4: Probability, Random Variables and the Binomial Model
    5
    • 4.1
      Probability Rules and Two-Way Tables
      50 mins
    • 4.2
      The Multiplication Rule, Trees and Counting
      50 mins
    • 4.3
      Random Variables, Expected Value and Variance
      50 mins
    • 4.4
      The Binomial Model
      50 mins
    • 4.5
      Performance Task — Expected-Value Decision Analysis
      150 mins

Populations, Samples and Bias

Unit 3  ·  Launch & Concept Development  ·  Lesson 11 of 20

Populations, Samples and Bias

A2.DS.A.1RLP-STATSA
Mixed population tokens beside balanced and biased samples, showing how sampling choices affect representation.
By the end of this lesson I can…

distinguish a parameter from a statistic, describe and compare sampling methods, and identify the biases that can make a sample unrepresentative.

Instruction

Statistics lets you learn about a large group from a small part of it, but only if the part is chosen well. A2.DS.A.1 frames the whole idea: a parameter is a number describing a population (usually unknown); a statistic is a number computed from a sample, used to estimate the parameter. Inference is the reasoning from statistic to parameter.

Sampling methods.

  • Simple random sample (SRS): every group of n individuals has the same chance of being chosen. Number the population, then use a random number generator.
  • Stratified random sample: divide the population into groups that are similar inside (strata, such as grade levels or regions of a state), take an SRS from each, and combine. This guarantees each group is represented and usually gives more precise estimates.
  • Cluster sample: divide the population into groups that each look like the population (such as classrooms or city blocks), randomly choose some clusters, and include everyone in them. It is cheaper, but usually less precise.
  • Systematic sample: choose a random start, then every k-th individual on a list.
  • Convenience and voluntary response samples (whoever is easy to reach, or whoever chooses to answer) are not random, and they cannot support inference.

Bias is a systematic tendency to overestimate or underestimate. It comes from the design, and a larger sample does not fix it.

  • Undercoverage: parts of the population are missing from the sampling frame (the list you sample from).
  • Nonresponse: chosen individuals do not respond, and they differ from those who do.
  • Response bias: answers are inaccurate because of wording, the interviewer, or a wish to give the socially acceptable answer.

A famous failure. In 1936 the Literary Digest magazine mailed about 10 million straw ballots and received about 2.4 million back (2,376,523, a response rate of about 24%). It predicted that Alf Landon would defeat Franklin Roosevelt, with 54% of the vote. Roosevelt won in a landslide, with about 61% of the popular vote and 523 of 531 electoral votes. The mailing lists were built largely from telephone directories and automobile registration lists, which leaned toward better-off households in the Depression (undercoverage), and only about one ballot in four came back, so the people who chose to mail a ballot back mattered a great deal (nonresponse). Two point four million responses could not rescue a biased design. The reading for this lesson prints these counts and says where each comes from.

A careful design. The Census Bureau’s American Community Survey (ACS) contacts about 3.5 million addresses each year, chosen by random sampling from the Bureau’s master address file. Response is required by law, and the Bureau follows up by mail, internet, phone and in-person visits to reduce nonresponse. The ACS publishes one-year estimates for areas of 65,000 people or more, and five-year estimates for all areas, down to small neighborhoods and rural counties. In Missouri alone it selects about 77,000 addresses a year. Every estimate comes with a margin of error, an honest statement of how far the statistic might plausibly be from the parameter because of random sampling alone. You will simulate where a margin of error comes from in Lesson 3.2.

Evaluating a claim (A2.DS.A.7). Before trusting a percentage from a survey, ask: Who was the population? What was the frame? How were people chosen? What fraction responded? What exactly was asked? A large, non-random sample is still non-random.

📖 Read for this lesson

The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.

  • Ten Million Ballots and 3.5 Million Addresses: Two Samples, Read by Their NumbersLiterary Digest 1936 returns as quoted in Lohr and Brick, Statistics, Politics and Policy 8(1), 2017 (doi 10.1515/spp-2016-0006) and arXiv 2107.14166; 1936 electoral vote from the National Archives; 1936 popular vote as tabulated on Wikipedia; ACS initial addresses and final interviews for Missouri 2005-2025 and the U.S. 2019-2025 from the Census Bureau’s Sample Size tables; retrieved November 10, 2026 · about 5 min

    The Literary Digest’s returned-ballot counts and response rate, and the Census Bureau’s own addresses-selected and interviews-completed figures for Missouri and the nation, so the failed design and the careful one can be compared by their numbers.

Vocabulary in context

  • parameter — A number describing a population, usually unknown (such as a population proportion p or mean μ).
  • statistic — A number computed from a sample and used to estimate a parameter (such as p̂ or x̄).
  • stratified random sample — A sample formed by taking a separate simple random sample within each of several similar-inside groups (strata).
  • sampling frame — The list or procedure from which the sample is actually drawn.
  • nonresponse bias — Bias that arises when the individuals who do not respond differ systematically from those who do.

Formative check

Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.

+50 XP

A district wants students’ views on a new schedule and wants each grade level represented in proportion. Which design fits best?

Stratifying by grade guarantees each grade is represented; random selection within grades avoids convenience bias.
Myth or Fact?

The Literary Digest’s 1936 poll failed because 2.4 million responses was too small a sample.

The sample was enormous. It failed because of undercoverage and nonresponse bias. A larger sample does not fix a biased design.
Flashcard

Parameter

Tap to reveal
Answer

Describes the population; usually unknown (p, μ).

Flashcard

Statistic

Tap to reveal
Answer

Computed from a sample (p̂, x̄); estimates a parameter.

Flashcard

Undercoverage

Tap to reveal
Answer

Some of the population is missing from the frame.

Flashcard

Voluntary response

Tap to reveal
Answer

People choose themselves; strong opinions are over-represented.

Flashcard

Cluster sample

Tap to reveal
Answer

Randomly choose whole groups and include everyone in them.

Find a news story or website that reports a percentage from a survey or poll. Identify the population, the sampling method (if stated) and the likely sources of bias, and state what the report leaves out. Then propose a better design in four sentences. If you cannot find one, critique this hypothetical example, invented for this lesson: ‘An online poll on a sports website found that 78% of respondents support building a new stadium in their city.’ Check your answer against the five questions in the reading, or ask a parent or guardian to read your four sentences and say whether the design would convince them.

0 words
Quick self-check

How confident are you that you can tell a parameter from a statistic, choose a sampling method and name the biases a design invites?

Not yetVery confident

Practice

Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.

Printable version: this unit’s practice workbook (PDF)

Fluency

Build speed and accuracy with the core skill.

  1. (Hypothetical data.) A survey of 400 randomly chosen households in a Missouri city finds that 23% have solar panels. Identify the population, the parameter, the sample and the statistic.
    Check your answer
    Answer: Population: all households in that city. Parameter: the proportion of all those households with solar panels (unknown). Sample: the 400 households surveyed. Statistic: p̂ = 0.23.
  2. Parameter or statistic? The mean daily high of the 30 randomly chosen days in a sample.
    Check your answer
    Answer: Statistic.
    It describes the sample.
  3. Parameter or statistic? The proportion of all 9,132 days from 2000 to 2024 on which St. Louis Lambert recorded rain.
    Check your answer
    Answer: Parameter (of that population of days).
  4. Name the sampling method: a school lists all 1,200 students, picks a random start between 1 and 20, and surveys every 20th student.
    Check your answer
    Answer: Systematic sampling.
  5. Name the sampling method: a district randomly selects 6 of its 40 homerooms and surveys every student in them.
    Check your answer
    Answer: Cluster sampling.
  6. Name the sampling method: a county takes a separate simple random sample of voters in each of its five townships.
    Check your answer
    Answer: Stratified random sampling.
  7. Name the sampling method: a reporter interviews shoppers at one mall on a Saturday.
    Check your answer
    Answer: Convenience sample (not random).
  8. Does increasing the sample size reduce bias?
    Check your answer
    Answer: No. Larger samples reduce random variability, not bias from the design.

Application

Use the skill in context. Show your reasoning.

  1. (Hypothetical.) A radio station asks listeners to text whether the city should build a new stadium; 78% of the 2,300 texts say yes. Identify the type of sample and the likely direction of bias.
    Check your answer
    Answer: A voluntary-response sample. People with strong opinions (likely enthusiasts) are over-represented, so support is probably overestimated.
  2. A phone survey of a county’s residents uses a list of landline numbers only. What is the bias, and who is likely missing?
    Check your answer
    Answer: Undercoverage: households without landlines (more often younger and lower-income residents) cannot be selected, so the sample may not represent them.
  3. A survey question reads: ‘Given the dangerous condition of our roads, do you support a small tax to fix them?’ Identify the problem and rewrite the question.
    Check your answer
    Answer: Loaded wording (response bias). Neutral version: ‘Do you support or oppose a 0.25% sales tax for road repairs?’
  4. The American Community Survey requires households to respond by law and follows up by mail, internet, phone and in-person visits. Which kind of bias is this design mainly fighting?
    Check your answer
    Answer: Nonresponse bias.
    Follow-up raises the response rate among groups less likely to respond at first.

Challenge

Stretch problems. Expect to think before you write.

  1. A district of 3,000 high-school students (four grades of roughly equal size) wants to estimate average nightly sleep, and suspects seniors sleep less. Design a stratified sample of 200 and explain why it beats a simple random sample of 200.
    Check your answer
    Answer: Take a simple random sample of 50 students from each grade (random numbers on each grade’s roster). Stratifying guarantees each grade is represented and, because sleep differs by grade, reduces the variability of the combined estimate compared with an SRS, which could by chance over- or under-represent seniors.

Review

Keep earlier skills sharp.

  1. Use technology to find the least-squares line and the correlation coefficient for (1, 8), (2, 3), (4, 6), (11, 12), (12, 16), (13, 13).
    Check your answer
    Answer: ŷ = 4.037 + 0.785x; r = 0.879
    Interpret the slope in context: the predicted change in y for each one-unit increase in x.
  2. (Unit 2) A regression of y on x has r = −0.6. What fraction of the variation in y is explained by the linear model?
    Check your answer
    Answer: r² = 0.36, so 36%.
  3. (Unit 1) Which summary is resistant: the SD or the IQR?
    Check your answer
    Answer: The IQR.

Missouri Learning Standards — Mathematics (data and probability) addressed: A2.DS.A.1, A2.DS.A.3, A2.DS.A.7, MP.3, MP.6

Instructional focus: Sampling design and the sources of bias in data collection

Reading

Ten Million Ballots and 3.5 Million Addresses: Two Samples, Read by Their Numbers

Literary Digest 1936 returns as quoted in Lohr and Brick, Statistics, Politics and Policy 8(1), 2017 (doi 10.1515/spp-2016-0006) and arXiv 2107.14166; 1936 electoral vote from the National Archives; 1936 popular vote as tabulated on Wikipedia; ACS initial addresses and final interviews for Missouri 2005-2025 and the U.S. 2019-2025 from the Census Bureau’s Sample Size tables; retrieved November 10, 2026 · SLS original, with public-domain figures from the U.S. Census Bureau and the National Archives and published figures quoted from the cited papers

Why this reading. The lesson tells the Literary Digest story and describes the American Community Survey in words. A design is judged by its numbers: how many were asked, how many answered, and who was on the list. Here are the numbers for both, from published accounts of the Digest poll and from the Census Bureau’s own sample-size tables for Missouri and the nation, so the failure and the careful design can be compared line by line.

The Literary Digest, 1936

The figures below come from the magazine’s own final tally as quoted by later researchers, and are checked against two papers: Sharon Lohr and J. Michael Brick, “Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll” (Statistics, Politics and Policy 8(1), 2017), and a review of poll failures posted on arXiv (2107.14166). The magazine’s headline for its last report read “Landon, 1,293,669; Roosevelt, 972,897” (Literary Digest, October 31, 1936, as listed in Lohr and Brick’s references). The classic study of why the poll failed, Peverill Squire’s “Why the 1936 Literary Digest Poll Failed” (Public Opinion Quarterly 52, 1988), could not be opened for this reading; its conclusion is taken from how those two papers summarize it, and none of its own figures are used.

ItemFigure
Ballots mailedabout 10,000,000
Sampling frametelephone directories and automobile registration lists, plus other lists (Lohr and Brick)
Ballots returned2,376,523 (the count quoted in the arXiv review; Lohr and Brick say “almost 2.4 million”)
Returned for Landon / for Roosevelt1,293,669 / 972,897 (the Digest’s headline); the other 109,957 returned ballots, found by subtraction, went to other candidates
Response rate24% (Lohr and Brick); 2,376,523 / 10,000,000 = 23.8%
Digest’s final poll sharesLandon 54.4%, Roosevelt 40.9% (Lohr and Brick: “54 percent” for Landon)
Election result (electoral vote)Roosevelt 523, Landon 8 (National Archives, Electoral College results 1936)
Election result (popular vote)Roosevelt 27,752,648 (60.8%), Landon 16,681,862 (36.5%) of 45,647,699 (as tabulated in the Wikipedia article on the 1936 election, a secondary source; the National Archives page lists only electoral votes)

Check the arithmetic: 1,293,669 / 2,376,523 = 54.4% for Landon among the returned ballots, against 36.5% of the actual vote: a miss of 17.9 points, in a poll with 2.4 million answers. Lohr and Brick describe two things going wrong together: the lists leaned toward households with telephones and cars (undercoverage of everyone else), and only about one ballot in four came back, so who chose to answer mattered a great deal (nonresponse). They also show that if the answers had been weighted by how respondents said they voted in 1932, the poll would have favored Roosevelt; the Digest did not do that. The lesson’s round figures, about 2.4 million returned and about 61% for Roosevelt, are these numbers rounded.

The American Community Survey: how many asked, how many answered

The Census Bureau publishes, for every state and year, the number of housing-unit addresses initially selected for the ACS sample and the number of final interviews obtained. Missouri, from the Bureau’s Sample Size table (retrieved November 10, 2026):

YearInitial addresses selectedFinal interviewsInterviews as % of addresses
2025 (note 1)70,11338,00054.2%
202477,59043,80756.5%
202377,66044,76157.6%
202277,63445,01858.0%
202177,10042,87855.6%
2020 (note 2)62,49430,56748.9%
201977,48845,01058.1%
2018 (note 3)77,50046,50060.0%
201776,90246,35660.3%
201676,98147,97562.3%
201577,56650,26564.8%
201477,50550,59565.3%
2013 (note 4)77,66748,35362.3%
201277,52452,26067.4%
201174,47848,85665.6%
201064,64343,44067.2%
200964,77543,39067.0%
200864,99543,76767.3%
200764,54143,94268.1%
200664,43444,64069.3%
200564,43843,49367.5%

And for the United States as a whole, the most recent seven years:

YearInitial addresses selectedFinal interviews%
2025 (note 1)3,215,5111,686,00052.4%
20243,543,6721,944,00254.9%
20233,544,3571,978,70455.8%
20223,538,3921,980,55056.0%
20213,538,4421,950,83255.1%
2020 (note 2)2,872,4021,406,93549.0%
20193,544,3012,059,94558.1%

The Bureau’s notes. (1) The lapse in appropriations in October and November 2025 suspended data collection and lowered both the sample and the interview counts for 2025, and the 2025 counts are rounded. (2) The pandemic reduced the number of addresses in sample and the number of interviews in 2020. (3) The 2018 counts were rounded to reduce disclosure risk. (4) The 2013 government shutdown removed the second mailing and the telephone and in-person follow-up for the October 2013 panel, which cut final interviews. Source: Census Bureau, ACS Sample Size page.

Read the Missouri column downward. The sample was about 77,000 to 78,000 addresses a year from 2012 to 2024, but interviews fell from 52,260 in 2012 to 43,807 in 2024, and the share of selected addresses that end in an interview slid from 67% to 56%. Some of the gap is vacant or nonexistent addresses; the rest is nonresponse, which the Bureau fights with mail, internet, telephone and in-person follow-up, and with the legal requirement (Title 13 of the U.S. Code) to respond. The Bureau then weights the interviews to match known population totals. A falling response rate does not make an estimate wrong, but it makes the weighting do more work, and it is the reason the ACS publishes a margin of error with every number. One-year estimates are published for areas of 65,000 people or more; five-year estimates cover every area, down to small neighborhoods and rural counties.

The difference in one sentence

The Digest reached ten million people it chose for convenience and heard back from only about one in four; the ACS reaches 3.5 million addresses chosen at random from a list of every address, follows up the ones that do not answer, and reports how far off it might still be.

What to do

  1. For the Digest poll, name the population, the frame, the sampling method, the response fraction and the two biases, each with a number from the table.
  2. For the ACS, compute Missouri’s 2020 interview percentage (30,567 / 62,494) and explain from the Bureau’s note why it is the lowest in the Missouri table.
  3. Now critique the lesson’s hypothetical sports-site poll (“78% of respondents support building a new stadium”) with the same five questions, and propose a design in four sentences that borrows at least two features from the ACS.

Sources: U.S. Census Bureau, American Community Survey: Sample Size (Missouri) (census.gov); U.S. Census Bureau, American Community Survey: Sample Size (United States) (census.gov); National Archives, Electoral College results: 1936 (archives.gov); S. Lohr and J. M. Brick, Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll, Statistics, Politics and Policy 8(1):65-84 (2017) (doi.org); Why Do Polls Fail? The Case of Four US Presidential Elections, Brexit, and Two India General Elections (arXiv 2107.14166) (arxiv.org); 1936 United States presidential election (Wikipedia; popular vote table) (en.wikipedia.org); P. Squire, Why the 1936 Literary Digest Poll Failed, Public Opinion Quarterly 52:125-133 (1988) (not opened; cited as summarized by the two papers above) (doi.org); U.S. Census Bureau, ACS: When to use 1-year or 5-year estimates (census.gov)

Up NextComplete & Continue →

Leave a Reply Cancel reply

Performance Task — Bivariate Investigation Report
Prev
Simulating Sampling Variability and the Margin of Error
Next

© 2026 The VR School | Career content by Wiley | Powered by Sejal Learning Systems.

Join free · Accreditation · Privacy Policy · Terms of Service

Modal title

Main Content