
distinguish sample surveys, experiments and observational studies, explain how randomization works in each, and judge what conclusions each design supports.
Instruction
Statistics is the science of drawing conclusions about a large group from a smaller one, and of saying how confident we can be (A2.DS.A.1). The population is the whole group we care about; a parameter is a number describing it, such as the proportion of all Missouri adults who have health insurance. We rarely can measure everyone, so we measure a sample and compute a statistic, such as the proportion in the sample. The whole unit is about the gap between the statistic we have and the parameter we want.
Three designs.
- A sample survey selects individuals from a population and records their answers or measurements. The goal is to estimate a population parameter. The key tool is random selection: every member of the population has a known chance of being chosen, so the sample tends to resemble the population. The Missouri Behavioral Risk Factor Surveillance System (BRFSS), run by the Missouri Department of Health and Senior Services with the federal Centers for Disease Control and Prevention, is a random-sample telephone survey of Missouri adults used to estimate health conditions, behaviors and insurance coverage across the state.
- An experiment deliberately imposes treatments on subjects and measures a response. The goal is to find out whether a treatment causes a change. The key tool is random assignment: subjects are assigned to treatments by chance, so the groups are alike, on average, in every way except the treatment. In the 1954 field trial of the Salk polio vaccine, part of the trial randomly assigned children to receive either the vaccine or a placebo, which is why its result could be read as the vaccine’s effect.
- An observational study records what individuals already do, without imposing anything. It can show an association but not, on its own, causation, because the groups may differ in other ways. Students who eat breakfast may earn higher grades, but they may also sleep more or have more support at home. Those other variables are called confounding variables.
Two kinds of randomness, two kinds of conclusions. Random selection lets you generalize from the sample to the population. Random assignment lets you conclude cause and effect. A study can have one, both or neither. An experiment on volunteers from one school supports a causal conclusion for those volunteers, but generalizing to all Missouri teenagers requires an argument that the volunteers resemble them. A random-sample survey supports generalization, but its associations are not causal.
Bias. A method that systematically favors some outcomes is biased, and a larger sample does not fix bias. An online poll where anyone can click is a voluntary response sample; people with strong opinions respond most. Surveying only people with landline phones undercovers younger adults. The wording of a question can push answers one way.
Reading reports (A2.DS.A.7). When you meet a claim based on data, ask four questions. Who was studied, and how were they chosen? Was a treatment imposed, and was it assigned at random? How large was the sample, and what is the margin of error? Does the conclusion match the design, or does it claim causation from an observational study or generalize from a convenience sample? The discourse habit (MP.3) is to name exactly which step of the argument the design does or does not support.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
Researchers compare the grades of students who choose to take an online course with those who choose an in-person section. What kind of study is this?
Random selection of subjects allows a cause-and-effect conclusion.
A website poll of 50,000 readers finds 80% oppose a new law. What is the main problem?
Find or use a provided news headline that claims one thing causes another based on data. Identify the design, the population and how the subjects were chosen, and write a paragraph judging whether the headline’s claim is supported, naming any possible confounding variable.
How confident are you that you can classify study designs, explain how randomization works in each and judge what they can conclude?
Practice
Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.
Printable version: this unit’s practice workbook (PDF)
Fluency
Build speed and accuracy with the core skill.
- Classify: researchers randomly select 1,200 Missouri adults by phone and ask about health insurance.
Check your answer
Answer: Sample survey (random selection) - Classify: 40 volunteers are randomly assigned to a new or standard reading app, and scores are compared.
Check your answer
Answer: Experiment (random assignment) - Classify: a study compares asthma rates in neighborhoods near and far from freeways.
Check your answer
Answer: Observational studyNothing is assigned. - Which design supports generalizing to a population, and which supports cause and effect?
Check your answer
Answer: Random selection supports generalizing; random assignment supports cause and effect. - An online poll where anyone can vote gets 30,000 responses. Name the type of sample and its main problem.
Check your answer
Answer: Voluntary response sample; it is likely biased, and size does not fix bias. - Define parameter and statistic, with one example of each from a survey of 500 Missouri teenagers.
Check your answer
Answer: Parameter: the proportion of all Missouri teens who…; statistic: the proportion among the 500 surveyed.
Application
Use the skill in context. Show your reasoning.
- A headline says: ‘Students who eat breakfast earn higher grades, so breakfast raises grades.’ Name the design, a possible confounding variable and a design that could test the causal claim.
Check your answer
Answer: Observational; for example, sleep or family support; a randomized experiment assigning breakfast. - A school surveys the first 50 students to arrive in the morning about the start time. Why might this be biased, and how could it be fixed?
Check your answer
Answer: Early arrivers may favor early starts; select students at random from the full roster. - In the 1954 Salk vaccine field trial, part of the trial randomly assigned children to vaccine or placebo. Why did random assignment matter?
Check your answer
Answer: It made the groups alike on average except for the vaccine, so the difference could be attributed to it.
Challenge
Stretch problems. Expect to think before you write.
- Design a study of whether a new study-hall schedule improves homework completion at your school that supports both generalizing to the school and a causal conclusion.
Check your answer
Answer: Randomly select students from the roster, then randomly assign the selected students to the new or current schedule; compare completion rates.
Review
Keep earlier skills sharp.
- A quantity starts at 15,000 and grows by 5% each year. Write a model and find its value after 4 years, to the nearest whole number.
Check your answer
Answer: A = 15,000(1.05)t; after 4 years ≈ 18,233The growth factor is 1 + 0.05 = 1.05. - Find the exact value of sin(3π/4).
Check your answer
Answer: √2/2Use the reference angle and the sign of the function in that quadrant.
Three Designs, Three Kinds of Conclusion: The 1954 Polio Trial, a State Survey and a Headline
1954 poliomyelitis vaccine field trial results as reported by Francis and colleagues, American Journal of Public Health, 1955; Missouri Department of Health and Senior Services, Behavioral Risk Factor Surveillance System (BRFSS) overview page; CDC, BRFSS home page; CDC BRFSS Prevalence Data (2011 to present), Missouri, 2024, general health status; retrieved November 5, 2026; the four headline cases are SLS original · SLS original, with published trial figures and public-domain U.S. and Missouri government survey data
Why this reading. The lesson’s last task says to “find or use a provided news headline” and judge it. Four headlines are provided below, with enough of each study behind them to judge. Before them are two real designs worth having in your head: the trial that settled the polio vaccine, and the survey that estimates Missouri’s health. The trial is the clearest example ever run of why one design supports a causal claim and another does not.
The 1954 polio vaccine field trial: two designs in one study
The trial had two arms, run in different parts of the country, because health officials disagreed about whether withholding a possible vaccine was acceptable.
Arm 1, randomized placebo control. In participating areas, children in grades 1–3 whose parents consented were assigned at random to receive either vaccine or a saline placebo. Neither the children, their parents, nor the doctors diagnosing polio knew which they had received.
Arm 2, observed control. In other areas, all consenting second-graders were offered vaccine, and first- and third-graders were counted as controls. Nothing was assigned; the comparison was between grades, and between children whose parents consented and children whose parents did not.
| Arm | Group | Children | Paralytic polio cases | Rate per 100,000 |
|---|---|---|---|---|
| Randomized placebo control | Vaccinated | 200,745 | 33 | 16.4 |
| Placebo | 201,229 | 115 | 57.1 | |
| Observed control | Vaccinated (grade 2, consented) | 221,998 | 38 | 17.1 |
| Controls (grades 1 and 3) | 725,173 | 330 | 45.5 |
Check the rates yourself: 33/200,745 × 100,000 = 16.4, and 115/201,229 × 100,000 = 57.1. In the randomized arm the placebo rate is 57.1/16.4 = 3.48 times the vaccinated rate, a reduction of 1 − 16.4/57.1 = 71%. In the observed arm the ratio is 45.5/17.1 = 2.66, a reduction of 62%.
Why the two arms disagree, and which to believe. The observed arm’s control group is not comparable to its treatment group. Its controls include children whose parents refused consent, and consent was more common in higher-income families — where, for reasons of early-childhood exposure, paralytic polio was more common. So the observed arm’s controls were on average at lower risk than its vaccinated children, which drags the apparent benefit down. Only the randomized arm can rule that out, because random assignment makes the two groups alike on average in everything, including the things nobody thought to measure. That is the whole argument for random assignment, and here it is worth nine percentage points of measured effect.
Note what the trial cannot tell you either way: it enrolled volunteers in selected districts, not a random sample of American children, so generalizing it rests on a judgment about similarity, not on the design.
The Missouri Behavioral Risk Factor Surveillance System: a design for the other question
The BRFSS is an ongoing telephone health survey of adults, conducted in all 50 states and coordinated by the federal Centers for Disease Control and Prevention (CDC). Nationally it was established in 1984 with 15 states and now completes more than 400,000 adult interviews a year. In Missouri it is run by the Department of Health and Senior Services (DHSS), which works with the University of Missouri-Columbia to make the calls. Households are randomly selected; listed and unlisted residential landline numbers and cell phone numbers are all included, and the interviewer does not ask for the respondent’s name. The interview takes about 25 minutes and covers health conditions, behaviors, preventive care and access to health care.
How big is the Missouri sample? In the CDC’s published BRFSS prevalence data for 2024, 7,271 Missouri adults answered the general-health question, and 20.07% of Missouri adults, after weighting, rated their health as fair or poor, with a 95% confidence interval of 18.82% to 21.32%. Because respondents are selected by a probability method, the BRFSS estimates population parameters — the share of all Missouri adults with a given condition or habit — and says how precisely.
What the BRFSS cannot do is establish cause. If it shows that adults without health insurance report worse health, no part of that design rules out the possibility that being in worse health affects employment, and employment affects insurance. Nobody assigned anything. Random selection bought generalization; it does not buy causation, and the two randomizations are not interchangeable.
| Random selection? | Random assignment? | Supports generalizing? | Supports causation? | |
|---|---|---|---|---|
| Polio trial, randomized arm | no | yes | by argument only | yes |
| Polio trial, observed arm | no | no | by argument only | no |
| Missouri BRFSS | yes | no | yes | no |
| Website click-in poll | no | no | no | no |
Four headlines to judge (SLS original)
These four cases were written for this lesson. The headlines are invented; each is built on a design that is common in real reporting. For each, name the design, the population, how subjects were chosen, what the conclusion would need, and at least one confounding variable if the design is observational.
A. “Students who play a musical instrument score higher in math.” — Researchers matched school records for 4,100 students in one large Missouri district and found that students enrolled in band or orchestra averaged 31 points higher on the state mathematics assessment than students who were not. No student was assigned to anything.
B. “Standing desks cut afternoon drowsiness, study finds.” — Ninety office workers at one company volunteered. Each was assigned by a random-number generator to a standing or a seated desk for four weeks, and all rated their alertness at 3 p.m. daily on the same 1–10 scale. The standing group averaged 0.8 points higher.
C. “Four in ten Missourians skipped dental care over cost.” — From a probability sample of 4,800 Missouri adults, weighted to match the state’s age, region and income distribution, with a reported margin of error of ±1.8 points.
D. “92% of readers say the new bus lanes are a failure.” — From a poll posted on a local news site over one weekend; 14,200 people clicked a response.
Three questions to settle across all four. Which one has the largest sample and the least trustworthy result, and why does that ordering not embarrass statistics? Which one supports a causal claim, and for whom exactly? And in case A, name two variables that could produce the 31-point gap without any music lesson ever causing anything — then say what study you would have to run instead, and why nobody runs it.
One thing to notice about B. It licenses a causal conclusion, and its subjects are ninety volunteers at one company. “Standing desks cut drowsiness” is a claim about everyone; the study supports “among these ninety volunteers, assignment to a standing desk raised mean self-rated alertness by 0.8 points.” Most of the distance between those two sentences is the headline writer’s, not the researcher’s, and learning to see exactly where a claim outruns its design is the skill this whole unit is for.
Sources: Missouri Department of Health and Senior Services, Behavioral Risk Factor Surveillance System (BRFSS) (health.mo.gov); Centers for Disease Control and Prevention, Behavioral Risk Factor Surveillance System (cdc.gov); CDC, BRFSS Prevalence Data (2011 to present), Missouri 2024 (data.cdc.gov dataset dttw-5yxu) (data.cdc.gov); University of Michigan School of Public Health, the 1955 polio vaccine trial announcement (sph.umich.edu)