
design and run a randomized experiment, analyze it with a randomization test, and evaluate a published Missouri survey’s margin of error and claim.
Instruction
This task puts the whole unit together. You will ask a question, answer it with a randomized experiment you run yourself, and then turn the same critical eye on a published Missouri survey.
Part 1: your experiment. Choose a question about cause and effect that volunteers can safely help answer in a short session. Good examples: Does background music change how many arithmetic problems people solve in two minutes? Does a printed checklist reduce errors in copying a table? Does reading a passage aloud rather than silently change comprehension scores? Avoid anything involving health, food, personal information or risk, and have a parent or guardian review and sign your written plan before you collect any data.
Your design must specify the treatments, the response variable and how it is measured, how many subjects, and exactly how you will randomly assign them, for example by drawing names from a bag or using a random-number generator. Every participant must agree to take part and may stop at any time; record no names with the data. Aim for at least 10 subjects per treatment.
Analysis. Describe the two groups with dot plots or box plots and their means and standard deviations. Compute the observed difference in means. Then run a randomization test with at least 1,000 shuffles in CODAP, a spreadsheet or JupyterLite, and report the proportion of shuffled differences at least as extreme as yours. State your conclusion in the language your design supports: random assignment licenses a causal conclusion for your subjects; your subjects were not a random sample, so say to whom the result might extend and why.
Part 2: a Missouri survey. Find a published survey of Missourians that reports a margin of error, such as a Saint Louis University/YouGov poll of likely Missouri voters or an estimate from the Missouri Behavioral Risk Factor Surveillance System. Identify the population, the sample size, how respondents were selected and the margin of error. Check whether the reported margin is close to 2√(p(1 − p)/n) for one reported percentage, and explain any difference (surveys often adjust for their design). Then evaluate one claim the report or a news story makes from it: does the design support it, and is the difference it highlights larger than the margin of error?
If your data are roughly normal, you may also fit a normal model and estimate a population percentage (A2.DS.B.8 / A2.DS.B.9), but only after showing the check. Presentations include live questions from classmates, who will be assessed on the quality of their questions.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
Which procedure is random assignment?
A poll of 1,600 adults reports 52% support with a margin of error of 3 points. What can you conclude about a majority?
Stage 1: state your question, treatments, response variable and how you will measure it, your randomization procedure and your planned number of subjects. Write the one-sentence conclusion you would draw if your randomization proportion came out at 0.01, and the one you would draw if it came out at 0.30.
How confident are you that you can design, conduct, analyze and present a simulation-based inference study?
Unit 4 checkpoint
Do this before you start the task. It covers the whole unit and tells you what to review first.
Students who choose an online section are compared with those who choose in-person. Design?
Random selection allows a cause-and-effect conclusion.
Normal, mean 40, SD 5: percent between 30 and 50?
For the same distribution, P(X < 45) ≈ .
The empirical rule can be applied to any data set with a mean and SD.
A fair coin gives 60+ heads in 100 flips about 2.8% of the time. Best conclusion for a coin with 60 heads?
In the $1 dice game paying $5 on 7 or 11, P(win) = /36.
Quadrupling a survey’s sample size makes the margin of error…
For p̂ = 0.62 and n = 400, the margin of error (2 SD) is about .
A randomization proportion of 0.0017 proves the treatment affected every subject.
52% ± 3 points. Does this establish a majority?
Submit: Simulation-Based Inference Study report (5-7 pages) + 6-minute presentation with questions
Task: Design and conduct a randomized experiment with at least 10 subjects per treatment, collected with consent and without names. Describe the data, run a randomization test with at least 1,000 shuffles in free software, and state a conclusion matched to the design. Then evaluate a published Missouri survey: its population, sample, selection method and margin of error, a check of the margin against 2√(p(1 − p)/n), and one claim made from it.
Submission workflow
- Stage 1 — Plan. Question, treatments, response variable, randomization procedure, consent script and sample size, reviewed and signed by a parent or guardian before data collection.
- Stage 2 — Peer-reviewed draft. A partner reruns your randomization test from your data and checks that your conclusion matches your design; submit a revision note.
- Stage 3 — Final report. 5-7 pages: design, data and graphs, simulation method and result, conclusion with limits, and the survey evaluation.
- Stage 4 — Six-minute presentation. Present your study and answer live questions. A recorded presentation followed by written answers to classmates’ questions is an equal-credit alternative.
How it is scored
| Criterion | What proficient looks like |
|---|---|
| Comparing treatments (A2.DS.A.3) | The randomization test is run correctly and its result is interpreted accurately. |
| Design (A2.DS.A.3) | Treatments, response and random assignment are specified and carried out as planned. |
| Inference language (A2.DS.A.1) | Conclusions distinguish sample from population and causation from association. |
| Margin of error (A2.DS.A.4 / A2.DS.A.5) | The survey’s margin is identified and checked with a correct calculation. |
| Evaluating reports (A2.DS.A.7) | The published claim is judged against the design and the margin of error. |
| Argument and communication (MP.3, MP.4) | The report and presentation state assumptions, limits and conclusions clearly and answer questions with evidence. |
A Dossier of Missouri Surveys, and a Randomization Test Worked to the Last Split
SLU/YouGov Poll, February 2026 Top Line Results (September 2026 revision; fieldwork February 9-22, 2026; 900 likely voters; margin of error 3.91%) and August 2026 Top Line Results (fieldwork August 13-24, 2026; 900 likely voters; margin of error 4.09%), Saint Louis University; Missouri Department of Health and Senior Services, BRFSS overview; CDC BRFSS Prevalence Data, Missouri, 2021 and 2024; retrieved November 5, 2026 · SLS original, with figures as published by Saint Louis University/YouGov and public-domain U.S. and Missouri government survey data
Why this reading. Part 2 of the task says “find a published survey of Missourians that reports a margin of error.” Finding one should not be the hard part, and it should not depend on a search engine cooperating on the day. Three are documented below, with everything the task asks you to identify. Choose one and work it. The randomization-test check at the end is for Part 1.
Survey 1. SLU/YouGov Poll, February 2026
| What the task asks for | As published |
|---|---|
| Population | Missouri likely voters |
| How respondents were selected | YouGov interviewed 1,448 Missouri registered voters online, matched them down to 1,085 to resemble a frame drawn from the Missouri part of the Current Population Survey, and kept the 900 likely voters; weighted by propensity scores and post-stratified |
| Fieldwork dates | February 9–22, 2026 |
| Sample size | 900 likely voters; some questions asked of half the sample (450) |
| Reported margin of error | ±3.91 points for the full sample (September 2026 revision of the results); ±5.82 for most half-sample questions |
| Reported design effect | 1.22, effective sample size 737 |
| Some reported findings | Missouri “on the right track and headed in a good direction”: 48% agree, 40% disagree, 12% not sure (asked of 450, ±5.82). Public schools in your community: 8% excellent, 36% good, 29% fair, 21% poor. A–F grades for public schools and districts: 59% support, 19% oppose. Top priority for state government: economy 49%, health care 19%, education 12% |
The check the task wants. 2√(0.25/900) = 0.03333, or 3.33 points, against a reported 3.91. Implied design effect = (3.91/3.333)2 = 1.38. Using the poll’s own effective sample of 737 instead, 2√(0.25/737) = 3.68 points — closer, and still a little short, for reasons Lesson 4.4’s reading explains. A question asked of only 450 has a larger margin, which is why the poll lists one for each such question rather than one number for the whole report.
Survey 2. SLU/YouGov Poll, August 2026
| Item | As published |
|---|---|
| Fieldwork | August 13–24, 2026 |
| Sample | 900 likely voters (1,276 registered voters interviewed, matched to 1,074) |
| Reported margin of error | ±4.09 points, full sample; design effect 1.25, effective sample 720 |
| Some reported findings | Missouri “on the right track”: 39% agree, 48% disagree, 13% not sure (asked of 451, ±5.67). Public schools in your community: 7% excellent, 32% good, 33% fair, 24% poor. A–F grades for public schools and districts: 60% support, 19% oppose, 21% not sure. A state-funded paid parental leave program for public school employees: 70% support, 14% oppose, 16% not sure. Own physical health in the last month: 14% excellent, 51% good, 26% fair, 9% poor |
Lesson 4.4’s reading works this survey’s margin in full, including the standard error of a difference between two shares from the same sample. Because the February and August polls asked several of the same questions, they also let you test a change — and two claims from August alone behave differently from each other.
Claim A: “Fewer Missouri voters think the state is on the right track: 48% in February, 39% in August.” This compares two separate samples, so the standard errors add in squares. Each reported margin is about two standard errors, so the margin of the change is 2√((5.82/2)2 + (5.67/2)2) = 8.13 points (both are half-sample questions, so the wider margins apply). The observed change is 9 points, just outside it: the drop is supported, though barely. Now try the same test on the share rating their local public schools “good,” 36% in February and 32% in August, both asked of the full sample: the margin of the change is 2√((3.91/2)2 + (4.09/2)2) = 5.66 points, and a 4-point change is inside it, so “fewer rate their schools good” is not supported. (A–F grades, at 59% and 60%, did not move at all.)
Claim B: “A majority of likely voters support paid parental leave for public school employees” (70%). The interval 70 ± 4.09 runs from 65.9% to 74.1%, entirely above 50%. This claim clears the margin comfortably. Compare “A majority of voters rate their physical health in the last month as good” (51%) in the same survey: 51 ± 4.09 runs from 46.9% to 55.1%, so a majority is not established. Two claims, one survey, one margin of error, different verdicts — which is the point of checking rather than quoting.
Survey 3. Missouri Behavioral Risk Factor Surveillance System, 2024
| Item | As published |
|---|---|
| Conducted by | Missouri Department of Health and Senior Services, with the University of Missouri-Columbia, coordinated nationally by the CDC |
| Population | Missouri adults, 18 and older |
| Selection | Random selection of households by telephone; listed and unlisted landline and cell numbers; responses weighted |
| 2024 sample (general-health question) | 7,271 Missouri adults |
| One estimate | Fair or poor general health: 20.07% of Missouri adults, 95% confidence interval 18.82% to 21.32% |
| Access | The CDC’s BRFSS prevalence data report a confidence interval and sample size with every estimate rather than one margin for the survey |
The published interval has a half-width of (21.32 − 18.82)/2 = 1.25 points. The formula gives 2√(0.2007 × 0.7993/7,271) = 0.94 points, so the implied design effect is (1.25/0.94)2 ≈ 1.77 — heavier weighting than the polls, because a telephone survey must correct for who answers the phone. At p = 0.5 the state-level margin would be 2√(0.25/7,271) = 1.17 points before the design effect. A subgroup of 400 (one age group, say) carries a margin of about 2√(0.25/400) = 5 points, which is useless for a 1-point difference and perfectly good for a 20-point one. Say which comparison your number is good enough for.
A claim to test: “More Missouri adults rate their health fair or poor than three years earlier.” The same CDC data give 17.4% for 2021 (interval 16.4% to 18.4%, from 12,227 adults). The change is 20.07 − 17.4 = 2.67 points. Turning each published 95% interval back into a standard error (half-width ÷ 1.96) and adding in squares, the margin of the change is about 1.63 points, so the change is larger than sampling error alone would usually produce. Whether it is a change in Missourians’ health or in who answered the phone is a question the margin cannot settle.
Choosing a claim to evaluate
Whichever survey you take, the evaluation has four steps and each needs a number.
- Does the design support the claim’s logic? All three surveys here select and weight respondents to represent their populations, so they support generalizing, with the caveats the methods state (the polls use a matched online panel, not a pure random sample). None assigns treatments, so none supports a causal claim. A headline saying one of these surveys shows X causes Y is wrong regardless of its sample size.
- Is the difference the claim highlights larger than the margin? Compute it. For a difference between two separate samples or subgroups, the standard error is √(SE12 + SE22); for two shares of the same respondents, use the formula in Lesson 4.4’s reading instead.
- Does the reported margin match 2√(p(1−p)/n)? If it is larger, compute the design effect and say what weighting bought in exchange.
- What is not in the margin at all? Nonresponse, panel recruitment, question wording, question order, and the timing of the fieldwork. None of the margins above includes any of them; say so in your report.
A randomization test, worked to the last split, for Part 1
Lesson 4.4’s word-recall experiment — 10 volunteers in silence, 10 with notifications — is the template for your own analysis, so here is its arithmetic in full, as a target for your Stage 2 partner check.
| Group | Scores (words recalled of 40) | Mean | Sample SD |
|---|---|---|---|
| Silence | 31, 28, 35, 30, 27, 33, 29, 32, 26, 34 | 30.5 | 3.03 |
| Notifications | 25, 29, 24, 27, 30, 22, 26, 28, 23, 26 | 26.0 | 2.58 |
Observed difference in means: 30.5 − 26.0 = 4.5 words. Under the null hypothesis that the treatment did nothing, each person’s score would have been the same either way, so the only thing random is which ten got the “silence” label. There are C(20, 10) = 184,756 ways to split twenty scores into two groups of ten. Enumerating every one of them — not sampling, enumerating — gives 316 splits with a difference of 4.5 or more in favor of silence, so the exact proportion is 316/184,756 = 0.00171.
That is the number the lesson quotes, and now you can see where it comes from. A simulation of 1,000 shuffles estimates the same quantity and will typically return 0, 1, 2 or 3 hits — which, as Lesson 4.3’s reading shows, is a very imprecise estimate of 0.0017. When your own experiment produces a small proportion, say how many shuffles produced it, and if your groups are small enough to enumerate completely, do that instead.
The conclusion, in the language the design supports. Random assignment licenses cause: among these twenty volunteers, the notification condition lowered mean recall by 4.5 words, and a difference that large arises from the assignment alone about 17 times in 10,000. The volunteers were not randomly selected, so nothing here establishes anything about students in general — extending it is an argument about similarity, and it belongs in a separate sentence that says so.
Sources: Saint Louis University, SLU/YouGov Poll, February 2026 Top Line Results, September 2026 revision (PDF) (slu.edu); Saint Louis University, SLU/YouGov Poll, August 2026 Top Line Results (PDF) (slu.edu); Missouri Department of Health and Senior Services, Behavioral Risk Factor Surveillance System (BRFSS) (health.mo.gov); CDC, BRFSS Prevalence Data (2011 to present), Missouri (data.cdc.gov dataset dttw-5yxu) (data.cdc.gov)