
use a randomization test to decide whether the difference between two treatment groups in an experiment is larger than chance assignment alone would plausibly produce.
Instruction
An experiment gives two groups different treatments and finds a difference in their responses. But random assignment itself creates differences: even with no treatment effect, the groups would not come out exactly equal. The question is whether the observed difference is larger than chance assignment alone would plausibly produce.
An experiment (illustrative data). Twenty student volunteers were randomly assigned, 10 and 10, to study a list of 25 words for two minutes either in silence or with music with lyrics playing. Each then wrote down as many words as they could recall. These numbers are illustrative, written for this lesson.
- Silence: 14, 12, 15, 11, 13, 16, 12, 14, 10, 15 (mean 13.2)
- Music: 11, 9, 13, 12, 10, 8, 12, 11, 14, 10 (mean 11.0)
The observed difference in means is 13.2 − 11.0 = 2.2 words.
The null model. Suppose the music makes no difference: each student would have recalled the same number of words either way. Then the 20 scores are fixed, and the only thing that made the groups differ was which 10 names happened to be drawn for “silence”. We can recreate that chance process.
The randomization test.
- Put all 20 scores in one pool.
- Randomly deal them into two groups of 10, and compute the difference in means (group 1 − group 2).
- Repeat many times (at least 1,000) and make a dot plot of the differences: the randomization distribution.
- Find the proportion of shuffles giving a difference at least as large as the observed 2.2. That proportion is the P-value.
Here the whole distribution can be computed exactly, because there are only C(20, 10) = 184,756 ways to split the 20 scores. Of these, 2,347 give a difference of 2.2 or more, so the one-sided P-value is 2,347 / 184,756 ≈ 0.013. Counting differences of 2.2 or more in either direction gives a two-sided P-value of about 0.025. A simulation of 1,000 shuffles will give a value close to these.
The conclusion. If music truly made no difference, a gap this large would arise from random assignment only about 1 time in 80 (one-sided). That is unusual enough that we have convincing evidence that, for these volunteers, studying with lyrical music reduced recall compared with silence. Because treatments were randomly assigned, a causal conclusion is justified. Because the volunteers were not a random sample, generalizing to all students is a judgment, not a statistical result.
How to read a P-value. A small P-value means the data would be surprising if the null model were true. It is not the probability that the null model is true, and it does not measure how large or important the effect is. Report the difference (2.2 words, about 17% of the silence mean) alongside the P-value. Statistics B formalises all of this as a significance test.
Deciding if a model is consistent with data (A2.DS.A.2). The null model (“no effect”) is a model of the data-generating process. Simulation shows what that model produces. When the real result sits far out in the tail of the simulated results, the model and the data disagree.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
In the randomization test, what does shuffling the 20 scores into two new groups of 10 simulate?
A one-sided P-value of 0.013 means there is a 1.3% chance that music has no effect.
Simulation & tools
Build a randomization distribution
Data: the illustrative recall scores in the lesson (type them in: two attributes, group and words).
- Open the Scrambler plugin (Plugins menu) and scramble group. Scrambling re-deals the group labels at random, so there are still exactly 10 in each group.
- Measure the difference in mean words between shuffled groups; have the Scrambler collect at least 500 differences.
- Dot-plot the differences. Count those at 2.2 or above. Your P-value should be near 0.013.
- Now change one music score from 8 to 13 and repeat. How does the P-value change, and why?
No CODAP: write the 20 scores on cards, shuffle and deal into two piles of 10 at least 30 times (a parent or guardian can deal a few rounds with you), or use a spreadsheet with =RAND() beside each score, sorting by it and averaging the top and bottom 10, and recalculate many times.
Open in a new tab ↗ · CODAP, Concord Consortium · MIT open source
Write a four-sentence conclusion for the music experiment that states the observed difference, the P-value and what it means, whether a causal conclusion is justified and why, and to whom the result can be generalized.
How confident are you that you can carry out a randomization test for an experiment and state a conclusion with its scope?
Practice
Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.
Printable version: this unit’s practice workbook (PDF)
Fluency
Build speed and accuracy with the core skill.
- Group A: 12, 15, 14, 17. Group B: 10, 11, 13, 10. Find the difference in means (A − B).
Check your answer
Answer: 3.50.14.5 − 11.0. - In a randomization test, what does each shuffle of the pooled responses represent?
Check your answer
Answer: One possible random assignment of the same responses to groups, assuming the treatment has no effect. - In 1,000 shuffles, 37 gave a difference at least as large as the observed one. What is the approximate one-sided P-value?
Check your answer
Answer: About 0.037. - A randomization test gives P ≈ 0.42. What should you conclude?
Check your answer
Answer: The observed difference is the kind chance assignment often produces; the data do not give convincing evidence of a treatment effect (this does not prove there is no effect). - With the same group sizes and the same spread of responses, how does a larger observed difference between treatment means affect the randomization P-value?
Check your answer
Answer: It makes the P-value smaller: fewer shuffles produce a difference that large. - Write a correct sentence interpreting a one-sided P-value of 0.013 from the music experiment.
Check your answer
Answer: If music had no effect, a difference of 2.2 words or more would occur about 1.3% of the time by random assignment alone.
Application
Use the skill in context. Show your reasoning.
- (Illustrative data) Sixteen paper helicopters were randomly assigned to long or short blades, and flight times (seconds) recorded. Long: 2.41, 2.58, 2.36, 2.62, 2.49, 2.55, 2.44, 2.60. Short: 2.30, 2.47, 2.28, 2.39, 2.52, 2.33, 2.41, 2.35. Find the difference in means and interpret the exact one-sided randomization P-value: 127 of the 12,870 possible splits give a difference at least that large.
Check your answer
Answer: Difference 0.125 s (long − short); P = 127/12,870 ≈ 0.0099. If blade length made no difference, a gap this large would arise by chance assignment less than 1% of the time: convincing evidence that longer blades increased flight time for these helicopters.Means 2.5063 and 2.3812 s. - State the scope of the helicopter conclusion.
Check your answer
Answer: Random assignment supports a causal conclusion for these 16 helicopters (built and dropped this way). Nothing was randomly selected from a larger population, so extending it to other designs is a judgment. - A student compares 15 volunteers who chose to study with music with 15 who chose silence, runs a randomization test and gets P = 0.02. What is wrong with concluding that music caused the difference?
Check your answer
Answer: The students chose their own groups, so the groups may differ in other ways (self-selection). The randomization test models random assignment, which did not happen; the result shows an association only.
Challenge
Stretch problems. Expect to think before you write.
- Treatment: 5, 8, 9. Control: 2, 4, 6. List all C(6, 3) = 20 ways to choose 3 values for ‘treatment’, compute the difference in means for each, and find the exact one-sided P-value.
Check your answer
Answer: The observed difference is 3.333. Only 2 of the 20 splits give a difference of at least 3.333, so P = 2/20 = 0.10.The two splits are {5, 8, 9} (3.333) and {6, 8, 9} (4.0).
Review
Keep earlier skills sharp.
- (Unit 2) Why is a randomized experiment the only way to show causation directly, when a correlation alone cannot?
Check your answer
Answer: Random assignment balances lurking variables, so differences in the response can be attributed to the treatment; with observational data, lurking variables may explain the association. - (Unit 1 skill) A group of scores has mean 13.2 and SD 1.93 words. Find the z-score of a score of 16.
Check your answer
Answer: z ≈ 1.45.
All 184,756 Splits: The Exact Randomization Distribution for the Music Experiment, and One Seeded Run
Exact enumeration of the C(20, 10) re-assignments of the lesson’s illustrative recall scores, and a 1,000-shuffle simulation with Python’s random module, seed 2026; computed for this course · SLS original (computed reference distribution for the lesson’s illustrative data)
Why this reading. The lesson says the whole randomization distribution can be computed exactly, quotes 2,347 of 184,756, and tells you your simulation “will give a value close to these”. Close is not checkable. Here is the exact distribution, every possible value of the difference in means with its count, and one simulated run of 1,000 shuffles with its seed, so that your CODAP or card-shuffle result can be laid beside both.
The data (illustrative, from the lesson)
Silence: 14, 12, 15, 11, 13, 16, 12, 14, 10, 15 (sum 132, mean 13.2). Music: 11, 9, 13, 12, 10, 8, 12, 11, 14, 10 (sum 110, mean 11.0). All twenty: sum 242. Observed difference 2.2 words. These scores were written for the lesson and measure no real students.
The exact randomization distribution
There are C(20, 10) = 184,756 ways to choose which ten of the twenty scores are labeled “silence”. For each, the difference in means is (S − (242 − S)) / 10, where S is the sum of the ten chosen scores, so every possible difference is a multiple of 0.2. The distribution is exactly symmetric about 0 (swap the labels and the sign flips), so one side is enough. The count of splits at exactly 0.0 is 14,856.
| Difference | Splits at exactly this value (each sign) | Proportion | Splits at or above (positive side) | One-sided P-value |
|---|---|---|---|---|
| ±0.2 | 14,610 | 0.07908 | 84,950 | 0.4598 |
| ±0.4 | 13,759 | 0.07447 | 70,340 | 0.3807 |
| ±0.6 | 12,520 | 0.06777 | 56,581 | 0.3062 |
| ±0.8 | 10,895 | 0.05897 | 44,061 | 0.2385 |
| ±1.0 | 9,145 | 0.04950 | 33,166 | 0.1795 |
| ±1.2 | 7,306 | 0.03954 | 24,021 | 0.1300 |
| ±1.4 | 5,617 | 0.03040 | 16,715 | 0.0905 |
| ±1.6 | 4,079 | 0.02208 | 11,098 | 0.0601 |
| ±1.8 | 2,834 | 0.01534 | 7,019 | 0.0380 |
| ±2.0 | 1,838 | 0.00995 | 4,185 | 0.0227 |
| ±2.2 | 1,135 | 0.00614 | 2,347 | 0.0127 |
| ±2.4 | 634 | 0.00343 | 1,212 | 0.0066 |
| ±2.6 | 335 | 0.00181 | 578 | 0.0031 |
| ±2.8 | 153 | 0.00083 | 243 | 0.0013 |
| ±3.0 | 64 | 0.00035 | 90 | 0.0005 |
| ±3.2 | 20 | 0.00011 | 26 | 0.0001 |
| ±3.4 | 6 | 0.00003 | 6 | 0.0000 |
Read the row for 2.2: 2,347 of 184,756 splits give a difference of 2.2 or more, P = 0.0127, which the lesson rounds to 0.013; counting both tails, 4,694 splits and P = 0.0254. The standard deviation of this distribution is 0.961 words, so the observed 2.2 sits 2.29 standard deviations from the center of what chance assignment produces.
One simulated run, seed 2026
Python: rng = random.Random(2026), then 1,000 times: shuffle the list of 20 scores with rng.shuffle, call the first ten “silence”, record (mean of first ten) − (mean of last ten). Frequencies:
| Difference | -3.0 | -2.8 | -2.4 | -2.2 | -2.0 | -1.8 | -1.6 | -1.4 | -1.2 | -1.0 | -0.8 | -0.6 | -0.4 | -0.2 | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 1.2 | 1.4 | 1.6 | 1.8 | 2.0 | 2.2 | 2.4 | 2.6 | 3.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Shuffles | 2 | 1 | 2 | 3 | 9 | 13 | 19 | 34 | 52 | 56 | 63 | 66 | 70 | 75 | 85 | 73 | 64 | 64 | 65 | 56 | 33 | 33 | 20 | 21 | 7 | 6 | 6 | 1 | 1 |
In this run, 14 of 1,000 shuffles reached 2.2 or more (P ≈ 0.014) and 22 had |difference| ≥ 2.2 (0.022); the mean of the 1,000 differences was -0.001 and their SD 0.964. The exact one-sided answer is 0.0127. A run of 1,000 estimates a probability near 0.013 with a standard error of about √(0.013 × 0.987 / 1000) = 0.0036, so anything from roughly 0.006 to 0.020 is an ordinary result. Two honest runs will disagree; neither is wrong. Report the exact value where it can be had, and the number of repetitions where it cannot.
The lesson’s follow-up: change one music score from 8 to 13
Music becomes 11, 9, 13, 12, 10, 13, 12, 11, 14, 10, mean 11.5, and the observed difference shrinks to 1.7. Exactly 5,676 of 184,756 splits give 1.7 or more, so the one-sided P-value rises to 0.031, more than double. One score moved five words and the evidence weakened from about 1 in 79 to about 1 in 33. A randomization test on twenty people is sensitive to every one of them.
Without a computer
Twenty index cards, one score each. Shuffle, deal ten to the left, compute the difference, record it, repeat. Do this at least 30 times, or add a spreadsheet column of =RAND() beside each score, sort by it, and average the top ten and bottom ten, then recalculate again and again. The count at or above 2.2 divided by the number of shuffles is your P-value. Lay it beside the tables above; with only 30 deals it will be rough, and the table shows how rough is normal. A parent or guardian can deal a few rounds with you if you would like company.