
explain Type I and Type II errors and power, distinguish statistical from practical significance, recognize the multiple-testing problem, and critique a report’s use of P-values.
Instruction
Two ways to be wrong. A test ends in a decision, and a decision can be wrong in two ways:
| H0 true | H0 false | |
|---|---|---|
| Reject H0 | Type I error (false alarm), probability α | Correct: probability = power |
| Fail to reject | Correct | Type II error (missed effect), probability β = 1 − power |
The consequences decide which matters more. For the transit example in Lesson 3.1, a Type I error means announcing that support differs from 60% when it does not. A Type II error means missing a real change in opinion. For a medical screening test, a Type II error can mean missing an illness. Choosing α is a judgment about consequences, not a law of nature.
Power. Power is the probability of rejecting H0 when a specific alternative is true. For the transit test (H0: p = 0.60, n = 150, α = 0.05, two-sided), you reject when p̂ falls below 0.522 or above 0.678. If the true support were 50%, p̂ would fall below 0.522 about 70% of the time: power ≈ 0.70. If the true support were 55%, power would be only about 0.24 at n = 150. It rises to about 0.42 at n = 300 and 0.70 at n = 600. Power increases with sample size, with the size of the true effect, and with α. A study that fails to reject with low power has not shown “no effect”. It was never likely to find one.
Statistical versus practical significance. With enough data, trivial differences become statistically significant. Illustrative: if two groups of 100,000 people each show 50.5% and 50.0% agreement, a two-proportion test gives z ≈ 2.24 and P ≈ 0.025, “significant”, for half a percentage point. Always report the size of the effect, with a confidence interval, and judge whether it matters in context. The St. Louis-Springfield temperature test in Lesson 3.2 had P < 0.001 and a difference under 1°F. Whether 1°F matters is a question about climate and people, not about P.
Multiple testing. If you run 20 independent tests at α = 0.05 and every null hypothesis is true, the chance of at least one Type I error is 1 − 0.9520 ≈ 0.64. Testing many comparisons and reporting only the significant one, sometimes called P-hacking, produces findings that do not replicate. Decide your hypotheses before seeing the data, report every test you ran, and treat a lone significant result among many as a lead to check, not a discovery.
0.05 is not a cliff. In Lesson 3.1, the normal approximation gave P ≈ 0.046 and the exact binomial gave P ≈ 0.055. Treating one as “significant” and the other as “not” would be absurd: the evidence is identical. In 2016 the American Statistical Association published a statement on P-values (Wasserstein and Lazar, The American Statistician). Its principles, paraphrased: a P-value measures how incompatible the data are with a specified model. It does not measure the probability that the hypothesis is true, or the size or importance of an effect. Scientific conclusions should not rest only on whether P passes a threshold, and proper inference requires full reporting and transparency.
In the discourse lab (MP.3) you will critique a report that uses significance language: a news article on a study, or a claim someone makes from the unit data. Ask four questions. What was H0? How big is the effect, with an interval? How many tests were run? Did the design allow the causal language used? Then reread your own critique and ask the same four questions of it.
📖 Read for this lesson
The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.
Formative check
Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.
A study with n = 40 finds P = 0.30 and concludes the treatment has no effect. What is the best critique?
Running 20 independent tests at α = 0.05 when all nulls are true gives P(at least one false positive) = 1 − 0.95^20 ≈ .
A P-value of 0.001 means the effect is large and important.
Record (or write) a 90-second critique of a report that uses the words ‘significant’ or ‘no effect’ (a news story about a study, or one of the three reports in this lesson’s reading). Answer: What was H₀? How large is the effect, and is there an interval? How many tests were run? Is the causal language justified by the design? Then add one correction or addition that a careful reader (a parent or guardian, if you like) might make to your critique.
How confident are you that you can judge significance-test claims using errors, power, effect size and multiple testing?
Practice
Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.
Printable version: this unit’s practice workbook (PDF)
Fluency
Build speed and accuracy with the core skill.
- A test of H₀: ‘the new filter does not reduce smoke particles’ rejects H₀, but in fact the filter does nothing. Which error occurred?
Check your answer
Answer: A Type I error. - A test fails to reject H₀: ‘this batch of lettuce is not contaminated’, but the batch is contaminated. Which error, and why does it matter here?
Check your answer
Answer: A Type II error; contaminated food reaches people. - What is the probability of a Type I error when H₀ is true and α = 0.05?
Check your answer
Answer: 0.05. - Name three ways to increase the power of a test.
Check your answer
Answer: Increase the sample size, increase α, or study a larger true effect (reduce variability also helps). - Ten independent tests at α = 0.05, all nulls true: find P(at least one false positive).
Check your answer
Answer: 1 − 0.95¹⁰ ≈ 0.401. - True or false: a very small P-value means the effect is large.
Check your answer
Answer: False.Large samples make small effects significant.
Application
Use the skill in context. Show your reasoning.
- A two-sided one-proportion test uses H₀: p = 0.5 and α = 0.05. If the true p is 0.6, find the power for n = 100, 200 and 400.
Check your answer
Answer: About 0.52, 0.81 and 0.98.Find the rejection cutoffs under H₀, then the probability of landing beyond them when p = 0.6. - (Illustrative) A survey of 100,000 people in each of two regions finds 50.6% and 50.2% agreeing with a statement. The two-proportion z is about 2.53 and P ≈ 0.011. Write a sentence that reports both statistical and practical significance.
Check your answer
Answer: The difference of 0.4 percentage points is statistically significant (P ≈ 0.011) only because the samples are huge; a gap this small is unlikely to matter in practice. - A study tests 20 foods for a link to headaches and reports only that chocolate was ‘significantly linked’ (P = 0.03). Critique the report.
Check your answer
Answer: With 20 tests at α = 0.05, about one false positive is expected even if no food matters (P of at least one ≈ 0.64). The chocolate result should be reported alongside all 20 tests and treated as a lead to check in a new study.
Challenge
Stretch problems. Expect to think before you write.
- A small study (n = 12 per group) finds P = 0.30 and concludes ‘the tutoring program has no effect.’ Its 95% interval for the effect runs from −4 to +11 points. What should the conclusion say instead?
Check your answer
Answer: That the study is inconclusive: the data are consistent with no effect, a small negative effect, or a large positive effect of up to 11 points. A larger study is needed.
Review
Keep earlier skills sharp.
- Test H0: p = 0.4 against Ha: p > 0.4, given 74 successes in a random sample of 200. Find the test statistic and the P-value.
Check your answer
Answer: p̂ = 0.370; z ≈ −0.87; P-value ≈ 0.8068. At α = 0.05, fail to reject H0: there is not convincing evidence that the population proportion is greater than 0.4.Use p0, not p̂, in the standard error of a test. State the conclusion in context: reject or fail to reject H0, never “accept”. - (Unit 2) Why is an interval for the effect more informative than a P-value alone?
Check your answer
Answer: It shows the size of the effect and which values remain plausible, not just whether 0 is ruled out.
Three Reports That Say ‘Significant’: Power, Effect Size and Multiple Testing, with the Numbers to Check Them
Power table computed for this course; Report A is illustrative; Reports B and C are SLS framings of real data from NOAA NCEI Global Summary of the Year (TAVG, three Missouri stations, 1950-1979 and 1995-2024) and GHCN-Daily (St. Louis PRCP, 2000-2024); retrieved 2026-11-10 · SLS original, with public domain NOAA data
Why this reading. The discourse task says to critique “a report that uses the words significant or no effect”, and offers a news story you would have to find or a claim you may not have come across. Three reports are supplied here, each with the numbers a critique needs. Report A is constructed for the lesson and labeled so. Reports B and C are written in the voice of a careless analyst about real NOAA data from this course; the framing is SLS original, the numbers are not. Before them is the power table behind the lesson’s figures, so “the study had low power” can be a computed statement rather than a phrase.
Table 1. Power of the two-sided transit test (H0: p = 0.60, α = 0.05)
Reject when p̂ falls outside 0.60 ± 1.96√(0.6 × 0.4/n). Power is the probability of that under a stated true p, using the normal model with SD √(p(1 − p)/n). Worked for n = 150, p = 0.55: the lower cutoff is 0.5216; SD = √(0.55 × 0.45/150) = 0.0406; z = (0.5216 − 0.55)/0.0406 = −0.70; Φ(−0.70) = 0.242, plus a negligible upper-tail piece, gives 0.24.
| n | Reject if p̂ below / above | true p = 0.50 | 0.55 | 0.58 | 0.62 | 0.65 |
|---|---|---|---|---|---|---|
| 150 | 0.5216 / 0.6784 | 0.70 | 0.24 | 0.08 | 0.08 | 0.23 |
| 300 | 0.5446 / 0.6554 | 0.94 | 0.43 | 0.11 | 0.11 | 0.42 |
| 600 | 0.5608 / 0.6392 | 1.00 | 0.70 | 0.17 | 0.17 | 0.71 |
| 1200 | 0.5723 / 0.6277 | 1.00 | 0.94 | 0.29 | 0.29 | 0.95 |
The lesson’s 0.70, 0.24, 0.42 and 0.70 are the first row’s first two entries and the 0.55 column at n = 300 and 600. Read across a row: power grows with the size of the true effect. Read down a column: it grows with n. Read the 0.58 column: a two-point difference from 0.60 is nearly invisible to a survey of 150 and still missed more often than not at n = 1,200. A survey that “found no significant change” at n = 150 has ruled out almost nothing.
Report A (illustrative; constructed for this lesson)
“A school piloted an after-school tutoring program. Twenty students who enrolled and twenty who did not were tested in September and again in January. The tutored group gained an average of 5.1 points on the district assessment; the comparison group gained 3.1. The difference was not statistically significant (P = 0.30). The program has no effect on achievement and will not be renewed.” Supplementary numbers: the standard deviation of the gains was 6.0 points in each group.
What a critique can compute. SE of the difference = √(6.02/20 + 6.02/20) = 1.897; t = 2.0/1.897 = 1.05; df ≈ 38; P = 0.30. The 95% interval for the difference: 2.0 ± 2.024(1.897) = -1.8 to 5.8 points. It includes zero, and it also includes a gain of five points, which would be a large effect for a program of this size. Power: if the true effect were 3 points, this design would detect it 34% of the time; if 4 points, 54%; if 5, 73%. The study was never likely to find the effect it was looking for. Then the design: students enrolled; nothing was assigned. Ask the lesson’s four questions in order.
Report B (SLS framing; real NOAA data)
“Our analysis of federal climate records finds that St. Louis has warmed dramatically: the mean annual temperature at Lambert Airport in 1995–2024 was 57.89°F against 55.32°F in 1950–1979, a highly statistically significant increase (t = 6.52, P < 0.0001). Climate change has raised Missouri’s temperature by two and a half degrees.”
The numbers are right; the Lesson 3.4 mini-lecture has them. Here is what the analyst had and did not print: the same comparison at all three stations the course uses.
| Station | n (1950–79 / 1995–2024) | Mean (SD), earlier | Mean (SD), later | Difference °F | t | df | P | 95% interval |
|---|---|---|---|---|---|---|---|---|
| St. Louis | 30 / 30 | 55.32 (1.36) | 57.89 (1.68) | 2.57 | 6.52 | 56 | < 0.0001 | 1.78 to 3.36 |
| Springfield | 30 / 30 | 55.66 (1.30) | 57.08 (1.52) | 1.42 | 3.89 | 57 | 0.0003 | 0.69 to 2.15 |
| Joplin | 30 / 30 | 57.39 (1.39) | 58.77 (1.36) | 1.38 | 3.87 | 58 | 0.0003 | 0.67 to 2.09 |
Questions for the critique. The St. Louis effect is 2.57°F with an interval of 1.78 to 3.36: is that “two and a half degrees” or “somewhere between about 1.8 and 3.4”? Why report one station when three were available? St. Louis shows the largest difference of the three, and Springfield and Joplin show about half of it; a reader given only the St. Louis row cannot know which was the most dramatic. A station inside a large metropolitan area can warm for local reasons (buildings, pavement, equipment changes) as well as regional ones. And “Missouri’s temperature” is a claim about a state from airports, none of which was chosen at random. Which of the ASA checklist items below does Report B fail?
Report C (SLS framing; real NOAA data)
“Analyzing 25 years of St. Louis rainfall, we tested each day of the week separately against the others. Thursday stands out with 431 rainy days against an expected 407, a nearly significant excess (P = 0.10), and Tuesday shows a corresponding deficit. We suggest that midweek atmospheric patterns, possibly linked to weekday commuter emissions, favor Thursday storms.”
The counts are the Lesson 4.1 data. Test them the analyst’s way, one day at a time: under H0 each day expects 2,846/7 = 406.6 rainy days with SD √(2,846 × (1/7)(6/7)) = 18.67.
| Day | Rainy days | z against 406.6 | Two-sided P |
|---|---|---|---|
| Mon | 408 | +0.08 | 0.94 |
| Tue | 383 | −1.26 | 0.21 |
| Wed | 426 | +1.04 | 0.30 |
| Thu | 431 | +1.31 | 0.19 |
| Fri | 401 | −0.30 | 0.77 |
| Sat | 397 | −0.51 | 0.61 |
| Sun | 400 | −0.35 | 0.72 |
Thursday’s two-sided P is 0.19; the analyst’s 0.10 is a one-sided P, with the direction chosen after seeing which day was high. “Nearly significant” is doing a great deal of work. Seven tests were run; at α = 0.05 the chance that at least one of seven true nulls produces P < 0.05 is 1 − 0.957 = 0.30, and the correct single test of the whole question (Lesson 4.1’s goodness of fit, χ2 = 4.18 on 6 df) gives P = 0.65. The effect, had it been real, is 24 days in 25 years, about one rainy day more per year. And the mechanism offered, commuter emissions, is a story told after the fact to fit one bar. Notice also that the analyst ran seven tests and reported the one that stood out; the other six, including the weekday with the largest deficit (Tue, 383), went unmentioned.
The ASA statement, in the lesson’s paraphrase, as a checklist
For whichever report you critique: (1) What model does the P-value measure incompatibility with? (2) Is the report treating P as the probability the hypothesis is true? (3) Is a conclusion resting on whether P crossed a line? (4) Is the effect size stated, with an interval? (5) Were all analyses reported? A report can fail several at once; Report C fails four.
Sources: NOAA NCEI, Global Summary of the Year data search (ncei.noaa.gov); NOAA NCEI Access Data Service (the query used here) (ncei.noaa.gov); American Statistical Association, Statement on Statistical Significance and P-Values (2016), as summarized in the lesson (amstat.org)