THE VR SCHOOL
  • Schools & Districts
    • License the academy
    • Free program for your students
    • Request a demo
  • Catalog
    • AI & Data
    • Cybersecurity & Web
    • Programming
    • Health Science
    • All Guilds
    • All courses
  • Families
  • Accreditation
  • Log In
  • Request a demo
  • Home
  • Courses

RLP Statistics B — Probability Models and Statistical Inference

Curriculum

  • 4 Sections
  • 20 Lessons
  • 10 Weeks
Expand all sectionsCollapse all sections
  • Unit 1: Continuous Distributions and Sampling Distributions
    5
    • 1.1
      Continuous Random Variables and Density Curves
      50 mins
    • 1.2
      The Normal Distribution
      50 mins
    • 1.3
      When Is a Normal Model Appropriate?
      50 mins
    • 1.4
      Sampling Distributions and the Central Limit Theorem
      50 mins
    • 1.5
      Performance Task — Sampling Distribution Simulation Study
      150 mins
  • Unit 2: Estimating with Confidence
    5
    • 2.1
      Estimating a Proportion with Confidence
      50 mins
    • 2.2
      Confidence Intervals for a Mean
      50 mins
    • 2.3
      What Confidence Means, and What It Does Not
      50 mins
    • 2.4
      Comparing Two Groups with Confidence Intervals
      50 mins
    • 2.5
      Performance Task — Estimation Study
      150 mins
  • Unit 3: Testing Claims: Significance Tests
    5
    • 3.1
      The Logic of a Significance Test
      50 mins
    • 3.2
      t-Tests for Means and Paired Data
      50 mins
    • 3.3
      Errors, Power and Significance in the Real World
      50 mins
    • 3.4
      Two-Sample Tests from Experiments and Observational Data
      50 mins
    • 3.5
      Performance Task — Significance Test Investigation
      150 mins
  • Unit 4: Chi-Square, Regression Inference and the Capstone Study
    5
    • 4.1
      Chi-Square Goodness of Fit
      50 mins
    • 4.2
      Chi-Square Tests for Two-Way Tables
      50 mins
    • 4.3
      Inference for the Slope of a Regression Line
      50 mins
    • 4.4
      Choosing the Right Procedure and Writing It Up
      50 mins
    • 4.5
      Capstone — Full Statistical Inference Study
      200 mins

Errors, Power and Significance in the Real World

Unit 3  ·  Statistical Discourse Lab  ·  Lesson 13 of 20

Errors, Power and Significance in the Real World

A2.DS.A.7RLP-STATSB
Student sorts survey dots into four outcome trays, showing false alarms, missed effects, and chance findings.
By the end of this lesson I can…

explain Type I and Type II errors and power, distinguish statistical from practical significance, recognize the multiple-testing problem, and critique a report’s use of P-values.

Instruction

Two ways to be wrong. A test ends in a decision, and a decision can be wrong in two ways:

H0 trueH0 false
Reject H0Type I error (false alarm), probability αCorrect: probability = power
Fail to rejectCorrectType II error (missed effect), probability β = 1 − power

The consequences decide which matters more. For the transit example in Lesson 3.1, a Type I error means announcing that support differs from 60% when it does not. A Type II error means missing a real change in opinion. For a medical screening test, a Type II error can mean missing an illness. Choosing α is a judgment about consequences, not a law of nature.

Power. Power is the probability of rejecting H0 when a specific alternative is true. For the transit test (H0: p = 0.60, n = 150, α = 0.05, two-sided), you reject when p̂ falls below 0.522 or above 0.678. If the true support were 50%, p̂ would fall below 0.522 about 70% of the time: power ≈ 0.70. If the true support were 55%, power would be only about 0.24 at n = 150. It rises to about 0.42 at n = 300 and 0.70 at n = 600. Power increases with sample size, with the size of the true effect, and with α. A study that fails to reject with low power has not shown “no effect”. It was never likely to find one.

Statistical versus practical significance. With enough data, trivial differences become statistically significant. Illustrative: if two groups of 100,000 people each show 50.5% and 50.0% agreement, a two-proportion test gives z ≈ 2.24 and P ≈ 0.025, “significant”, for half a percentage point. Always report the size of the effect, with a confidence interval, and judge whether it matters in context. The St. Louis-Springfield temperature test in Lesson 3.2 had P < 0.001 and a difference under 1°F. Whether 1°F matters is a question about climate and people, not about P.

Multiple testing. If you run 20 independent tests at α = 0.05 and every null hypothesis is true, the chance of at least one Type I error is 1 − 0.9520 ≈ 0.64. Testing many comparisons and reporting only the significant one, sometimes called P-hacking, produces findings that do not replicate. Decide your hypotheses before seeing the data, report every test you ran, and treat a lone significant result among many as a lead to check, not a discovery.

0.05 is not a cliff. In Lesson 3.1, the normal approximation gave P ≈ 0.046 and the exact binomial gave P ≈ 0.055. Treating one as “significant” and the other as “not” would be absurd: the evidence is identical. In 2016 the American Statistical Association published a statement on P-values (Wasserstein and Lazar, The American Statistician). Its principles, paraphrased: a P-value measures how incompatible the data are with a specified model. It does not measure the probability that the hypothesis is true, or the size or importance of an effect. Scientific conclusions should not rest only on whether P passes a threshold, and proper inference requires full reporting and transparency.

In the discourse lab (MP.3) you will critique a report that uses significance language: a news article on a study, or a claim someone makes from the unit data. Ask four questions. What was H0? How big is the effect, with an interval? How many tests were run? Did the design allow the causal language used? Then reread your own critique and ask the same four questions of it.

📖 Read for this lesson

The reading opens over the lesson; close it to come back to exactly where you were. The checks below draw on it.

  • Three Reports That Say ‘Significant’: Power, Effect Size and Multiple Testing, with the Numbers to Check ThemPower table computed for this course; Report A is illustrative; Reports B and C are SLS framings of real data from NOAA NCEI Global Summary of the Year (TAVG, three Missouri stations, 1950-1979 and 1995-2024) and GHCN-Daily (St. Louis PRCP, 2000-2024); retrieved 2026-11-10 · about 6 min

    The power table behind the lesson’s 0.70 and 0.24, and three reports to critique with the numbers to do it: a low-power ‘no effect’, a selectively reported ‘dramatic warming’ on real data, and seven weekday tests with a ‘nearly significant’ Thursday.

Vocabulary in context

  • Type I error — Rejecting H₀ when it is true; its probability is α.
  • Type II error — Failing to reject H₀ when it is false; its probability is β.
  • power — The probability that a test rejects H₀ when a particular alternative is true; 1 − β.
  • practical significance — Whether an effect is large enough to matter in context, judged apart from its P-value.
  • multiple testing — Running many tests, which raises the chance that at least one gives a false positive.

Formative check

Work through these before moving on. They are not graded — they tell you whether the standard below has landed yet.

+50 XP

A study with n = 40 finds P = 0.30 and concludes the treatment has no effect. What is the best critique?

A non-significant result from a low-power study is weak evidence either way. Report the confidence interval to show which effect sizes remain plausible.
Fill in the blank

Running 20 independent tests at α = 0.05 when all nulls are true gives P(at least one false positive) = 1 − 0.95^20 ≈ .

Myth or Fact?

A P-value of 0.001 means the effect is large and important.

With a large sample, tiny effects give tiny P-values. Size and importance come from the estimate and its interval, judged in context.

Record (or write) a 90-second critique of a report that uses the words ‘significant’ or ‘no effect’ (a news story about a study, or one of the three reports in this lesson’s reading). Answer: What was H₀? How large is the effect, and is there an interval? How many tests were run? Is the causal language justified by the design? Then add one correction or addition that a careful reader (a parent or guardian, if you like) might make to your critique.

0 words
Quick self-check

How confident are you that you can judge significance-test claims using errors, power, effect size and multiple testing?

Not yetVery confident

Practice

Work these on paper or in your notebook, then open Check your answer. Aim for all of Fluency and Application; try at least one Challenge.

Printable version: this unit’s practice workbook (PDF)

Fluency

Build speed and accuracy with the core skill.

  1. A test of H₀: ‘the new filter does not reduce smoke particles’ rejects H₀, but in fact the filter does nothing. Which error occurred?
    Check your answer
    Answer: A Type I error.
  2. A test fails to reject H₀: ‘this batch of lettuce is not contaminated’, but the batch is contaminated. Which error, and why does it matter here?
    Check your answer
    Answer: A Type II error; contaminated food reaches people.
  3. What is the probability of a Type I error when H₀ is true and α = 0.05?
    Check your answer
    Answer: 0.05.
  4. Name three ways to increase the power of a test.
    Check your answer
    Answer: Increase the sample size, increase α, or study a larger true effect (reduce variability also helps).
  5. Ten independent tests at α = 0.05, all nulls true: find P(at least one false positive).
    Check your answer
    Answer: 1 − 0.95¹⁰ ≈ 0.401.
  6. True or false: a very small P-value means the effect is large.
    Check your answer
    Answer: False.
    Large samples make small effects significant.

Application

Use the skill in context. Show your reasoning.

  1. A two-sided one-proportion test uses H₀: p = 0.5 and α = 0.05. If the true p is 0.6, find the power for n = 100, 200 and 400.
    Check your answer
    Answer: About 0.52, 0.81 and 0.98.
    Find the rejection cutoffs under H₀, then the probability of landing beyond them when p = 0.6.
  2. (Illustrative) A survey of 100,000 people in each of two regions finds 50.6% and 50.2% agreeing with a statement. The two-proportion z is about 2.53 and P ≈ 0.011. Write a sentence that reports both statistical and practical significance.
    Check your answer
    Answer: The difference of 0.4 percentage points is statistically significant (P ≈ 0.011) only because the samples are huge; a gap this small is unlikely to matter in practice.
  3. A study tests 20 foods for a link to headaches and reports only that chocolate was ‘significantly linked’ (P = 0.03). Critique the report.
    Check your answer
    Answer: With 20 tests at α = 0.05, about one false positive is expected even if no food matters (P of at least one ≈ 0.64). The chocolate result should be reported alongside all 20 tests and treated as a lead to check in a new study.

Challenge

Stretch problems. Expect to think before you write.

  1. A small study (n = 12 per group) finds P = 0.30 and concludes ‘the tutoring program has no effect.’ Its 95% interval for the effect runs from −4 to +11 points. What should the conclusion say instead?
    Check your answer
    Answer: That the study is inconclusive: the data are consistent with no effect, a small negative effect, or a large positive effect of up to 11 points. A larger study is needed.

Review

Keep earlier skills sharp.

  1. Test H0: p = 0.4 against Ha: p > 0.4, given 74 successes in a random sample of 200. Find the test statistic and the P-value.
    Check your answer
    Answer: p̂ = 0.370; z ≈ −0.87; P-value ≈ 0.8068. At α = 0.05, fail to reject H0: there is not convincing evidence that the population proportion is greater than 0.4.
    Use p0, not p̂, in the standard error of a test. State the conclusion in context: reject or fail to reject H0, never “accept”.
  2. (Unit 2) Why is an interval for the effect more informative than a P-value alone?
    Check your answer
    Answer: It shows the size of the effect and which values remain plausible, not just whether 0 is ruled out.

Missouri Learning Standards — Mathematics (data and probability) addressed: A2.DS.A.7, A2.DS.A.2, MP.3, MP.6

Extends beyond Missouri Statistics (enrichment): content Missouri places after Statistics

Instructional focus: Critical evaluation of significance-test reasoning in reports

Reading

Three Reports That Say ‘Significant’: Power, Effect Size and Multiple Testing, with the Numbers to Check Them

Power table computed for this course; Report A is illustrative; Reports B and C are SLS framings of real data from NOAA NCEI Global Summary of the Year (TAVG, three Missouri stations, 1950-1979 and 1995-2024) and GHCN-Daily (St. Louis PRCP, 2000-2024); retrieved 2026-11-10 · SLS original, with public domain NOAA data

Why this reading. The discourse task says to critique “a report that uses the words significant or no effect”, and offers a news story you would have to find or a claim you may not have come across. Three reports are supplied here, each with the numbers a critique needs. Report A is constructed for the lesson and labeled so. Reports B and C are written in the voice of a careless analyst about real NOAA data from this course; the framing is SLS original, the numbers are not. Before them is the power table behind the lesson’s figures, so “the study had low power” can be a computed statement rather than a phrase.

Table 1. Power of the two-sided transit test (H0: p = 0.60, α = 0.05)

Reject when p̂ falls outside 0.60 ± 1.96√(0.6 × 0.4/n). Power is the probability of that under a stated true p, using the normal model with SD √(p(1 − p)/n). Worked for n = 150, p = 0.55: the lower cutoff is 0.5216; SD = √(0.55 × 0.45/150) = 0.0406; z = (0.5216 − 0.55)/0.0406 = −0.70; Φ(−0.70) = 0.242, plus a negligible upper-tail piece, gives 0.24.

nReject if p̂ below / abovetrue p = 0.500.550.580.620.65
1500.5216 / 0.67840.700.240.080.080.23
3000.5446 / 0.65540.940.430.110.110.42
6000.5608 / 0.63921.000.700.170.170.71
12000.5723 / 0.62771.000.940.290.290.95

The lesson’s 0.70, 0.24, 0.42 and 0.70 are the first row’s first two entries and the 0.55 column at n = 300 and 600. Read across a row: power grows with the size of the true effect. Read down a column: it grows with n. Read the 0.58 column: a two-point difference from 0.60 is nearly invisible to a survey of 150 and still missed more often than not at n = 1,200. A survey that “found no significant change” at n = 150 has ruled out almost nothing.

Report A (illustrative; constructed for this lesson)

“A school piloted an after-school tutoring program. Twenty students who enrolled and twenty who did not were tested in September and again in January. The tutored group gained an average of 5.1 points on the district assessment; the comparison group gained 3.1. The difference was not statistically significant (P = 0.30). The program has no effect on achievement and will not be renewed.” Supplementary numbers: the standard deviation of the gains was 6.0 points in each group.

What a critique can compute. SE of the difference = √(6.02/20 + 6.02/20) = 1.897; t = 2.0/1.897 = 1.05; df ≈ 38; P = 0.30. The 95% interval for the difference: 2.0 ± 2.024(1.897) = -1.8 to 5.8 points. It includes zero, and it also includes a gain of five points, which would be a large effect for a program of this size. Power: if the true effect were 3 points, this design would detect it 34% of the time; if 4 points, 54%; if 5, 73%. The study was never likely to find the effect it was looking for. Then the design: students enrolled; nothing was assigned. Ask the lesson’s four questions in order.

Report B (SLS framing; real NOAA data)

“Our analysis of federal climate records finds that St. Louis has warmed dramatically: the mean annual temperature at Lambert Airport in 1995–2024 was 57.89°F against 55.32°F in 1950–1979, a highly statistically significant increase (t = 6.52, P < 0.0001). Climate change has raised Missouri’s temperature by two and a half degrees.”

The numbers are right; the Lesson 3.4 mini-lecture has them. Here is what the analyst had and did not print: the same comparison at all three stations the course uses.

Stationn (1950–79 / 1995–2024)Mean (SD), earlierMean (SD), laterDifference °FtdfP95% interval
St. Louis30 / 3055.32 (1.36)57.89 (1.68)2.576.5256< 0.00011.78 to 3.36
Springfield30 / 3055.66 (1.30)57.08 (1.52)1.423.89570.00030.69 to 2.15
Joplin30 / 3057.39 (1.39)58.77 (1.36)1.383.87580.00030.67 to 2.09

Questions for the critique. The St. Louis effect is 2.57°F with an interval of 1.78 to 3.36: is that “two and a half degrees” or “somewhere between about 1.8 and 3.4”? Why report one station when three were available? St. Louis shows the largest difference of the three, and Springfield and Joplin show about half of it; a reader given only the St. Louis row cannot know which was the most dramatic. A station inside a large metropolitan area can warm for local reasons (buildings, pavement, equipment changes) as well as regional ones. And “Missouri’s temperature” is a claim about a state from airports, none of which was chosen at random. Which of the ASA checklist items below does Report B fail?

Report C (SLS framing; real NOAA data)

“Analyzing 25 years of St. Louis rainfall, we tested each day of the week separately against the others. Thursday stands out with 431 rainy days against an expected 407, a nearly significant excess (P = 0.10), and Tuesday shows a corresponding deficit. We suggest that midweek atmospheric patterns, possibly linked to weekday commuter emissions, favor Thursday storms.”

The counts are the Lesson 4.1 data. Test them the analyst’s way, one day at a time: under H0 each day expects 2,846/7 = 406.6 rainy days with SD √(2,846 × (1/7)(6/7)) = 18.67.

DayRainy daysz against 406.6Two-sided P
Mon408+0.080.94
Tue383−1.260.21
Wed426+1.040.30
Thu431+1.310.19
Fri401−0.300.77
Sat397−0.510.61
Sun400−0.350.72

Thursday’s two-sided P is 0.19; the analyst’s 0.10 is a one-sided P, with the direction chosen after seeing which day was high. “Nearly significant” is doing a great deal of work. Seven tests were run; at α = 0.05 the chance that at least one of seven true nulls produces P < 0.05 is 1 − 0.957 = 0.30, and the correct single test of the whole question (Lesson 4.1’s goodness of fit, χ2 = 4.18 on 6 df) gives P = 0.65. The effect, had it been real, is 24 days in 25 years, about one rainy day more per year. And the mechanism offered, commuter emissions, is a story told after the fact to fit one bar. Notice also that the analyst ran seven tests and reported the one that stood out; the other six, including the weekday with the largest deficit (Tue, 383), went unmentioned.

The ASA statement, in the lesson’s paraphrase, as a checklist

For whichever report you critique: (1) What model does the P-value measure incompatibility with? (2) Is the report treating P as the probability the hypothesis is true? (3) Is a conclusion resting on whether P crossed a line? (4) Is the effect size stated, with an interval? (5) Were all analyses reported? A report can fail several at once; Report C fails four.

Sources: NOAA NCEI, Global Summary of the Year data search (ncei.noaa.gov); NOAA NCEI Access Data Service (the query used here) (ncei.noaa.gov); American Statistical Association, Statement on Statistical Significance and P-Values (2016), as summarized in the lesson (amstat.org)

Up NextComplete & Continue →

Leave a Reply Cancel reply

t-Tests for Means and Paired Data
Prev
Two-Sample Tests from Experiments and Observational Data
Next

© 2026 The VR School | Career content by Wiley | Powered by Sejal Learning Systems.

Join free · Accreditation · Privacy Policy · Terms of Service

Modal title

Main Content