You built one test in Lesson 5. Here you run it four times on the same kind of data — and watch it hand back four different answers. Each stage: commit a prediction first, then watch which input changed the verdict.
Three groups. Two pairwise tests are already on the screen. The third one isn't. Predict it before the simulator does.
y ~ β · indicator — fit to two groups at a time. The slope β is the mean difference between the two groups. The empirical P is the probability of a shuffled β at least this far from zero. Every "trap" below is one way that exact β + empirical-P pair gives a verdict that doesn't say what it looks like it says.
Three groups: Low, Middle, High. Defaults: Low vs. Middle and Middle vs. High both return P > 0.20. Commit to a prediction for P_LH (the Low vs. High comparison) before unlocking the simulator and finding out.
set.seed(42)n <- 30sigma <- 1.00muL <- -0.40muH <- 0.40L <- rnorm(n, muL, sigma)M <- rnorm(n, 0, sigma)H <- rnorm(n, muH, sigma)empP <- function(a, b) { y <- c(a, b); g <- c(rep(0, length(a)), rep(1, length(b))) d <- coef(lm(y ~ g))[2] dN <- replicate(1000, coef(lm(y ~ sample(g)))[2]) mean(abs(dN) >= abs(d))}empP(L, M); empP(M, H); empP(L, H)
Identical biology. Two sample sizes (n = 30 and n = 300). Run both. Read each verdict.
Two experiments are running side-by-side with the same true mean difference. The left one has 30 individuals per group; the right has 300. Move the true-difference slider and watch what happens to each verdict.
set.seed(42)delta <- 0.30sigma <- 1.00runP <- function(n) { y <- c(rnorm(n, 0, sigma), rnorm(n, delta, sigma)) g <- c(rep(0, n), rep(1, n)) d <- coef(lm(y ~ g))[2] dN <- replicate(1000, coef(lm(y ~ sample(g)))[2]) mean(abs(dN) >= abs(d))}runP(30); runP(300)
Two experiments side by side. Decide which has the bigger underlying effect, and which one the test calls "clearly different from zero."
Look at the two experiments below. Two labs ran them, on the same trait in two different species. Decide which one has the bigger underlying effect, and which one the test signals as "clearly different from zero." See if they agree.
set.seed(42)# Experiment A: small effect, clean data, lots of samplesA0 <- rnorm(200, 0, 0.5); A1 <- rnorm(200, -0.3, 0.5)# Experiment B: huge effect, messy data, tiny samplesB0 <- rnorm(12, 0, 4.0); B1 <- rnorm(12, -1.5, 4.0)empP(A0, A1); empP(B0, B1) # empP() defined in Stage A code
Two groups with the same mean. One slider controls the spread of the second. Run the test.
Two groups, same mean. One slider controls how spread out the second group is. Watch the histograms. Watch the P-value.
set.seed(42)n <- 50sigma_B <- 1.00A <- rnorm(n, 0, 1)B <- rnorm(n, 0, sigma_B)empP(A, B)
Same machine — empirical-P of a regression slope — applied to weekend-level data on movie attendance and crime. The slope is clearly negative. The interpretation is wrong.
Each point is one weekend (synthetic, faithful to the structure of Dahl & DellaVigna 2009). x = total movie attendance that weekend, in millions. y = change in violent crime rate that weekend, %. Color = whether the dominant new release was a "violent" movie (think action / superhero / horror) or not.
Fit the slope on all the points. Then color by movie type. Then ask: is "violent movies cause less crime" what the data actually said?
# Pooled slope: violent-movie attendance → fewer crimespooled <- lm(crime ~ attendance, data = weekends)# Stratify by movie type — within-stratum slopes near zerostrat <- lm(crime ~ attendance * violent, data = weekends)summary(pooled)$coefficients[2, ]summary(strat)$coefficients
DellaVigna's reading: "young men in theaters" is the unmeasured cause. "Violent movie" is a noisy proxy for it (violent releases skew toward young-male audiences). Build it below: add type → attendance (violent movies draw bigger weekends) and type → crime (those audiences are off the streets). Watch the pooled scatter mimic the empirical pattern even though "type" never directly affects "crime" through violence per se.