Statistics is a framework for evaluating what we know, what we think we know, and how confident we are that we know it. The core mechanism we rely on — in science and in life — is prediction: seeing how well our understanding of the world predicts something we have not seen.
This part is close to what you did before. You are going to choose one number to be the best possible prediction of human size.
nh <- read.csv("data/clean/nhanes_adults.csv")y <- nh$Weightguess <- 60.0mean(guess - y) # which way you leansum(y > guess); sum(y < guess)
One number is usually not enough. Not all people are identical, which is why the error above is so large: your one number does a bad job of fitting the data.
How do we make it do better? Usually there are groups, or factors, we are not accounting for — in any model there are always things we do not see. Here we separate the 7,414 adults by biological sex and compare the body mass of females and males.
That needs two numbers. You could fit a separate body mass for females and for males. Instead we use one number for females and an adjustment for males. You could do it the other way round, and it would work just as well.
nh <- read.csv("data/clean/nhanes_adults.csv")is_male <- as.numeric(nh$Gender == "male")base <- 60.0extra <- 0.0guess <- base + extra * is_maletapply(guess - nh$Weight, is_male, mean)# the same two numbers, found for youcoef(lm(Weight ~ is_male, data = nh))
The bigger the data set, the easier it is to find a clear difference: with a lot of data, even a small difference is relatively easy to detect. With a smaller sample you get less reliable — and sometimes counterintuitive — results. The smaller the sample, the more likely you are to find something counterintuitive, or even flatly wrong in direction.
Here you estimate the female number and the male adjustment in a small sample, and find a case where the adjustment for males comes out negative — males lighter than females. In the full data set that does not happen. In a small sample it can.
Think about how many people you want in each group for that to happen. Weighing handful after handful blindly will take a long time; look at the graph before you fit, and stop when you see a handful you suspect will hold.
nh <- read.csv("data/clean/nhanes_adults.csv")n <- 6f <- sample(nh$Weight[nh$Gender == "female"], n)m <- sample(nh$Weight[nh$Gender == "male"], n)base <- 70.0extra <- 0.0mean(base - f); mean(base + extra - m)# the same two numbers, and the same band, found for yout.test(m, f, var.equal = TRUE)# how wide the band is, for any ns <- sd(nh$Weight[nh$Gender == "male"])2 * s * sqrt(2 / n)
So far we have predicted within one data set, or with one extra factor. Another key aspect of statistics — and of understanding the world — is how well a model, a way of predicting one system, works in a different system entirely.
First, fit a model of growth to my older son Beren's real growth data. The setup is simple: he weighed some amount at age two, and he grows at some rate over time. Age two is the baseline. Find the weight at age two and the growth rate that best explain the rest of his growth.
k <- read.csv("data/clean/kids_growth.csv")k$kg <- ifelse(k$units == "g", k$value / 1000, k$value)d <- subset(k, measure == "mass" & age_years >= 1)base <- 12.0grow <- 0.00extra <- 0.0guess <- base + grow * (d$age_years - 2) + extra * (d$kid == "cyrus")tapply(d$kg - guess, d$kid, mean)# the same three numbers, found for youcoef(lm(kg ~ I(age_years - 2) + kid, data = d))
The Lord of the Rings is a trilogy of books by J. R. R. Tolkien, turned into three films by Peter Jackson. The films are very faithful to the books, often using quotes from them verbatim, and the order in which those quotes appear on screen is very close to the order in which they appear on the page.
Across the bottom: the minute of the film a quote is spoken. Up the side: the page of its own book the quote falls on. Were the three films made in the same way — do they begin in the same place, and do they move through their books at the same rate? Or are there fundamental differences in how the three books were adapted?
q <- read.csv("data/clean/lotr_quotes.csv")# one opening page, one pacelm(page_total ~ minute, data = q)# one for each film, opening page onlylm(page_total ~ minute + film, data = q)# one of each for each filmfit <- lm(page_total ~ minute * film, data = q)mean(abs(residuals(fit)))