Pals Polar Waffl
Guide 3 · Statistical inference

Infer with Care

Inference is the step from the data you have to the population you care about. It is powerful only when you can say how the sample was chosen and how much it might vary.

Handwritten confidence interval calculation on squared paper, pencil resting beside it, shallow focus
Illustrative photograph — not measured evidence.

Population, sample, parameter, statistic

The population is everyone or everything you want to learn about; a parameter is a fixed but usually unknown number describing it, such as the true share of voters who support a policy. A sample is the part you actually observe, and a statistic is a number computed from it, such as the share of supporters among 1,000 respondents.

Different random samples give different statistics. That sampling variability is not an error to be embarrassed about; it is the thing inference measures. The central limit theorem explains why it is so predictable: for reasonably large random samples, sample means and proportions are approximately normally distributed around the true value, whatever the shape of the population.

Random beats big

Variability shrinks as samples grow, but bias does not. In 1936 the Literary Digest magazine collected millions of mail-in responses and predicted that Alf Landon would beat Franklin Roosevelt; Roosevelt won by a landslide. The mailing lists and the people who chose to reply did not represent the electorate. A smaller, well-designed random sample would have done better.

Worked example

A 95% confidence interval for a survey result

A random sample of 1,000 adults finds 420 who favor a proposal, so p̂ = 0.42.

SE = √(0.42 × 0.58 ÷ 1000) ≈ 0.0156 0.42 ± 1.96 × 0.0156 ≈ 0.42 ± 0.031 → 38.9% to 45.1%

The margin of error is about ±3.1 percentage points. The 95% describes the method: if we repeated the survey many times, about 95% of intervals made this way would capture the true population share. It does not mean there is a 95% probability that this particular interval is correct, and it accounts only for random sampling error — not for unclear question wording, people who refused to answer, or a sampling frame that missed part of the population.

Useful estimate: support is probably around 39–45%. Defensible conclusion: "a minority of adults in this population favored the proposal when surveyed", provided the sample was genuinely random and non-response was low.

The normal curve, labeled

Standard normal curve with the 68–95–99.7 rule A bell-shaped curve centered on the mean. About 68% of values lie within one standard deviation of the mean, about 95% within two, and about 99.7% within three. 68%μ = mean, σ = standard deviation95%99.7% μ−3σμ−2σμ−σμμ+σμ+2σμ+3σ
Theoretical shape of a normal distribution; percentages are the standard 68–95–99.7 approximations, not measured data. The 1.96 in the interval above is the more exact multiplier for 95%.

Hypothesis tests and p-values

A hypothesis test asks: if a specific "null" model were true — say, a new teaching method makes no difference — how surprising would data like ours be? The p-value is the probability, computed under that model, of a result at least as extreme as the one observed.

In 2016 the American Statistical Association published a statement on p-values. Among its principles: a p-value does not measure the probability that the hypothesis is true; scientific conclusions should not rest only on whether it passes a threshold such as 0.05; it does not measure the size or importance of an effect; and proper inference requires full reporting and transparency.

Correlation is not causation

Observational data can mislead through confounders — a third factor driving both variables. Randomized experiments break that link by assigning treatments by chance.

From estimate to defensible conclusion

What each kind of result can support (scroll sideways on small screens)
You haveA useful estimateA defensible conclusion needs
A convenience sample (volunteers, online poll)Ideas worth testingA random or carefully weighted sample of the target population
A random sample with a confidence intervalA plausible range for the parameterLow non-response and clearly worded questions
A small p-valueEvidence against the null modelThe effect size, the number of tests run, and replication
A strong correlation in observational dataA possible relationshipControl of confounders or a randomized design

Questions to ask any inferential claim

  • Who was sampled, how, and who was missed?
  • How large is the effect, and how wide is the interval?
  • How many comparisons were made before this one was reported?
  • Was treatment assigned at random? If not, what else differs between groups?
  • Has anyone repeated the result?

Sources: Wasserstein & Lazar (2016), The ASA Statement on p-Values; Freedman, Pisani & Purves, Statistics; OpenIntro Statistics, chapters 5–6.

Unsure which test fits?

Describe your data and question and we'll explain the options and their assumptions.

Then continue to Test the Model.

Chat on WhatsApp