Guide 4 · Mathematical modeling

Test the Model

A model is a deliberate simplification. The useful question is never "is it true?" but "is it good enough for this decision, and where does it break?"

Steel caliper measuring a small wooden block, with the jaw markings in sharp focus — Illustrative photograph — not measured evidence.

What a model is — and is not

A mathematical model translates a situation into equations: how a population grows, how a drug concentration falls, how house prices relate to floor area. Every model keeps some features of reality and throws the rest away on purpose. The statistician George Box put it memorably in the 1970s: all models are wrong, but some are useful.

Deterministic models give one output for each input, like a compound-interest formula. Stochastic models include randomness and produce a distribution of possible outcomes, like a queue simulation in which customers arrive at random times. Choosing between them is itself a modeling decision: if the randomness matters for your question — say, how often a queue grows beyond 20 people — a deterministic average will hide exactly what you need.

The first discipline of modeling is to write the assumptions down. "Growth rate stays constant." "Measurements are independent." "The relationship is linear across the range we care about." A model is only as trustworthy as the least checked of these sentences.

Worked example

Doubling time — and when it stops applying

Something growing by a steady 7% per period doubles in about ln 2 ÷ ln 1.07 ≈ 10.2 periods. The "rule of 70" gives a quick estimate: 70 ÷ 7 = 10 periods.

That is a useful estimate while conditions stay constant. Real populations, user bases and infections slow down as they approach limits: food, market size, the number of people still susceptible. An exponential model fitted to the early part of a logistic curve fits beautifully — and then overshoots dramatically. Early data alone often cannot tell the two models apart, which is why the assumption about limits has to come from knowledge of the system, not from the curve.

Useful estimate: "at the current rate, this doubles in about 10 periods". Defensible conclusion: a projection stated with its assumption ("if the rate holds") and a scenario in which it does not.

Fitting a line and reading the leftovers

Linear regression chooses the line that makes the sum of squared vertical distances — the residuals — as small as possible. That is the method of least squares. The fitted line is a summary; the residuals are where the checking happens.

Plot residuals against fitted values. A shapeless cloud around zero is what you want. A curve means the relationship is not linear. A funnel that widens means the variability grows with the size of the prediction, so intervals from the simple model will be too narrow at one end. One point far from the rest may be steering the whole fit; Anscombe's quartet, from our description guide, shows how.

Be especially careful with extrapolation. A line fitted to houses of 50 to 200 square meters says nothing reliable about a 900-square-meter mansion. The data never visited that region, so the model has no evidence there.

Overfitting

A model with enough parameters can pass through every training point, including the noise. It then predicts new data badly. The cure is to judge models on data they have not seen: hold back a test set, or use cross-validation, which rotates which part of the data is held out and averages the error across rounds. If performance drops sharply out of sample, the model learned the sample, not the system.

Sensitivity and simulation

Sensitivity analysis changes one input at a time — growth rate, cost, failure probability — and watches the output. If a decision flips when an uncertain input moves slightly, that input deserves better data. Monte Carlo simulation, developed by Stanisław Ulam and John von Neumann at Los Alamos in the 1940s, runs a model thousands of times with random inputs to show the full range of outcomes rather than one number.

Choosing a model type

Common model families compared (scroll sideways on small screens)
Model	Typical use	Key assumption	How to check it
Linear regression	Relating an outcome to predictors	Straight-line relationship, independent errors with roughly constant spread	Residual plots; out-of-sample error
Exponential growth	Early-stage growth or decay	Constant proportional rate	Plot the logarithm: should be a straight line
Logistic growth	Growth approaching a limit	A stable carrying capacity	Does the slowdown match known limits?
Monte Carlo simulation	Complex systems with random inputs	Input distributions are realistic	Compare simulated with historical outcomes

A model-checking routine

Write down every assumption in plain words.
Fit the model; plot the residuals.
Test on data the model has not seen.
Vary the uncertain inputs and note which ones change the decision.
Report the range of outcomes, not just the central prediction.

Sources

Box (1976), "Science and Statistics", Journal of the American Statistical Association; NIST/SEMATECH e-Handbook, process modeling chapter; OpenIntro Statistics, chapters 8–9.

Questions? See the Answers page.