What statistics is
A machine fills cartons that must hold at least a litre. Filling varies, so the target is set at 1.10 litres: a target of exactly one litre would leave about half the cartons short. Ten cartons are measured. The mean is 1.0512 litres. Tomorrow’s ten would give a different mean. Statistics is the discipline that says what can still be claimed from numbers that would have come out differently, which claims the way the numbers were collected will not support, and how to check the claim.
That is the whole of this site. Probability is the language in which “another ten cartons” is a precise sentence. Inference is the move from the ten numbers back to the setting of the machine. The later parts are what changes when the response is a count, a curve, a label, a person measured four times, or a queue length.
Read it straight through if the subject is new, then the five chapters named on the main page. If you are already in the middle of a topic, use the contents: each part says what question it answers and which chapters carry it. The order here follows the questions. The topics list is grouped by part, and that grouping is a different path.
Ten cartons
The ten volumes, in litres, are 1.035, 1.019, 1.068, 1.084, 1.077, 0.998, 1.041, 1.037, 1.051 and 1.102. One of them, 0.998, is just under a litre. The mean is \overline x=1.0512, the median is 1.0460, and the bias-corrected standard deviation is s=0.0317. The standard error of the mean, s/\sqrt{10}, is 0.0100. Sorted, the lower quartile is 1.035 and the upper quartile is 1.077.
None of those summaries answers the question that matters, which is whether the machine is still filling to 1.10 litres on average. Every one of them would move if another ten cartons were taken. The numbers need a model of how they came about.
The move has four steps. There is a question about the world, and a table. There is a probability model whose unknown constants are what the question asks about. There is a calculation inside that model. There is a sentence back in the language of the question, together with a clear account of what the design does not let you add. For the cartons the model is ten independent normal volumes, mean \mu and variance \sigma^2, both unknown. Setting \mu equal to 1.0512 would describe this batch and no other. The parameter stays unknown, and the data are used to learn about it.
Statistical inference is that argument, and it is where the carton numbers are worked. The same page names three branches. Data analysis summarises the table so the structure is visible. Classical methods judge a procedure by what it does over repeated samples, with the parameter fixed. Bayesian methods put a distribution on the parameter and update it. A further tradition, fiducial inference, is named on the Bayesian foundations page and is not developed.
What the design will support
The model is chosen after a decision about the study, and the study decides which sentences are available. In an observational study the investigator does not assign the conditions being compared. In a designed experiment some columns are controlled, often by random assignment, and those columns stay put if the measurements are repeated. After a large enough randomisation the treatment groups agree, before treatment, on the distribution of every variable, measured or not. A difference that then appears in the response is evidence about the treatments.
A confounding variable moves both the explanatory variable and the response, so an association can appear, vanish, or reverse when the confounder is taken into account. Taller people tend to have shorter hair; within each sex the relationship may be absent, because sex drives both. Attributes that cannot be assigned, such as age, support an association and, by themselves, not a causal sentence. Blinding, and a placebo that resembles the active treatment, keep the knowledge of the assignment from leaking into the response.
Alice asks whether caffeinated cola raises the heart rate more than decaffeinated cola. She measures twenty friends, assigns the drink at random, ten to each, and uses diet cola against decaf diet cola so that sugar does not ride along with the caffeine. The mean changes are 15.8 and 5.1 beats per minute, a difference of 10.7. Because the drink was assigned, a later test can read that difference as an effect for people like those friends. It cannot read it as an effect for everyone.
How the units were chosen matters in the same way. A simple random sample gives every subset of the right size the same chance. Stratified sampling draws inside subgroups so that each subgroup appears. Cluster sampling draws a few groups and measures inside them, which is cheaper and sees less of the variation between groups. Studies, data, and evidence is this chapter: the design, the confounding, Alice’s trial, and the three sampling schemes.
The sample, before a model
Description comes before a fit. The mean is the average, and it is pulled by a long tail; the median is not. A spread, a quartile, and a plot show shape and odd points that the average hides. A histogram that looks bell-shaped is one reason the carton model is normal, and a histogram that does not is a reason to refuse that model.
The same habit applied to 299 patients admitted after heart failure separates a rate from a shape. Anaemia and death differ by six percentage points, and a chi-squared test of that table gives p=0.25. Serum creatinine in the same patients runs from 0.50 to 9.40, with mean 1.39 and median 1.10. A rate that a test does not separate, and a long right tail, are different facts about the same cohort. Descriptive statistics is the chapter for the summaries and the plots. The commands that hold a table, name its types, and draw it are in Using R, taken up again below.
Probability
“Another ten cartons” needs a model of an experiment whose outcome is not fixed in advance and can still be described exactly. The model has three pieces. The sample space is the set of outcomes under discussion: the two faces of a pair of dice, or a lifetime as a point of [0,\infty). An event is a subset. A probability gives each event a number, with the whole space equal to 1 and with disjoint events adding.
On a finite set of equally likely outcomes the probability is a ratio of counts. Twenty-three people, with birthdays treated as independent and uniform across 365 days, already share a birthday with probability just over one half. There are many pairs, and each pair is only mildly unlikely. Conditioning replaces the sample space by what was observed and renormalises. Three factories supply a chip with probabilities 0.15, 0.35 and 0.50, and their defective rates are 0.01, 0.05 and 0.02. The chance a chip is defective at all is 0.029. Given that it is defective, the chance it came from the middle factory is about 0.60, and from the largest factory about 0.34: the higher defective rate outweighs the smaller share. That is Bayes’ rule for events. The same algebra, read for densities, is the update used when the unknown is a parameter.
Independence is the modelling assumption that lets a joint probability factor into a product. For two events of positive probability it is the same as saying that conditioning on one leaves the other unchanged. Pairwise products are not enough once three or more events are in play.
A random variable is a number attached to the outcome, such as the sum of two dice. Its distribution is everything a question about that number still needs: a mass function when the values can be listed, a density when they lie on a continuum. A density is not a probability. The probability of a small interval is about the density times the length, and a single point has probability zero. Expectation is the centre of mass. It is linear whether or not the pieces are independent, so a count of overlapping events still has mean equal to the sum of the probabilities. Variance is the mean squared distance from the centre.
The named families are experiments written as formulas. A binomial count is the number of successes in independent trials; twenty items with defective rate 0.05 have mean one defective, and the chance of at most one defective in the box is 0.736. The geometric law on these pages is the trial of the first success, with mean 1/p. A Poisson count is the limit of many rare trials with a steady expected number, and its variance equals its mean. Counts that vary more than their mean are called overdispersed, and they come back when a mean count is modelled on covariates. An exponential waiting time is memoryless: a bulb with mean life 1000 hours that has already burned for 1000 hours still has probability e^{-0.5}=0.607 of lasting another 500 hours, the same as a new bulb. The normal law is fixed by a mean and a variance, and the second parameter here is the variance, so \mathrm{N}(100,16^2) has standard deviation 16. About 68\% of a normal law lies within one standard deviation of its mean, which is why “estimate plus or minus one standard error” is a weak interval. Gamma is shape and rate. The exponential is the gamma of shape 1. A chi-squared law on k degrees of freedom is the gamma of shape k/2 and rate 1/2.
Those parameterisations are the ones on the common distributions page, which includes the uniform law on an interval and the hypergeometric law for a sample drawn without replacement. The hypergeometric variance is smaller than the with-replacement binomial variance by the factor (N-n)/(N-1).
Random experiments and probability builds the sample space, the events, the conditioning, and the independence. Random variables and distributions is the distribution, the expectation, and the families. Simulation runs the model forwards: uniform draws on [0,1] are turned into a specified law by reading the distribution function backwards, or by keeping the draws that fall under the target density.
A system of unreliable parts is a structure on events. Series needs every part. Parallel needs one. A bridge needs a path through a cross link that neither picture can draw. With each of five components working independently with probability 0.9, the bridge works with probability 0.9785. Reliability turns the structure into a probability and then into a lifetime. A Weibull hazard can rise with age. An exponential hazard cannot.
Several numbers, and a limit
Two measurements on the same unit have a joint distribution. That joint law fixes each separate law, the probability of a region, and the conditional law of one variable given the other. Independence is factorisation of the joint. Covariance is the dependence that enters the variance of a sum: the variance of a sum is the sum of the variances plus twice the covariance, so ignoring dependence misstates the spread of a total. Jointly normal coordinates with covariance zero are independent. That is a property of the normal law, not of dependence in general.
A smooth function of a random vector changes the density by the absolute value of the Jacobian. The sample mean is the function that matters most. If the mean exists, the strong law says the average settles on it with probability one. The central limit theorem says the fluctuations of that average, scaled by \sqrt n, settle on a normal law when the variance is finite and positive. Those two theorems are why a mean of ten cartons is less wild than one carton, and why a normal approximation to a count or a mean becomes an honest calculation as the sample grows. They are also why a procedure’s long-run behaviour can be calculated.
A Markov chain is a process with a short memory: the next state depends on the past only through the present. On a finite set of states the whole process is one transition matrix P. The distribution after t steps is the corresponding row of P^t. A stationary distribution satisfies \pi=\pi P. When the chain is one closed aperiodic class, every start forgets itself. A deterministic two-cycle can have a stationary distribution while P^t keeps oscillating, so “stationary” and “forgets its start” are different sentences. The Markov chain laboratory draws that distinction with a matrix you can edit.
Joint distributions, functions of random variables and limit theorems, and Markov chains are these three chapters. The stochastic-process pages take the path seriously, and they can wait until a single sample’s parameter is in hand.
From the sample back to the parameter
A probability model says what a sample would look like if the parameter were known. The sample is in hand and the parameter is not. A statistic is a function of the sample that does not use the unknown parameter. The sample mean and the bias-corrected variance
\overline X=\frac1n\sum_{i=1}^n X_i, \qquad s^2=\frac1{n-1}\sum_{i=1}^n (X_i-\overline X)^2
are the two used constantly. For any random sample with mean \mu and variance \sigma^2, the mean of \overline X is \mu, its variance is \sigma^2/n, and the mean of s^2 is \sigma^2. No normality is required for those three facts. Dividing the sum of squares by n instead of n-1 produces a variance estimator whose mean is ((n-1)/n)\sigma^2, a little low on average.
An estimator is that function, thought of before the sample is drawn. The number it produces on the data in hand is the estimate. Bias is the difference between the estimator’s mean and the parameter. Mean squared error is variance plus squared bias, and it is the comparison among estimators one already has. Likelihood is the joint density with the data fixed and the parameter left free. It is not a density in the parameter. Maximum likelihood for a normal variance divides by n and is biased; s^2 is unbiased. At n=10 the mean squared errors are 0.222\,\sigma^4 for the unbiased estimator and 0.190\,\sigma^4 for the biased one. Unbiased is a restriction that makes a competition possible. It is not a synonym for accurate.
Estimation is bias, variance, mean squared error, and the likelihood.
How sharp an estimator can be
Mean squared error compares estimators in hand. It does not say whether a better unbiased estimator is still possible. The score, the derivative of the log-likelihood, has mean zero under regularity conditions that let the derivative pass inside the integral, and the Fisher information is the variance of the score. The Cramér–Rao bound turns that information into a lower bound on the variance of an unbiased estimator. Information and bounds builds the bound. The same information is what a fitted model later prints as a standard error, and it is the large-sample variance of maximum likelihood.
The bound is attained only in special families. Conditioning an unbiased estimator on a sufficient statistic cannot raise its variance, and the result is still unbiased: that is the Rao–Blackwell improvement. When the sufficient statistic is complete, the improvement is the unique best unbiased estimator. For a normal sample, s^2 is that estimator, and it still does not meet the information bound. Rao–Blackwell and UMVU is the argument. It is worth the effort because without it “best unbiased” is a slogan, and with it you can see that the variance estimator used in every t interval is best inside a class that does not contain every sensible competitor.
Exact unbiasedness often has no solution. Large-sample estimation is the replacement: a regular maximum likelihood estimator is approximately normal for large n, with variance the reciprocal of the information. The sample median of a normal law has asymptotic efficiency 2/\pi\approx 0.637 relative to the mean. The mean wins when the normal model is right, and it is the more fragile of the two when a few wild points own the average. The efficiency is the exchange rate under the model, not a reason to discard the median.
An interval, and a test
A point estimate does not say how far it might fall. A confidence interval (L,U) of level 1-\alpha satisfies
\mathbb{P}_\theta(L<\theta<U)=1-\alpha
for every \theta. The probability is computed before the sample is drawn. After the sample is seen, the endpoints are numbers, and the 95\% has already been used: it is the coverage of the procedure over repeated samples. Plus or minus one standard error covers a normal mean only 68.3\% of the time. Plus or minus two covers it about 95\% of the time. Choosing the multiplier is choosing the coverage.
For a normal sample with unknown variance the multiplier comes from a t law, because s is random. Using a normal quantile with s in place of \sigma covers less often than the stated level. For a proportion, the Wald interval’s true coverage falls away from the stated level when the probability is near 0 or 1, and the Wilson interval tracks the level more closely. The interval coverage laboratory draws the misses. Confidence intervals is the chapter.
A test asks whether the data would be surprising if a stated hypothesis were true. The probability in that question is computed under the hypothesis. For the cartons,
\frac{1.0512-1.10}{0.0317/\sqrt{10}}=-4.87
lies far in the tail of a t law on 9 degrees of freedom. The two-sided p-value quoted for that calculation is 0.0009. The machine is filling below its target. The markers 0.10, 0.05 and 0.01 are conventions about long-run error rates. A small p-value does not measure the size of the departure: the carton mean is only about five hundredths of a litre low, and the sample is large enough, relative to the fill variation, for that small shift to be hard to explain as noise.
The same pivot rejects a point at level \alpha exactly when the point lies outside the matching interval. Hypothesis tests is the chapter. For two fully specified hypotheses, the most powerful test of a given size rejects when the likelihood ratio is large. A one-sided test for a normal mean can be most powerful against every alternative on that side, because they all pick the same region. A two-sided test has no such test: the equal-tailed z or t test is most powerful inside the smaller class of unbiased tests. Most powerful tests draws that line, and it is the reason a two-sided p-value is not “the” optimal answer to every two-sided question.
Most hypotheses leave parameters free. The generalised likelihood ratio compares the best likelihood inside the null with the best likelihood overall. In a regular model, twice the drop in maximised log-likelihood is approximately chi-squared on the number of restrictions. The two-sample t, the analysis-of-variance F, and Pearson’s \sum(O-E)^2/E are that comparison in special cases. A very small chi-squared statistic, a left tail, means the fit is closer than the model expected. Generalised likelihood-ratio tests is the general construction, and chi-squared tests is the count version: a fully specified discrete distribution, or independence in a table. Each estimated parameter spends a degree of freedom.
The chi-squared approximation needs expected counts that are not tiny. An exact test uses the probability of the observed table and of tables at least as extreme, under the null, without that approximation. Fisher’s exact test conditions on the margins and uses a hypergeometric probability. The arithmetic is the same for independence in one sample and for homogeneity across groups; the hypothesis follows the design. Exact tests for tables is that chapter.
Lines, groups, and a table of counts
Regression fits a line and gives the line a probability model, so a slope, a mean response, and a new observation can have standard errors. The model is a statement about the mean at a given x. A high R^2 does not make the line straight, and it does not make the slope an effect of setting x. For one advertising table R^2=0.77; the residual plot is the check the number cannot replace. Linear regression is the chapter.
Analysis of variance compares several means by splitting the total sum of squares into a part between groups and a part within groups. With two groups the F statistic is the square of the pooled two-sample t. Blocking pulls a known source of variation out of the error: in one hardness comparison the between-tip sum of squares stays 0.385 while the error falls from 0.905 to 0.080, and the evidence for a tip difference appears only once that noise is removed. Analysis of variance is the chapter. Tukey intervals handle all pairwise differences of means at a single familywise level, which is sharper than correcting each pair by a Bonferroni factor.
One-way analysis of variance, two-way analysis of variance, and a polynomial regression are the same normal linear model Y=X\beta+\varepsilon with different columns. A hypothesis about several coefficients is one linear restriction, tested by comparing residual sums of squares. Linear hypotheses is that unification. Two factors cross the groups, and the mean may depend on each factor separately or on their combination. The test of no interaction does not depend on whether a coefficient is written as a deviation from an average or as a change from a reference cell; the meaning of a main-effect coefficient does. In one set of eight pots the interaction F is 6.25 on (1,4) degrees of freedom, short of the 5\% critical value 7.71, while the water gap is 3 at low fertiliser and 8 at high. Two-factor analysis of variance is the chapter.
Sensitivity is the probability of a positive test given that the condition is present. Specificity is the probability of a negative test given that it is absent. The predictive values run the other way, and they need a prevalence. A test with sensitivity and specificity both 0.95, used where the prevalence is 0.005, has positive predictive value 0.087 and negative predictive value 0.9997. The test is rarely wrong among people who have the condition, and a positive result is still usually a false alarm, because the condition is rare. A p-value does not supply the missing prevalence. Predictive values of a test is the chapter.
The same procedures on a particular table
The data-analysis chapters carry those procedures onto studies, and the point of each one is a habit that the formula does not mention.
Estimation examples compute an estimate and an interval on particular samples. The confidence remains a property of the procedure. Testing examples carry a stated hypothesis through a statistic and a p-value. Alice’s difference of 10.7 beats per minute has a Welch p-value of about 0.002, allowing the two groups their own variances, and a two-sided 95\% Welch interval of (4.0,\ 17.4). An effect is clear. Its size is not pinned down. The pooled t test gives a similar p-value; the choice is about the variances, not about whether to hunt for a smaller number.
Analysis of variance examples compare a quantitative response across the levels of a factor. Thirty patients with cold sores were assigned at random to five treatments, six each. The placebo mean is 7.5 days, F=3.90 on (4,25) degrees of freedom, and the p-value is 0.014. Tukey’s method at familywise level 0.05 separates only placebo against treatment 3, by 3.17 days. “The treatments differ” is a weaker sentence than “this pair differs”, and the second one needs a procedure that looked at every pair.
Regression examples fit a line and read an interval for a mean response or for a new response. The interval for a new response is wider, because it carries the residual variance as well as the uncertainty in the fitted mean. Linear model examples put simple regression, several covariates, and the analysis of variance into one normal linear model. On a birth-weight table, with the mother’s weight, age, and race held fixed, smoking goes with babies about 402 grams lighter. That is an adjusted association. The mothers were not assigned to smoke. Age alone looks like nothing in that additive fit. Allowing the age slope to differ by smoking, the non-smokers’ babies gain about 27.7 grams per year of the mother’s age and the smokers’ lose about 18.8. The two slopes point opposite ways and cancel when they are forced to be one slope. A main-effect coefficient in a model with an interaction has to be read with the interaction; the smoking coefficient in that second fit is the gap at age zero, which is an extrapolation.
Chi-squared examples compare observed counts with expected counts, including independence in a two-way table. The reference distribution loses a degree of freedom for each parameter the fit estimated.
When a normal line is the wrong shape
An unnamed “regression” hides four assumptions. The responses are independent given the covariates. The conditional mean is linear in the covariates. The variance is constant. The errors are normal. Counts of runs in a week, a clotting time, and a disease indicator each break that list, and the repair does not transform the observations so that a normal line can be forced through them. The link moves the mean. The distribution supplies the variance.
A mean number of runs has to stay non-negative for every age, including ages outside the sample. The log of the mean can be linear while the mean stays positive. A binary mean is a probability, so a logit, a probit, or a complementary log-log keeps it inside (0,1), and the variance p(1-p) is then largest at one half. Outside the normal model these pages do not write the response as mean plus error. The normal linear model is the special case with an identity link and a free constant variance.
A generalised linear model has a distribution for the response, a linear predictor, and a link applied to the mean. The score and the fitting algorithm, Fisher scoring, cover the exponential-family members in one argument. The standard errors come from the inverse Fisher information. Introduction to generalised linear models is that chapter.
Standard errors, and models for counts is the precision of the fit and the Poisson model in which the variance equals the mean. A quasi-Poisson multiplies that variance by a dispersion estimated afterwards. It changes the standard errors and leaves the fitted mean alone, and it is a description you cannot simulate from. A negative binomial can add extra variation. It cannot make the counts tighter than a Poisson. The Conway–Maxwell–Poisson can, and only a mean parameterisation puts that fit on the same scale as a log-linear mean. Underdispersion, positive responses, and quasi-likelihood fits that mean-parameterised model, then turns to a positive continuous measurement whose variance grows with a power of the mean, and to a quasi-likelihood that specifies a variance function without a full distribution. The link is chosen for the scientific reading of the mean. The family mainly chooses the standard errors. A quasi-likelihood has no likelihood, so AIC does not apply to it.
Category labels have no mean worth interpreting. Ordered categories can share one slope across cumulative logits, or across complementary log-log tails. Unordered categories get a separate logit against a baseline. Categorical responses is the chapter. A paired comparison, which of two items wins, is a difference of skills in a Bradley–Terry model. Distance from the sample is the deviance against a saturated description of the same data, compared by chi-squared or F for nested models and by AIC when the candidates are not nested. The residual is still not the observation minus the mean. Paired comparisons, and how far a fit is from the data carries both.
A plot of ordinary residuals from a binary or a count fit is striped, because the response can take only a few values. A probability-integral residual asks whether each observation sits where the fitted distribution said it would, on a uniform scale, without first pushing the data towards normality. Probability-integral residuals is that check.
A curve instead of a coefficient
A coefficient is worth having when a unit change has a meaning. When no formula for the mean is known, a penalty on the squared second derivative lets the mean be a smooth curve rather than a curve through every point. Smooth means is the roughness penalty. Additive smooths put one smooth on each covariate and keep an ordinary coefficient only for the term that needs a unit-change sentence. The basis dimension is an upper bound on the wiggles the curve is allowed, not the complexity of the curve that is returned.
Many measurements, and a label
Several measurements on one unit move together. The multivariate normal is the distribution fixed by a mean vector and a covariance, and it is the joint-normal law from the probability chapters with the dimension left free. Covariance and the multivariate normal is that model: margins are normal, conditionals are normal, and zero covariance still means independence inside this family. When the mean and the covariance are not given, they are estimated from the sample. Under normal sampling the centred cross-product matrix has a Wishart law, of which the chi-squared law is the one-dimensional case, and a hypothesis about the mean vector is Hotelling’s T^2. Estimation and inference for the multivariate normal is the chapter.
A display in two dimensions has to choose two directions. Principal components choose the orthogonal directions that keep as much of the sample’s variance as possible, which is a statement about the sample cloud, not a guarantee that those directions are the ones a later label cares about. Principal component analysis is the construction. It returns in high dimension as a way to summarise a covariance that is too large to invert.
Groups you were not given
Clustering looks for groups when the group label is hidden. K-means places centres so that each point is near the centre of its own group, and the centres are means of the points assigned to them. The result depends on the start, and the number of groups is a choice, not a parameter the algorithm discovers. K-means clustering is the chapter.
A point near a boundary need not belong entirely to one group. A mixture draws a hidden component and then draws the measurements from that component. EM alternates between filling in the component probabilities and refitting the components. Mixtures and the EM algorithm is that pair of steps. Training likelihood alone rewards a more flexible mixture, so the largest model that was tried wins that comparison. A penalty, AIC or BIC, or a held-out likelihood, can prefer a smaller one. Model selection for stochastic models is the chapter, and the same split between fitting and scoring returns for classifiers.
Hierarchical clustering records which groups merge first, instead of forcing one partition. A kernel density estimate turns the same points into a smooth picture of where the sample sits. The bandwidth is the choice: a small bandwidth leaves a bump on the observations, and a large one washes the modes out. Hierarchical clustering and kernel density estimation covers both, and the kernel density laboratory keeps one sample still and moves the bandwidth, with the true density known so the error is visible.
A label on new units
A new unit arrives without a label, and labelled units are available. Discriminant analysis models the measurements inside each class and turns those models into a predicted class. A shared covariance gives a linear boundary. A separate covariance for each class gives a quadratic one. The two estimates are not the logistic fit written below, even when the population log-odds are linear. Statistical classifiers and discriminant analysis is the chapter.
The question after a classifier is fit is how often it fails on a new observation. Scoring the same sample that built the classifier answers an easier question: the apparent error is pulled down because the same pairs enter the rule and the score. A pure tree, or a one-nearest-neighbour rule, can show an apparent error of zero. A test set, cross-validation, and the bootstrap are ways to score cases the rule did not use, and when the rule itself was chosen by looking at those scores the choice has to be nested inside a further split. Estimating error rates is the chapter. It is the reason a later method’s training error is never the last number on the page.
Logistic regression models the probability of the label given the measurements, rather than the measurements inside each class. The log odds are linear in the covariates. A coefficient is a change in log odds: multiplying the odds by e^{\beta_j} does not add a constant to the probability. If a direction in the covariates separates the labels perfectly, the likelihood approaches its upper bound without attaining it at a finite coefficient, and a printed standard error from a fit that did not converge is not a standard error. Logistic regression is the chapter. The same log-odds model is the binary generalised linear model, seen here from the classifier side.
Nearest neighbours store the sample and vote in a neighbourhood. The metric decides which differences count, so the scale has to be learned from the training rows alone. The apparent error always prefers k=1, because each training point is its own nearest neighbour. In a fixed dimension, with k growing slowly, the rule is consistent for the best possible error; a single neighbour pays at most twice that error in the limit. In high dimension an irrelevant coordinate can overwhelm the ones that were actually close. Nearest neighbours is the chapter.
A classification tree asks a sequence of questions and sends each unit to a leaf. The questions are chosen to make the leaves purer. Gini impurity or cross-entropy is used because a raw misclassification count can stay flat until a majority flips. A tree that fits the training sample too closely is then cut back, by a leaf penalty or by cross-validation. Splits are invariant to a rescaling of one coordinate and are not invariant to rotation. Pruning does not make the first split stable. Classification trees is the chapter.
If one tree changes when the training sample is perturbed, fit many perturbed trees and combine them. Bagging bootstraps the rows. A random forest also offers a subset of the predictors at each split, so the trees disagree more, and commonly grows them deep. Boosting reweights the sample so that the next tree attends to what the current combination gets wrong; AdaBoost’s weights are stagewise minimisation of an exponential loss, whose population minimiser is half the conditional log odds. Averaging helps when the members are reasonably accurate and disagree. The variance of an average of correlated errors has a floor set by that correlation, so more copies of the same tree do not drive the variance to zero. Ensembles, random forests, and boosting separates the three.
One linear score cannot separate every labelling of a few points. The exclusive-or pattern is the small example: four inequalities that no single line can satisfy. A neural unit passes a linear score through a nonlinear function, and a layer of those units can build the missing coordinates. One hidden layer of continuous sigmoids can approximate a continuous function on the unit cube uniformly. That existence result supplies neither a sample size nor a training algorithm. Neural units is the representation.
Learning the network means changing the weights to reduce a loss. Backpropagation computes that gradient by the chain rule, from the output back to the earliest weights. A finite-difference check can catch a missing sign. It does not certify that the fitted network will do well on new cases. Backpropagation is the gradient. Networks in practice is everything the gradient does not decide: weights started small on the scale of the incoming links, inputs scaled from the training set, class probabilities from a softmax cross-entropy, and the stopping epoch, the width, and the weight decay chosen on a validation set whose test set is still untouched. Early stopping is a point on the optimisation path. Weight decay is a penalty. They are not the same lever.
Several lines can separate two classes perfectly, so training error cannot choose among them. A support vector machine prefers the slab of greatest width, and only the points on the boundary of that slab determine the line. A cost C prices slack against the margin when the classes overlap. The calculation sees the data through inner products, so a polynomial or Gaussian kernel moves the same linear margin into a richer feature space. The resulting score is not yet a probability. Support vector machines is the chapter.
More coordinates than units
A covariance of four measurements is manageable. Thousands of measurements on a few dozen units is a different problem: the sample covariance has rank at most n-1, so any rule that inverts it is undefined, while a least-squares fit can still produce fitted values with coefficients that are not unique. The tools on the site are a rank-limited principal-component summary, marginal tests with a Bonferroni correction or the Benjamini–Hochberg procedure (and a harmonic threshold that survives arbitrary dependence), a ridge penalty that returns one shrunk vector, and the lasso, whose soft threshold sets some coefficients to zero. High-dimensional analysis is the chapter. A selected set of coordinates is itself a random object, which is why the error-rate chapter’s warning about scoring the selection on the same sample applies here with particular force.
Two measurements can move together because both track a third. An undirected graphical model puts the conditional independences on the missing edges. In a nonsingular Gaussian model a missing edge is a zero of the precision matrix, the inverse of the covariance, and the partial correlation is read from that inverse. When the graph is known, the zeros are constraints on the likelihood. When it is not, the graphical lasso puts a cost on nonzero precision entries. A fitted zero is an estimate of a missing edge, not a proof of one. Graphical models is the chapter. The graph here is a conditional-independence picture. It is not a picture of what would happen if a variable were set.
A probability for the parameter
The intervals and tests above keep the parameter fixed and put the probability on the procedure. A prior is a probability distribution for the unknown before the data arrive. Bayes’ rule multiplies it by the likelihood and divides by the marginal probability of the data. The result is the posterior, a distribution for the parameter given the data that were seen. A credible interval is a set with a stated posterior probability. Its 95\% is a probability for the parameter given this sample. A confidence interval’s 95\% is a coverage for the interval given the parameter. They can have the same endpoints and still be different sentences.
Bayesian foundations is the update, the credible interval, and prediction of a new observation by averaging over the posterior. Fiducial inference is named on that page and left undeveloped. Plugging the posterior mean into the sampling model and predicting from there ignores the remaining uncertainty in the parameter. The predictive distribution is wider.
A beta prior on a coin’s heads probability, updated by binomial data, is another beta. For the posterior mean, the prior parameters act as prior heads and prior tails. For an interior posterior mode the corresponding counts are smaller by one. A pseudo-count sentence has to say which summary it means. The probability of a future head is the posterior mean of the heads probability. The Beta–Binomial laboratory shows the update and the predictive distribution. The Beta–Binomial model is the chapter.
A normal sample with a normal prior for the mean has a normal posterior, a precision-weighted average of the sample mean and the prior mean. A very wide prior escapes every fixed window; it does not become a flat distribution on the whole line. Unknown variance brings an inverse-gamma partner, in the shape-and-rate convention used throughout. With both parameters unknown, the joint posterior couples them even though each full conditional is a standard law. Conjugate analysis for normal models is the chapter.
Those full conditionals are Gibbs sampling: each coordinate is drawn from its conditional given the others, and every draw is kept. When a conditional cannot be drawn, Metropolis–Hastings proposes a move and accepts or rejects it with a ratio that leaves the target stationary. A rejection repeats the current state. A warm-up period is not a proof of convergence. An independent sample on a grid is not a Markov chain. Markov chains, Gibbs sampling, and Metropolis–Hastings is the chapter, and it uses the stationary-distribution facts from the Markov chapters.
Several categories replace the two outcomes of a coin. A Dirichlet prior is the beta on the simplex, conjugate to the multinomial, with negative covariances between the category probabilities: if one category’s share goes up, the others share the remainder. The uniform Dirichlet spreads over probability vectors. It does not sit as a point mass at the centre. Categorical data and the Dirichlet distribution is the update.
A Bayes factor compares two models by the probability each gives to the observed data, averaging over its own prior. It updates the prior odds of the models. It is not itself the posterior probability of either model; that probability comes only after the prior odds are multiplied in. Bayesian hypotheses and model comparison is the chapter, and it is the Bayesian counterpart of the p-value, not a rephrasing of it.
The conjugate updates for a coin, a normal mean, and a Poisson rate are one pattern. In an exponential family the prior that stays in the same family adds its natural statistic to the statistic from the sample. For the Poisson, the power in the kernel and the gamma shape differ by one, which is the same off-by-one as the two beta pseudo-count conventions. Exponential families and conjugacy is the pattern. Sufficiency, which the Rao–Blackwell chapter uses, is the same statistic.
Related groups can share a distribution for their parameters instead of being pooled into one group or left completely separate. That shared distribution is the hierarchy. Small groups are pulled harder towards the common mean, because their own data say less. Treating the hyperparameters as known is a different analysis from integrating them; the second one is the full hierarchy. Hierarchical Bayes is the chapter.
Bayesian linear regression gives each observation a mean linear in the predictors and puts a prior on the coefficients and the variance. With prior density proportional to 1/\sigma^2, the posterior for the coefficients given the variance is the normal law centred at the least-squares fit, the marginal posterior for \sigma^2 is inverse-gamma with shape (n-p)/2, and the marginal posterior for the coefficients is a multivariate t. The coefficient intervals then have the same endpoints as the usual t intervals and a different meaning. A new observation is wider than the mean response by the extra residual draw. Bayesian linear regression is the chapter.
The kernels, the pseudo-count conventions, and a short list of the traps on these pages — a uniform prior that is not a distribution on the line, a Bayes factor read as a posterior probability, a Gibbs sampler confused with an independent grid — are gathered in the formula lookup.
A path
A single distribution is the wrong object when the question is about a trajectory. A stochastic process is a family of random variables indexed by time, so the state, the clock, and the independence have to be named before a theorem applies. Stationary increments fix the law of a change of a given length and leave the level free to spread. Stochastic processes sets that vocabulary. An extinction probability, a stationary distribution, and a limit of maxima are exact for a fully specified joint law. A short record need not identify which law you have.
A branching process counts a population in which each individual independently produces offspring. The mean number of offspring, m, decides the threshold: for a non-degenerate law that can produce no offspring, extinction has probability 1 when m\le 1, and probability less than 1 when m>1. The mean does not give the numerical value. An offspring law that is 0 with probability 1/4 and 2 with probability 3/4 has mean 3/2 and extinction probability 1/3, so the line survives with probability 2/3 while the expected size still grows like (3/2)^n. The deterministic one-child law is the critical case that does not die out. Galton–Watson branching is the chapter.
A random walk adds independent steps. At a fixed time the position is a law-of-large-numbers and central-limit calculation. The path question is different: the chance of hitting a barrier, and the time it takes. On a fair walk that starts at x and stops at 0 or d, the chance of hitting d is x/d, and the mean time to absorption is x(d-x). Random walks and first passage is the chapter.
The finite-state Markov chapter is the start. Discrete-time Markov chains add the first-step analysis for a hitting probability and a mean hitting time, and they allow a countable set of states. Classes, recurrence, and reversibility sort the states into classes that can or cannot reach one another. A finite irreducible chain has one stationary distribution, strictly positive, and \pi_i is the reciprocal of the mean return time to i. Aperiodicity is what makes the n-step probabilities converge to \pi. Detailed balance, \pi_i p_{ij}=\pi_j p_{ji}, implies the global balance equations and is often easier to check. A deterministic cycle can be stationary with probability flowing one way, which is balance without reversibility.
A Poisson process counts events that arrive with no memory of the last one. Counts on disjoint intervals are independent and Poisson, with mean proportional to length, and the gaps are exponential. Conditional on n events in an interval, their times are like n uniform draws. A rate that varies with time keeps the independent increments and replaces the mean by the integrated rate. The same construction on the plane is a spatial Poisson process. Poisson processes is both.
A continuous-time chain waits an exponential time and then jumps. The rate matrix Q replaces P, the transition probabilities solve P'(t)=P(t)Q, and on a finite irreducible space they converge to the unique \pi with \pi Q=0. Exponential holding times remove the discrete period. Continuous-time Markov chains is the chapter.
A queue is a birth-death chain: arrivals increase the number present and services decrease it. For an M/M/1 queue the traffic intensity is \rho=\lambda/\mu, the number present is geometric with mean \rho/(1-\rho) when \rho<1, and the mean time in system is 1/(\mu-\lambda). Near \rho=1 that mean moves sharply when the rates move a little. A finite waiting room restores a stationary law at every load by turning arrivals away, and the throughput then falls below the offered rate. Little’s law, L=\lambda_{\mathrm{eff}} W, relates the mean number present to the mean time in system and does not need the exponential assumptions. Birth–death chains and queues is the chapter.
The processes above treat events and random variables as already meaningful. On a finite set every subset can be an event. On an uncountable set of paths, a sigma-algebra says which subsets are events, so that probability is defined where the later arguments need it. Countable additivity makes “infinitely many of these events occur” an event, which is the set in the Borel–Cantelli lemmas. Probability spaces and events is that construction. It sits after the queues in the topics list because the queue formulas can be read from the Poisson and exponential chapters; the sigma-algebra is the justification of the uncountable sample space those formulas already used.
A random variable, in that language, is a measurable map from outcomes to numbers, and its law is the probability it pushes forward onto the line. Two variables can share a law without living on the same outcomes. Random variables and laws is the chapter. Expectation is rebuilt as a limit of simple variables.
A sequence of random variables can settle in several senses: almost surely, in probability, in L^p, and in distribution. They are not the same. A spike that is n with probability 1/n and 0 otherwise goes to 0 almost surely and keeps mean 1, so an expectation can be owned by an event that probability says is vanishing. Convergence in distribution is the sense in the central limit theorem, and it does not ask the variables to live on one probability space. Expectation and convergence separates the four, and it is where the strong law is proved with the measure just built.
An average is pulled by many observations. A maximum is pulled by the upper tail, and the scaling that gives it a limit is not the \sqrt n scaling of an average. For an exponential tail the maximum, after a logarithmic centring, settles on a Gumbel law. A heavy tail of Pareto type settles on a Fréchet law after scaling by a power of n. A first look at extremes is the chapter.
Writing the calculation down
A command is useful when the rows, the assumption, and the scale of the printed number match the question. The chapters above are the questions. These are the commands, and several of them exist to stop a plausible command from answering a different question.
Using R is the language: a table, a summary, a plot, and a fit of a model that has already been defined. Exploratory data analysis in R looks before the fit. The New York air-quality file has 153 days, May through September 1973, and 111 of them have ozone, solar radiation, wind, and temperature all present. A model that needs all four is a model of those 111 days. Dropping missing ozone with a missing-value flag describes the measured days, and a band of missing days on particular dates is a different problem from ozone missing uniformly. Neither pattern is repaired by deleting the blanks.
Probability and simulation in R is a density, a cumulative probability, a quantile, and a draw, together with a permutation test that compares a statistic with a shuffle of the labels. Bootstrap and numerical methods in R redraws the observed units when the sampling distribution has no formula. A bootstrap of days treats the days as independent. Weather persists, so the interval can understate the uncertainty of a seasonal mean. One-dimensional integration and optimisation are the numerical checks for a density whose integral or mode can also be written down. Mixtures and clustering in R fits the mixture and the clustering, and it keeps a label out of the fit when the label is what you hope to recover. Bayesian computation in R computes a conjugate update, a Bayes factor, and the two samplers. A Monte Carlo average over the wrong target is a precise answer to the wrong question.
Fitted models as objects treats a fit as something with several views. The printed summary is one of them. Coefficients, the covariance, the rows that were used, an interval for the mean, and an interval for a new observation are different extractions. A temperature supplied in Celsius to a model fitted in Fahrenheit is a silent change of the question. Generalised linear models in R is the command and the scale of each printed number, for binary data, counts, and positive responses. A gamma log-link coefficient is a multiplier of the mean, not an additive shift. Generalised additive models in R is the smooth as a contribution to that mean.
Repeated measures in R uses the craniofacial distances of 27 children at ages 8, 10, 12 and 14: 108 visits, 27 independent units. A regression that treats the visits as independent pretends there are 108 children. The exchangeable working correlation for four visits cannot lie below -1/3, and a smoothly rising distance is unlikely to have one common correlation at every lag in any case. Time series in R fits on a past stretch only, scores a later stretch, and reads the sign in front of a lagged term. A random holdout that mixes future months into the fit leaks the forecast.
Checking a calculation is the discipline under the commands. A surprising number is usually a row, a scale, or a unit. A check that changes two of those at once cannot say which one moved the answer.
The laboratories are the four places where the population is known, so the error is visible: kernel density, Beta–Binomial updating, a Markov chain, and interval coverage.
A way through the site
Each chapter is marked beginner, intermediate, or advanced, according to whether it uses only events, variables, samples, and tables, whether it adds one method, or whether that method is already assumed. A question has the level of its chapter. The topics page is the list, part by part.
The first five chapters, in the order the main page gives them, are the experiments, the random variables, the studies, the descriptive summaries, and Using R. After those, the carton argument in statistical inference is the shortest path into a claim about a parameter. Bayesian foundations is the shortest path into a probability for the parameter. The generalised linear model is the shortest path out of a normal line. The multivariate normal is the shortest path into the learning chapters.
The question bank is every question in one list. By topic is the same list under each chapter, which is also how a chapter ends. A random question draws one. The worked answer is folded away at the bottom of the question page.
Final preparation is a set of questions that draw on more than one chapter. Bayesian final questions and applied final questions are the two halves.
The glossary and index is every defined term, theorem, and section. The reference page points at the glossary, the distribution sheet, and the formula lookup.
What these pages do not yet do
A few questions the story runs into are named, and then left.
Fiducial inference is named, so that the frequentist and Bayesian accounts are not mistaken for a complete list, and no fiducial method is built.
Forecasting stops before a model for a changing variance, and before a count-valued series. Both boundaries are stated on that page.
Reliability builds a lifetime from a structure and a hazard, including a Weibull hazard that rises with age. A regression for a lifetime that is only known to have exceeded the end of the study is not a chapter.
Studies and regression are explicit that a randomised assignment can support a causal sentence and that an observational coefficient, even with other covariates held fixed, is an association. The further machinery that sometimes recovers an effect when the treatment was not assigned is not a chapter. A graphical model records conditional independence. It does not record an intervention.
Sampling designs are taught as ways of choosing units. A formula for a population total when units are included with unequal probabilities is not developed.
Missing entries are treated as a decision about which rows the claim describes. A tree has to send an incomplete row down a split. A model whose subject is the process that made values missing is not a separate chapter.