Laboratory: odds and probabilities for two iris species
Section of Logistic regression, part of Statistical learning
Laboratory: odds and probabilities for two iris species
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Setosa is easy to separate on these measurements, so a three-class fit would hide the binary difficulty. Restrict training to versicolor and virginica and use sepal length alone. Set Y=1 for virginica, so a positive coefficient means larger modelled odds of virginica. The probability threshold 1/2 is the equal-cost rule. The Brier score below scores the probabilities themselves.
btrain <- droplevels(subset(train, Species != "setosa"))
btest <- droplevels(subset(test, Species != "setosa"))
btrain$y <- as.integer(btrain$Species == "virginica")
logit_fit <- glm(y ~ Sepal.Length, data = btrain, family = binomial())
round(coef(summary(logit_fit)), 4)
#> Estimate Std. Error z value Pr(>|z|)
#> (Intercept) -12.446 3.3824 -3.680 2e-04
#> Sepal.Length 1.995 0.5439 3.669 2e-04
exp(coef(logit_fit)["Sepal.Length"])
#> Sepal.Length
#> 7.355
prob <- predict(logit_fit, btest, type = "response")
truth <- as.integer(btest$Species == "virginica")
c(test_error = mean((prob >= 0.5) != truth),
Brier_score = mean((prob - truth)^2))
#> test_error Brier_score
#> 0.2000 0.1844The estimated odds multiplier for one additional centimetre of sepal length is 7.35. That number multiplies the odds. It does not add 7.35, or 1.995, to the probability, because the logistic curve is nonlinear. The Brier score averages the squared difference between the fitted probability and the binary label, so two classifiers can make the same hard decisions at the threshold 1/2 and still receive different Brier scores when their probabilities differ. On these held-out flowers the threshold-1/2 error is 0.2000 and the Brier score is 0.1844.
Logistic regression models the log odds as \beta^\top x, fits \beta by the Bernoulli score X^\top(y-p)=0, and classifies by the sign of that linear predictor when the threshold is 1/2. A coefficient multiplies the odds. The probability moves by ap/(1-p+ap), which depends on where the curve already sits.
Used in
See also
Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Interpretation and the decision boundary