04 · Learning path
Logistic Regression
The classic model for estimating probabilities and making binary decisions from a linear combination.
Logistic regression models the probability that the positive class occurs. It starts from a linear combination of the input variables, but squeezes the output between 0 and 1 using a sigmoid function.
Experiment with the decision boundary
Move the slope, the intercept and the threshold to see how the logistic curve, the decision boundary and the confusion matrix change on a simulated binary sample. The controls redefine the process that generates the data: you are not retraining a model on a fixed sample, but exploring different scenarios.
Logistic regression lab
Sigmoid, likelihood and threshold
Parameters
It changes the decisions, not the probabilities.
Likelihood
Loss 0.367
Attainable minimum: 0.259. |∇| = 0.421 · boundary at x* = 0.00.
∂L/∂β₀
0.043
∂L/∂β₁
−0.419
Accuracy
85.4%
Precision
80.0%
Recall
90.9%
Confusion matrix
- Correctly classified
- Error
- Threshold
What to look for
- The data come from a fixed model; you look for the β that best recover it.
- The gradient is zero at the optimum: that is where the loss stops decreasing.
- Raising the threshold sacrifices recall to gain precision.
The fundamental equation
The estimated probability of belonging to the positive class is
Instead of predicting an unbounded line, the model transforms the linear combination into an S-shaped curve.
From the logit to classification
Solving for the linear combination gives the logit (log of the odds):
The parameters can be read from it:
shifts the curve to the left or right. controls how abrupt the transition between classes is; also, is the factor by which the odds are multiplied when increases by one unit. - The threshold
turns the probability into a decision: if , the model predicts class 1. With the boundary is at .
How it is fitted
The coefficients are estimated by maximizing the likelihood, which is equivalent to minimizing the log loss:
What to look for
- If you raise the threshold, the model becomes more conservative when labeling positives.
- If you increase the slope, the transition between classes becomes sharper.
- Accuracy can improve or worsen depending on the balance between false positives and false negatives.
- Log loss penalizes confident but wrong predictions more heavily.
Applications
Fraud detection, customer churn prediction, disease risk estimation and, in general, any problem with a yes/no, success/failure or buy/no buy response.