Linear Regression

Linear regression looks for the line that best fits a set of points. It is the foundation of much of modern machine learning and the standard tool for predicting numerical values.

Experiment with the OLS line

Draw points and watch the line refit automatically by ordinary least squares (OLS).

Regression lab

Ordinary least squares

ExampleClear

y = 0.54x + 21.7

Click the plane to add a point; click a point to remove it14/40

Model

Mode

Automatic (OLS)Manual

Squares

NoneResidualsVs. mean

Add random point

Goodness of fit

R² = 0.656

This line minimizes the sum of squared residuals.

Variance

SSE

937

SST

2725

β₁ (OLS)

0.540

β₀ (OLS)

21.7

Minimum SSE

937

• OLS line
• Your line
• OLS (reference)

What to look for

• R² = 1 − SSE/SST: how much variance the line explains compared with using only the mean.
• Each square has a side equal to the residual; its area is its contribution to the SSE.
• In manual mode, try to match the minimum SSE by moving both sliders.

The model

With one explanatory variable , the model is

where is the intercept (the expected value of when ), is the slope and is the random error.

OLS chooses the coefficients that minimize the sum of squared errors (SSE):

In the simple case there is a closed-form solution, and it connects with the earlier concepts:

How well it fits

The coefficient of determination measures the fraction of the variance of that the model explains:

In simple regression, . A high does not guarantee that the model is correct: always check the residuals.

Applications

From predicting the price of a house from its square footage to estimating the effect of a marketing campaign on sales.