06 · Learning path
k-Nearest Neighbors (k-NN)
Classifies a new case by majority vote among the closest examples, without training an explicit model.
k-NN does not learn a global formula: it stores the data and, when a new case arrives, looks at its nearest neighbors and decides by majority.
Explore a local decision
k-NN lab
Local majority vote among neighbors
Settings
Always odd to avoid ties.
Local prediction
Class 0 (squares)
5 of 5 neighbors vote for this class.
Class 1 votes
0
Class 0 votes
5
- Class 1
- Class 0
- Neighbor used
What to look for
- k = 1 memorizes the noise: the boundary is very irregular.
- Large k smooths it out, but on the ring it ends up erasing the pattern.
- Only the ringed neighbors influence the prediction.
The idea
The decision depends on the immediate neighborhood. If positive examples dominate around the new point, the prediction will be positive; if negatives dominate, the opposite happens.
What to look for
- With a small
, the model reacts strongly to local noise. - With a large
, the boundary becomes smoother and loses detail. - The scale of the variables matters: distance (usually Euclidean) decides everything, so it is a good idea to standardize.
- In high dimensions, distances lose discriminating power (the “curse of dimensionality”).
Applications
Useful for prototypes, local pattern recognition, and settings where an intuitive rule based on proximity is preferred over a complex parametric model. Its cost is at prediction time: every new case has to be compared with the whole sample.