Naive Bayes

Naive Bayes uses Bayes’ theorem to estimate which class is most likely given the observed evidence. Its “naive” assumption is that the variables are conditionally independent within each class.

Combine evidence step by step

Naive Bayes lab

Probabilistic evidence in text

Evidence observed in the email

Contains “free”presentUrgent tonepresentShortened linkpresentKnown senderabsent

Prior P(spam)

0.42

How common spam is before seeing the email.

Contribution of each piece of evidence to the log-odds (in favor of spam →)

• Prior−0.32
• Contains “free” (yes)2.25
• Urgent tone (yes)1.29
• Shortened link (yes)1.71
• Known sender (no)1.52

Decision (MAP)

Spam

total log-odds = 6.45. Spam is predicted when it is greater than 0.

Posterior probability

99.8%

P(spam | e) = 0.998 · P(legitimate | e) = 0.002

What to look for

• Conditional independence turns the product of probabilities into a sum of log-odds.
• The absence of a cue is also evidence: “no free” pushes toward legitimate.
• Change the prior: with little evidence it dominates; with a lot, it hardly matters.

The decision rule

Bayes’ theorem gives the posterior probability of each class given the attributes :

The “naive” assumption factorizes the likelihood as a product of simple terms, one per variable:

Since is the same for all classes, it is enough to compare the numerators and pick the largest:

In practice, logarithms are summed instead of multiplying probabilities, to avoid numerical underflow.

What to look for

• With only a few counts you already get a useful posterior probability.
• The independence assumption is strong, but it often works well, especially for ranking classes.
• Its probabilities tend to be poorly calibrated: it gets the class right but overstates its confidence.
• It is especially competitive on text and on high-dimensional problems.

Applications

Spam filtering, sentiment analysis, document classification, and tasks where training and prediction speed matter a lot.