Fatal Error ClubFatal Error
Roadmap

Week 6 · Classical ML

Model evaluation, accuracy, precision/recall, overfitting/underfitting

01

Is my model good?

You trained a classifier in Week 4. The first question anyone asks is simple: how many did it get right? That is accuracy, correct guesses divided by total guesses. Easy to trust. Easy to put in a slide. Let us see why that comfort is dangerous.

100 messages · one model

5 of 100 messages are spam. This model always predicts ham. It never even looks at the message. Hover the grid.

0%

Accuracy

95%

Recall on spam

A high headline number hid a model that catches zero spam. When classes are imbalanced, accuracy alone lies.

02

The trap

Ninety-five messages are real mail, five are junk. A model that always says ham gets ninety-five out of one hundred correct. Accuracy: 95%. But it caught zero spam. Every phishing message sailed through. That is not a Python bug. On skewed data, accuracy rewards ignoring the rare class. Before we go further, remember the pairing you just saw: a hero grade and zero spam caught.

03

Four kinds of mistakes

Accuracy mixes every error into one pile. A confusion matrix sorts them. Rows: what the message really was. Columns: what the model said. Four boxes: spam correctly flagged, spam missed, real mail wrongly flagged, real mail left alone. Every score for the rest of the night is math on these four counts. Below we score a real filter on one hundred messages, twelve spam and eighty-eight ham. Same inbox world as Week 4. Tap a cell and read what that mistake looks like.

Confusion matrix · 100 messages

Click a cell to read a real message. Hover to preview. Four mistake types, one grid.

actual ↓ · predicted → spam | ham

Example · True positive

“FREE $$$ click now to claim your prize!!!”

Actual spam · Predicted spam

Accuracy=TP+TNtotal\text{Accuracy} = \frac{TP + TN}{\text{total}}

93%

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}

73%

Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}

67%

04

Two questions your user cares about

Stakeholders usually ask two questions, not one. When the model cries spam, is it usually right? That is . High precision means few false alarms. Of all the spam that existed, how much did we catch? That is . High recall means an aggressive filter. You cannot max both. Lower the threshold from Week 4 and recall rises while precision falls. Raise it and the opposite happens. If someone wants one number to compare models, blends both.

Metrics

precision=TPTP+FP\text{precision} = \frac{TP}{TP + FP}
recall=TPTP+FN\text{recall} = \frac{TP}{TP + FN}
F1=2⋅P⋅RP+RF_1 = \frac{2 \cdot P \cdot R}{P + R}

Threshold trade-off

20 scored messages. Slide the threshold and watch precision and recall pull in opposite directions. Hover a dot for its score.

t
0.0score →1.0
0.50
TP 8 · FP 2 · FN 2
Precision80%
Recall80%
F180%

Slide the threshold. Precision and recall tug in opposite directions.

05

Which metric should I trust?

There is no universal score. The right metric depends on which mistake hurts more.

If a false alarm is costly, prioritize : when the model cries spam, is it usually right? If a miss is costly, prioritize : of all real spam, how much did we catch? On balanced data, accuracy can be enough. On skewed inboxes, reach for or the full report. To see every mistake type, start with the confusion matrix.

Which metric should I trust?

Pick a situation. The stakes dial tilts, the answer lights up.

Miss hurtsFalse alarm hurts
Recall
Precision

Reach for

Precision

Precision asks: when we flag spam, how often are we right?

Quick reference: false alarm costly → . Miss costly → . Balanced classes → accuracy. Skewed inbox → or the classification report. Need every mistake type → confusion matrix first.

06

The classification report

Printing precision, recall, and F1 by hand gets old fast. sklearn's classification_report does it in one call.

Each row is a class. **support** counts real examples of that class in the test set. **macro avg** averages each metric across classes equally, even when spam is rare. **weighted avg** averages while respecting support, so the bigger class pulls more. On imbalanced mail, compare macro and weighted before you brag about a headline number.

classification_report output

One sklearn call prints every per-class metric. Watch the report stream in, then x-ray a column.

spam_filter_eval.py
precisionrecallf1-scoresupport
ham0.960.970.9688
spam0.730.670.7012
accuracy0.93100
macro avg0.840.820.83100
weighted avg0.930.930.93100
ham
88
spam
12

support = how many real examples of that class were in the test set. Spam has 12, ham has 88.

print(classification_report(y_test, y_pred, target_names=['ham', 'spam']))

07

Did it learn or memorize?

One test score tells you how you did once. Train vs test curves tell you whether tuning survives new mail. Underfit: both errors stay high, the model is too simple. Good fit: test error falls with train error, then levels off with a small gap. Overfit: train error keeps falling while test error climbs back up. Same trap as Week 2 and Week 5. The goal is not the lowest train error. The goal is low test error with train and test still close.

Train vs test error

Same model family, three amounts of capacity. Hover the chart to scrub along the x-axis and watch the gap between curves.

model capacity →error
Train · 9%Test · 15%gap 6 pts

Test error tracks train error with a small, steady gap. This generalizes.

08

The cheat column

You can split train and test correctly and still fool yourself.

Picture a school exam: if someone slips you the answer key while you study, your practice score looks amazing. On the real exam, without the key, you fail.

09

The answer key in your table

Machine learning has the same cheat. Sometimes a column carries information from after the decision you are trying to predict.

Example: already_marked_spam_by_admin, filled only after a human reviews the email. The model learns that shortcut instead of reading the message. Test accuracy hits 99%. Deploy without that column and the score collapses.

That cheat is called .

  • Overfitting: memorizes quirks in normal features.
  • Leakage: uses a column you would never have at prediction time.

Before you train, ask for every column: would I know this value the moment I need to predict?

The suspiciously good feature

Toggle the cheat column in the feature table. Train looks great until you deploy without it.

messageword_counthas_linkresolved_by_refund
msg_1120yes
msg_2120yes
Test accuracy99%

99% looks like a win, but it's an illusion. Hit simulate deploy to see what happens in production.

You do not need ten metrics on day one. You need a ladder: accuracy, then a confusion matrix, then precision and recall, then which metric fits your stakes, then sklearn's classification report, then train vs test curves, then a leakage check. Climb in order. Skip a rung and you ship blind.