Is my model good?
You trained a classifier in Week 4. The first question anyone asks is simple: how many did it get right? That is accuracy, correct guesses divided by total guesses. Easy to trust. Easy to put in a slide. Let us see why that comfort is dangerous.
100 messages · one model
5 of 100 messages are spam. This model always predicts ham. It never even looks at the message. Hover the grid.
0%
Accuracy
95%
Recall on spam
A high headline number hid a model that catches zero spam. When classes are imbalanced, accuracy alone lies.
The trap
Ninety-five messages are real mail, five are junk. A model that always says ham gets ninety-five out of one hundred correct. Accuracy: 95%. But it caught zero spam. Every phishing message sailed through. That is not a Python bug. On skewed data, accuracy rewards ignoring the rare class. Before we go further, remember the pairing you just saw: a hero grade and zero spam caught.
Four kinds of mistakes
Accuracy mixes every error into one pile. A confusion matrix sorts them. Rows: what the message really was. Columns: what the model said. Four boxes: spam correctly flagged, spam missed, real mail wrongly flagged, real mail left alone. Every score for the rest of the night is math on these four counts. Below we score a real filter on one hundred messages, twelve spam and eighty-eight ham. Same inbox world as Week 4. Tap a cell and read what that mistake looks like.
Confusion matrix · 100 messages
Click a cell to read a real message. Hover to preview. Four mistake types, one grid.
actual ↓ · predicted → spam | ham
Example · True positive
“FREE $$$ click now to claim your prize!!!”
Actual spam · Predicted spam
93%
73%
67%
Two questions your user cares about
Stakeholders usually ask two questions, not one. When the model cries spam, is it usually right? That is . High precision means few false alarms. Of all the spam that existed, how much did we catch? That is . High recall means an aggressive filter. You cannot max both. Lower the threshold from Week 4 and recall rises while precision falls. Raise it and the opposite happens. If someone wants one number to compare models, blends both.
Metrics
Threshold trade-off
20 scored messages. Slide the threshold and watch precision and recall pull in opposite directions. Hover a dot for its score.
Slide the threshold. Precision and recall tug in opposite directions.
Which metric should I trust?
There is no universal score. The right metric depends on which mistake hurts more.
If a false alarm is costly, prioritize : when the model cries spam, is it usually right? If a miss is costly, prioritize : of all real spam, how much did we catch? On balanced data, accuracy can be enough. On skewed inboxes, reach for or the full report. To see every mistake type, start with the confusion matrix.
Which metric should I trust?
Pick a situation. The stakes dial tilts, the answer lights up.
Reach for
Precision
Precision asks: when we flag spam, how often are we right?
Quick reference: false alarm costly → . Miss costly → . Balanced classes → accuracy. Skewed inbox → or the classification report. Need every mistake type → confusion matrix first.
The classification report
Printing precision, recall, and F1 by hand gets old fast. sklearn's classification_report does it in one call.
Each row is a class. **support** counts real examples of that class in the test set. **macro avg** averages each metric across classes equally, even when spam is rare. **weighted avg** averages while respecting support, so the bigger class pulls more. On imbalanced mail, compare macro and weighted before you brag about a headline number.
classification_report output
One sklearn call prints every per-class metric. Watch the report stream in, then x-ray a column.
support = how many real examples of that class were in the test set. Spam has 12, ham has 88.
print(classification_report(y_test, y_pred, target_names=['ham', 'spam']))
Did it learn or memorize?
One test score tells you how you did once. Train vs test curves tell you whether tuning survives new mail. Underfit: both errors stay high, the model is too simple. Good fit: test error falls with train error, then levels off with a small gap. Overfit: train error keeps falling while test error climbs back up. Same trap as Week 2 and Week 5. The goal is not the lowest train error. The goal is low test error with train and test still close.
Train vs test error
Same model family, three amounts of capacity. Hover the chart to scrub along the x-axis and watch the gap between curves.
Test error tracks train error with a small, steady gap. This generalizes.
The cheat column
You can split train and test correctly and still fool yourself.
Picture a school exam: if someone slips you the answer key while you study, your practice score looks amazing. On the real exam, without the key, you fail.
The answer key in your table
Machine learning has the same cheat. Sometimes a column carries information from after the decision you are trying to predict.
Example: already_marked_spam_by_admin, filled only after a human reviews the email. The model learns that shortcut instead of reading the message. Test accuracy hits 99%. Deploy without that column and the score collapses.
That cheat is called .
- Overfitting: memorizes quirks in normal features.
- Leakage: uses a column you would never have at prediction time.
Before you train, ask for every column: would I know this value the moment I need to predict?
The suspiciously good feature
Toggle the cheat column in the feature table. Train looks great until you deploy without it.
99% looks like a win, but it's an illusion. Hit simulate deploy to see what happens in production.
You do not need ten metrics on day one. You need a ladder: accuracy, then a confusion matrix, then precision and recall, then which metric fits your stakes, then sklearn's classification report, then train vs test curves, then a leakage check. Climb in order. Skip a rung and you ship blind.