Fatal Error ClubFatal Error
Roadmap

Week 3 · Classical ML

Regression

01

What regression means

predicts a number. Features go in. A quantity comes out: a price, a temperature, how many minutes a trip takes.

Several models can do that job. They disagree about the shape of the function. Each card is one model: the picture is the shape, the equation is the function. is the one we fit next, on house prices.

Linear regression

Tonight

A straight line through the cloud. More features turn the line into a flat plane.

Function

y^=b0+b1x\hat{y} = b_0 + b_1 x

Polynomial regression

The same weights, with powers of the input, so the fit is allowed to bend.

Function

y^=b0+b1x+b2x2\hat{y} = b_0 + b_1 x + b_2 x^2

Ridge

Still a weighted sum. Every weight is pulled toward zero so one feature cannot dominate.

AreaRoomsAgeLot

Function

y^=b0+∑jbjxj\hat{y} = b_0 + \sum_j b_j x_j

Penalty

λ∑jbj2\lambda \sum_j b_j^2

Lasso

Still a weighted sum. Some weights land on exactly zero, and that feature drops out.

AreaRoomsAgeLot

Function

y^=b0+∑jbjxj\hat{y} = b_0 + \sum_j b_j x_j

Penalty

λ∑j∣bj∣\lambda \sum_j |b_j|

Regression tree

Yes or no questions cut the input into ranges. Each range predicts one number.

Function

y^=mean(y in the leaf)\hat{y} = \mathrm{mean}(y \text{ in the leaf})

k-NN regression

Find the nearest examples and average their numbers. The rows are the model.

Function

y^=1k∑i∈Nk(x)yi\hat{y} = \frac{1}{k} \sum_{i \in N_k(x)} y_i
02

Intro: Selling a house

I'm an AI engineer selling my house for the best reasonable price. I gather data on houses in my region: area, rooms, and so on, plus price, the label. With that labeled data, the computer can find the function that predicts it best.

Each dot is a house.

area (m²)price (k)4TH HOUSEwith an area of 74 m²and a price of 188k7TH HOUSEwith an area of 108 m²and a price of 262k
03

Here's a line. It's bad.

A line predicts price from area. m is the slope: how fast price rises with area. p is the starting point: the price when area is zero. Feed in an area, the line predicts a price. Below is a bad first try: red predictions sit far from the green facts.

areapricey = x + 55

● real price·● line prediction

Bad line. Red predictions sit far from the green facts. Eyeball the gaps first. No total yet.

04

Zoom in: what error means for one house

Same 4th house from the intro: 74 m², 188k. Green is the real price. Red is what the bad line predicts for that area. The gap between them is the error.

areapricereal 188kpred 129kerror +59
05

Now do that for every house at once

Every house has a gap like that one. We don't want to shrink just one: we want to shrink all of them together. Square each gap so over- and under-predictions can't cancel out, and bigger misses count for more. Add them up. The better line has the smaller total.

=∑(y−y^)2= \sum (y - \hat{y})²
areaprice

● real price·● line prediction

Total Error

40985

06

Try. Measure. Adjust.

Now let the computer run the loop: start with a bad line, measure the squared gaps, adjust m and p, repeat until the error won't shrink further. Same game as last week: guess, check, correct. No new formula here, just the process.

The search

Same loop as last week. Each Next tries a line, measures the squared gaps, then adjusts. Watch the error getting smaller.

areaprice

Press Next to try the first line.

07

More features, same idea

Real houses have more than area: rooms, location, age. Add features and the line becomes a plane through the cloud. Same try, measure, adjust.

Plane

y^=b0+b1⋅\hat{y} = b_0 + b_1\cdot+b2⋅+ b_2\cdot
Loading 3D fit…

Recap

Cheat sheet.

Recap only. No new ideas. The formulas you already met, lined up.

  1. 01

    Line

    y=mx+py = mx + p
    y^=b0+b1x\hat{y} = b_0 + b_1x

    You met this when the bad line first appeared. Slope and starting point.

  2. 02

    Error

    error=y−y^error = y - \hat{y}
    SSE=∑(y−y^)2SSE = \sum (y - \hat{y})²

    One gap first. Then every house, squared and summed.

  3. 03

    More features

    y^=b0+b1⋅\hat{y} = b_0 + b_1\cdot+b2⋅+ b_2\cdot

    Extra inputs. The fit becomes a plane.

Linear regression predicts a number by fitting a line. The better line is the one with the smaller total of squared gaps.