What regression means
predicts a number. Features go in. A quantity comes out: a price, a temperature, how many minutes a trip takes.
Several models can do that job. They disagree about the shape of the function. Each card is one model: the picture is the shape, the equation is the function. is the one we fit next, on house prices.
Linear regression
Tonight
A straight line through the cloud. More features turn the line into a flat plane.
Function
Polynomial regression
The same weights, with powers of the input, so the fit is allowed to bend.
Function
Ridge
Still a weighted sum. Every weight is pulled toward zero so one feature cannot dominate.
Function
Penalty
Lasso
Still a weighted sum. Some weights land on exactly zero, and that feature drops out.
Function
Penalty
Regression tree
Yes or no questions cut the input into ranges. Each range predicts one number.
Function
k-NN regression
Find the nearest examples and average their numbers. The rows are the model.
Function
Intro: Selling a house
I'm an AI engineer selling my house for the best reasonable price. I gather data on houses in my region: area, rooms, and so on, plus price, the label. With that labeled data, the computer can find the function that predicts it best.
Each dot is a house.
Here's a line. It's bad.
A line predicts price from area. m is the slope: how fast price rises with area. p is the starting point: the price when area is zero. Feed in an area, the line predicts a price. Below is a bad first try: red predictions sit far from the green facts.
● real price·● line prediction
Bad line. Red predictions sit far from the green facts. Eyeball the gaps first. No total yet.
Zoom in: what error means for one house
Same 4th house from the intro: 74 m², 188k. Green is the real price. Red is what the bad line predicts for that area. The gap between them is the error.
Now do that for every house at once
Every house has a gap like that one. We don't want to shrink just one: we want to shrink all of them together. Square each gap so over- and under-predictions can't cancel out, and bigger misses count for more. Add them up. The better line has the smaller total.
● real price·● line prediction
Total Error
40985
Try. Measure. Adjust.
Now let the computer run the loop: start with a bad line, measure the squared gaps, adjust m and p, repeat until the error won't shrink further. Same game as last week: guess, check, correct. No new formula here, just the process.
The search
Same loop as last week. Each Next tries a line, measures the squared gaps, then adjusts. Watch the error getting smaller.
Press Next to try the first line.
More features, same idea
Real houses have more than area: rooms, location, age. Add features and the line becomes a plane through the cloud. Same try, measure, adjust.
Plane
Recap
Cheat sheet.
Recap only. No new ideas. The formulas you already met, lined up.
- 01
Line
You met this when the bad line first appeared. Slope and starting point.
- 02
Error
One gap first. Then every house, squared and summed.
- 03
More features
Extra inputs. The fit becomes a plane.
Linear regression predicts a number by fitting a line. The better line is the one with the smaller total of squared gaps.