Machine learning basics.
Three tasks. One framework.
Most "AI products" do one of three things: predict a number, predict a category, or find groups. Those are the three core tasks of machine learning — regression, classification, and clustering. This lesson covers all three, plus the two concepts that separate working models from broken ones: train/test split and overfitting. No math beyond what you learned in high school. Every example runs in your browser.
Three tasks cover 99% of machine learning: regression (predict a number), classification (predict a category), and clustering (find groups). Regression and classification are supervised (need labels); clustering is unsupervised (doesn't). The two practices that separate working models from broken ones: train/test split (evaluate on data the model hasn't seen) and avoiding overfitting (don't memorize the training data). Every ML engineer follows these rules — no exceptions.
Before this lesson: complete Lesson 1 for the vocabulary (AI vs ML vs DL, training vs inference, supervised vs unsupervised).
🎯 The three tasks of ML
Every ML problem is one of these. Knowing which one you have is the first step.
1. REGRESSION — predict a number ───────────────────────────────────── Input: Features (size, rooms, location) Output: A number (price: $`450,000) Examples: • Predict house price • Predict tomorrow's temperature • Predict delivery time • Predict a movie's rating Supervised? YES — needs labeled (input, number) pairs 2. CLASSIFICATION — predict a category ───────────────────────────────────── Input: Features (email text, image pixels, patient data) Output: A category from a fixed set (spam / not spam, cat / dog / bird) Examples: • Detect spam • Recognize faces • Diagnose disease • Classify sentiment Supervised? YES — needs labeled (input, category) pairs 3. CLUSTERING — find groups ───────────────────────────────────── Input: Features only (no labels) Output: Groups of similar items (customers split into 5 segments) Examples: • Customer segmentation • Anomaly detection • Document grouping • Gene expression analysis Supervised? NO — no labels needed QUICK DECISION TREE ───────────────────────────────────────────── Do you know the labels in advance? → YES → regression or classification → Output is a number? → regression → Output is a category? → classification → NO → clustering
Regression
Predict a continuous number. House prices, temperatures, ages, probabilities. The output is a value on a scale — not a category. If you can plot it on a number line, it's regression.
Classification
Predict a category from a fixed set. Spam/not-spam, cat/dog, positive/negative. The output is one of N labels. If you can list all possible answers, it's classification.
Clustering
Find natural groupings without labels. The algorithm discovers structure you didn't tell it to look for. Used to explore data, segment customers, and detect anomalies. No "right answer" — just useful groupings.
Why the distinction matters
Each task uses different algorithms, different loss functions, and different evaluation metrics. Using classification for a regression problem is a fundamental mistake. Knowing your task is step one of ML engineering.
When you're starting, classification is the best default. Most business problems are "which of these categories?" — is this spam, is this fraudulent, is this positive, is this at-risk. Regression is easier to explain but harder to make useful. Clustering is only worth it when you truly don't know what you're looking for. If you're unsure, ask: "Am I predicting a number, a category, or looking for patterns?" That question resolves 90% of "which ML task is this?"
📈 Regression — predict a number
The simplest form: fit a line through data. Everything else is a variation.
# The entire algorithm in 20 lines.
Linear regression finds the line y = mx + b
that minimizes the distance from each point to the line.
import numpy as np
Training data: house size (sqft) → price (`$thousands)
X = np.array([1000, 1500, 2000, 2500, 3000]) # features
y = np.array([300, 350, 400, 500, 550]) # targets
Simple linear regression: y = mx + b
m, b = np.polyfit(X, y, 1) # fit a degree-1 (linear) polynomial
print(f"Formula: price = {m:.2f} × size + {b:.2f}")
Predict for a new house
new_size = 1800
predicted = m * new_size + b
print(f"A {new_size} sqft house costs $`{predicted:.0f}k")
Evaluate — mean absolute error
predictions = m * X + b
errors = np.abs(predictions - y)
print(f"Mean absolute error: `${errors.mean():.1f}k")
What just happened:
The algorithm found the m and b that minimize
the average vertical distance from each point
to the line. That's ordinary least squares.
What it does
Fits a line (or curve) through data points. For any new input, the model reads the line's value. Simple, fast, interpretable — you can explain exactly why the model predicted what it did.
The math
Minimize the sum of squared vertical distances from points to line. That's "ordinary least squares." The result is a slope (m) and intercept (b) — two numbers that define the entire model.
When to use it
When the relationship is roughly linear, when you need explainability, or as a baseline before trying complex models. If a linear model gets 80% of the way, a neural network usually isn't worth the complexity.
Logistic regression
Despite the name, this is classification — it predicts a probability (0-1) and classifies based on a threshold. The "regression" name is historical. It's used for spam, disease diagnosis, fraud detection.
Linear regression is the "hello world" of ML — and it's not a toy. Many production models are linear regression. It's interpretable, fast, works with small data, and gives you a baseline that fancier models have to beat to justify their complexity. Every ML engineer starts with linear regression. If a linear model solves the problem, ship it. Save the deep learning for problems linear regression can't touch.
🏷️ Classification — predict a category
The most-used ML task. Spam filters, medical diagnosis, image recognition, sentiment.
# Classification example: spam detection
Input: number of "spammy" words in the email
Output: 1 (spam) or 0 (not spam)
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
Dataset: [spam_word_count] → is_spam
X = np.array([[15], [3], [20], [1], [12], [2], [8], [0]]) # word counts
y = np.array([1, 0, 1, 0, 1, 0, 0, 0]) # spam labels
Train/test split — critical step
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42
)
75% for training, 25% for testing
Train the model
model = LogisticRegression()
model.fit(X_train, y_train)
Predict on NEW emails
new_emails = np.array([[18], [1], [10]])
predictions = model.predict(new_emails)
probabilities = model.predict_proba(new_emails)[:, 1]
for count, pred, prob in zip(new_emails[:, 0], predictions, probabilities):
label = "SPAM" if pred == 1 else "NOT SPAM"
print(f"{count} spam words → {label} (prob: {prob:.2f})")
Evaluate on test set — data the model has NOT seen
accuracy = model.score(X_test, y_test)
print(f"Test accuracy: {accuracy:.2f}")
What just happened:
1. Split data into train/test
2. Fit a logistic regression — it found weights that
separate spam from not-spam
3. Predicted on NEW emails the model has never seen
4. Evaluated on the test set — this is the real metric
Key insight: the model learned a threshold
below X spam words → not spam
above X spam words → spam
The threshold was found automatically from the data.
Binary classification
Two classes: spam/not-spam, fraud/legit, positive/negative. The most common type. Output is a single probability (0 to 1) and a threshold-based label.
Multi-class
Three or more classes: cat/dog/bird, sentiment negative/neutral/positive, disease A/B/C. Output is a probability for each class. The highest-probability class wins.
Multi-label
Each item can have multiple labels: a news article can be about "technology" AND "politics." Not the same as multi-class (where only one label applies). Less common than binary or multi-class.
Evaluation metrics
Accuracy (correct / total), precision (of what it flagged, how many were right), recall (of what was right, how much it caught), F1 (balance of both). Choose based on what matters more: false positives or false negatives.
Accuracy is a misleading metric for imbalanced data. If 99% of emails are not spam and you build a model that always predicts "not spam," it gets 99% accuracy — and catches zero spam. This is why precision, recall, and F1 exist. Use the metric that matches what the business cares about. For spam filtering, precision matters (don't block real email). For disease screening, recall matters (don't miss cases). Choose deliberately.
🔍 Clustering — find groups without labels
The unsupervised task. No "right answer" — just useful structure.
# Clustering example: customer segmentation
Input: customer behavior (no labels)
Output: groups of similar customers
from sklearn.cluster import KMeans
import numpy as np
Dataset: [annual_spend, visits_per_month]
Note: no labels — we don't know the groups yet
customers = np.array([
[150, 20], [200, 22], [180, 18], # high spenders, frequent
[50, 2], [45, 3], [55, 2], # low spenders, rare
[100, 10], [90, 12], [110, 11], # medium spenders
])
Cluster into 3 groups
kmeans = KMeans(n_clusters=3, random_state=42)
kmeans.fit(customers)
Assign each customer to a cluster
labels = kmeans.labels_
centers = kmeans.cluster_centers_
print("Cluster centers:")
for i, center in enumerate(centers):
spend, visits = center
print(f" Cluster {i}: ${spend:.0f}/year, {visits:.1f} visits/mo")
print("\nCustomer assignments:")
for customer, label in zip(customers, labels):
print(f" {customer} → cluster {label}")
What just happened:
K-means found 3 groups by minimizing the distance
from each customer to its cluster center.
The "high spenders" cluster emerged automatically.
No one told the algorithm about "high spenders" —
it discovered that pattern in the data.
Business use: now you can target each segment differently
- Cluster 0: premium (high spend, frequent) → loyalty program
- Cluster 1: occasional (low spend, rare) → reactivation campaign
- Cluster 2: regular (medium spend) → upsell to premium
K-means
The most common clustering algorithm. Pick a number of clusters (k), and it groups items by minimizing the distance to cluster centers. Fast, simple, but you have to choose k.
How to choose k
Trial and error with the elbow method (plot error vs k — the "elbow" is a good k), or domain knowledge (3 tiers of customers, 5 product categories), or business need (2 segments is easy to explain).
Anomaly detection
Clustering finds "normal" groups. Anything far from every cluster center is an anomaly. Used for fraud detection, system monitoring, and quality control. Isolation Forest is a modern alternative.
No "right answer"
Unlike classification, clustering has no ground truth to compare against. You evaluate by business usefulness — does this segmentation lead to better actions? Different algorithms give different groupings. Try a few, choose the one that makes sense.
Clustering is for exploration, not prediction. Use it when you want to understand your data, not when you want to make a prediction. "Which segment is this customer in?" is a clustering question (unsupervised). "Will this customer churn?" is a classification question (supervised) — you need labels for the second. Knowing the difference prevents a lot of wasted work.
✂️ Train/test split — the most important rule
Evaluate on data the model has never seen. Anything else is self-deception.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np
Simulate a dataset: 1000 samples, 5 features
np.random.seed(42)
X = np.random.randn(1000, 5)
y = (X[:, 0] + X[:, 1] > 0).astype(int) # some pattern
❌ WRONG — evaluate on the same data you trained on
model_wrong = LogisticRegression()
model_wrong.fit(X, y)
accuracy_wrong = model_wrong.score(X, y)
print(f"Wrong way: {accuracy_wrong:.3f}") # looks great, but...
✅ CORRECT — split first, evaluate on unseen data
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
80% for training, 20% for testing
model_right = LogisticRegression()
model_right.fit(X_train, y_train)
accuracy_right = model_right.score(X_test, y_test)
print(f"Correct way: {accuracy_right:.3f}") # the real metric
Why the "wrong" way looks better:
The model has already seen those exact points.
It memorized them. That's not generalization.
The "correct" way tests generalization —
how well the model will do on future data.
The test set is a proxy for the real world.
More rigorous: cross-validation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model_right, X, y, cv=5)
print(f"Cross-validation: {scores.mean():.3f} ± {scores.std():.3f}")
Trains and evaluates 5 times on different splits
More reliable estimate of real-world performance
The rule
Always split your data before training. Train on one part, evaluate on another. The test set represents the "real world" — data the model has never seen. It's the only honest measure of performance.
Typical split
80/20 (train/test) for most problems. 70/15/15 (train/validation/test) for complex problems — the validation set is used for tuning hyperparameters, the test set for final evaluation only.
Cross-validation
Split into K folds, train on K-1, test on 1, rotate. Average the results. More reliable than a single split, especially with small datasets. K=5 or K=10 are common.
Never peek at test data
Don't use the test set to make decisions (which model to pick, which features to use). Every peek makes the test result less honest. The test set is sacred — use it once, at the end.
Data leakage is the #1 mistake in applied ML. It happens when information from the test set leaks into training. Common forms: using test data to normalize features, using future information in time-series data, or including the target variable as a feature. The model looks perfect on paper and fails in production. If your test accuracy is 99%+, suspect leakage before celebrating. Real-world ML is rarely that clean.
⚠️ Overfitting — the model that memorized instead of learning
Great on training data, terrible on new data. The universal enemy of ML.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import numpy as np
Small dataset — easy to overfit
np.random.seed(42)
X = np.random.randn(100, 2)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
Try different tree depths
for depth in [1, 2, 3, 5, 10, 20, None]:
model = DecisionTreeClassifier(max_depth=depth)
model.fit(X_train, y_train)
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
gap = train_acc - test_acc
print(f"depth={str(depth):5} train={train_acc:.2f} "
f"test={test_acc:.2f} gap={gap:.2f}")
Typical output:
depth=1 train=0.83 test=0.80 gap=0.03 ← underfit, too simple
depth=2 train=0.91 test=0.87 gap=0.04 ← getting better
depth=3 train=0.94 test=0.90 gap=0.04 ← sweet spot
depth=5 train=0.97 test=0.87 gap=0.10 ← starting to overfit
depth=10 train=0.99 test=0.83 gap=0.16 ← overfitting
depth=20 train=1.00 test=0.80 gap=0.20 ← definitely overfit
depth=None train=1.00 test=0.77 gap=0.23 ← worst (no limit)
What to notice:
1. Training accuracy always goes UP with more complexity
2. Test accuracy goes up, then DOWN
3. The GAP is the warning sign
4. The sweet spot (depth=3) has high test accuracy
and small gap — the model learned general patterns,
not just memorized the training data
What overfitting is
The model memorizes training data instead of learning general patterns. It performs perfectly on training examples but fails on new ones. Like memorizing test answers without understanding the material.
How to spot it
Training accuracy is much higher than test accuracy. The gap (train − test) grows as the model gets more complex. A 20% gap means the model has memorized, not learned.
How to fix it
More training data, simpler model (fewer layers, fewer parameters), regularization (L1, L2), early stopping (stop training when test accuracy stops improving), dropout, data augmentation.
Underfitting
The opposite problem — the model is too simple to capture the pattern. Both training and test accuracy are low. Fix: more complex model, more features, longer training, better architecture.
Watch the gap between train and test accuracy — not the training accuracy itself. Training accuracy almost always improves with more complexity. It's meaningless in isolation. What matters is how well the model generalizes: does the test accuracy keep up? If train accuracy is 99% and test is 75%, the model has overfit. If train accuracy is 82% and test is 80%, you have a good model — even though the training accuracy is lower. This is the single most important concept in applied ML.
🏆 Challenge
Three tasks. Each exercises a different ML concept.
Task 1 — Classify the task
For each problem, identify the ML task (regression, classification, or clustering) and whether it's supervised or unsupervised:
1. Predict tomorrow's high temperature
2. Group news articles by topic
3. Detect whether a credit card transaction is fraudulent
4. Estimate a car's value from age, mileage, and condition
5. Segment customers into marketing personas
6. Predict whether a customer will churn
7. Find unusual network traffic patterns
For each one, also answer: what would be a good evaluation metric?
Task 2 — Diagnose overfitting
Here are three model results. Diagnose each:
Model A: train=0.72, test=0.70
Model B: train=0.99, test=0.65
Model C: train=0.65, test=0.62
1. Which is overfit?
2. Which is underfit?
3. Which is well-fitted?
4. What would you do to improve each?
Hint: look at the gap and the absolute numbers.
Task 3 — Design an ML project
Pick one of these problems and design the full ML approach:
1. Predict whether a student will drop out
2. Classify support tickets by urgency
3. Detect manufacturing defects in product photos
4. Recommend movies to a new user
For your chosen problem, answer:
• Regression, classification, or clustering?
• What features (inputs) would you use?
• How would you get labels?
• How would you split the data?
• What metrics would you measure?
• What could go wrong?
• How would you know the model is good enough to ship?
This is exactly the plan a real ML engineer writes before any code. The design matters more than the algorithm.
⚠️ Common mistakes
The ML bugs that catch every beginner.
Training accuracy is meaningless on its own. You must evaluate on data the model hasn't seen. Split first, always. No exceptions.
Information from the test set leaks into training — through normalization, feature selection, or time-series ordering. The model looks great on paper and fails in production. Sanity check: if test accuracy is 99%+, suspect leakage.
Using classification for a regression problem (or vice versa). The math, metrics, and behavior are different. Ask: number, category, or grouping?
99% of emails are not spam. A model that always predicts "not spam" gets 99% accuracy and catches zero spam. Fix: use precision, recall, or F1. Or resample (SMOTE, undersampling).
High training accuracy hides the problem. Watch the gap. If train-test gap grows while training, you're overfitting. Fix: simplify, add data, regularize.
Jumping straight to a neural network. Start with linear regression or a decision tree — if those work, they're cheaper, faster, and easier to explain. Only escalate complexity when the baseline fails.
Accuracy is misleading on imbalanced data. For fraud detection, prioritize recall. For spam, prioritize precision. For ranking, use NDCG or MAP. Choose based on what the business cares about, not what's easy to compute.
A single train/test split can be lucky or unlucky. Cross-validation (K=5 or 10) gives a more reliable estimate. Use it for anything you'll actually deploy.
➡️ What's next
You know the three tasks. Next: the machinery that powers the most powerful ones.
❓ Frequently asked questions
What is regression in machine learning?
Regression predicts a continuous number. Given input features, the model outputs a value on a continuous scale — price, temperature, age, probability. Linear regression fits a line through data points. Regression is used for anything numeric: house prices, stock prices, delivery time, patient outcomes.
What is classification in machine learning?
Classification predicts a category. Given input features, the model outputs one of a fixed set of labels — spam/not spam, cat/dog/bird, positive/negative/neutral. Binary classification has two classes; multi-class has three or more. It's the most common ML task in production.
What is the difference between regression and classification?
Regression predicts a number (continuous output). Classification predicts a category (discrete output). Predicting tomorrow's temperature: regression. Predicting whether it will rain: classification. The underlying math is different — regression uses least squares, classification uses cross-entropy.
What is clustering and how is it different from classification?
Clustering groups similar items together — but without predefined labels. Classification needs labeled examples. Clustering doesn't — it just discovers natural groupings in data. Used for customer segmentation, anomaly detection, and data exploration.
What is a train/test split and why does it matter?
You split your data into two sets: training (usually 70-80%) and test (20-30%). The model trains on the training set, then you evaluate it on the test set — data it has never seen. Without a test set, you're evaluating on data the model memorized, which gives falsely optimistic results. It's the single most important practice in ML.
What is overfitting in machine learning?
Overfitting is when a model memorizes the training data instead of learning general patterns. It performs perfectly on training data but poorly on new data. Causes: model too complex, not enough training data, training too long. Fixes: simpler model, more data, regularization, early stopping, cross-validation.
How much training data do I need?
For simple linear regression: 100+ examples. For a good classifier: 1,000+ per class. For deep learning: often 10,000+ per class. For LLMs: billions of examples. The general rule: more data is better, but eventually you hit diminishing returns.
What is the difference between parameters and hyperparameters?
Parameters are what the model learns during training — the weights in a neural network, the coefficients in linear regression. Hyperparameters are what you set before training — learning rate, number of layers, batch size, regularization strength. Parameters are learned; hyperparameters are chosen.