Tag supervised learning

Conceptual Introduction & Fundamentals

Intuition First: Nested Circles, Not Synonyms

The three terms are not variations on a theme. They are nested categories, like Russian dolls.

text

 ┌──────────────────────────────────────────────────────┐ │ ARTIFICIAL INTELLIGENCE (AI) │ │ Any technique that makes a machine act intelligently│ │ ┌────────────────────────────────────────────────┐ │ │ │ MACHINE LEARNING (ML) │ │ │ │ Systems that improve at a task by learning │ │ │ │ from data │ │ │ │ ┌───────────────────────────────────────┐ │ │ │ │ │ DEEP LEARNING (DL) │ │ │ │ │ │ ML using multi-layer neural networks │ │ │ │ │ └───────────────────────────────────────┘ │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘

The single most useful mental model:

AI is the goal. ML is a way of reaching it. DL is a way of doing ML.

A chess engine using alpha-beta search is AI but not ML — it was hand-coded, and it never “learns” from games. A spam filter trained on labeled emails is AI and ML. An image classifier with 50 convolutional layers is AI and ML and DL.

Why It Matters

The distinction is not academic. It determines how much data you need, how much compute you need, how interpretable your model is, and how long your project will take.

If your problem is…	Use	Because
A clearly specifiable rule (tax calculation, unit conversion)	Symbolic AI / plain code	ML adds risk and cost for zero benefit
Tabular data with 1k–1M rows	Classical ML (trees, linear models, boosting)	SOTA on tabular; interpretable; fast to train
Images, audio, raw text, video	Deep Learning	Learns features that would take years to engineer
Sequential decisions with reward	Reinforcement Learning (a branch of ML, often DL)	No labeled dataset exists
Small data, high stakes, regulatory audit needed	Classical ML (linear/logistic)	Coefficients are explainable to a regulator

Real-world anchor: In 2012, AlexNet (a deep CNN) cut the ImageNet top-5 error rate from 26% to 15% in one competition. That single result triggered the modern DL era. But in 2024, a well-tuned gradient-boosted tree still beats a neural network on most tabular benchmarks with fewer than ~100k rows. “Deep” is not automatically “better.”

Key Terminology

• Symbolic AI / GOFAI — Good Old-Fashioned AI. Systems whose behavior is written by humans as explicit rules or search procedures. Expert systems, logic programming, A* search.
• Machine Learning — algorithms that infer a mapping $f:X\to Y$f:X→Y from examples, without the mapping being written by hand.
• Feature engineering — human-designed transformations of raw input into informative variables. Classical ML depends on it heavily.
• Representation learning — a model learning its own features from raw input. DL’s defining capability.
• Neural network — a composition of parameterized linear maps and element-wise non-linearities.
• Deep — a neural network with many layers (historically ≥ 3; in practice, dozens to hundreds).
• Inductive bias — the assumptions a learning algorithm makes about the form of $f$f. CNNs assume spatial locality; RNNs assume sequential order; transformers assume permutation-equivariance with attention.

The Critical Difference: Feature Engineering vs. Representation Learning

This is where DL genuinely separates from classical ML.

Classical ML pipeline:$$\text{raw data}\stackrel{\text{human}}{\to }\text{features}\stackrel{\text{algorithm}}{\to }\text{prediction}$$raw datahuman​featuresalgorithm​prediction

Deep Learning pipeline:$$\text{raw data}\stackrel{\text{learned layers}}{\to }\text{features}\stackrel{\text{learned layers}}{\to }\text{prediction}$$raw datalearned layers​featureslearned layers​prediction

In DL, feature extraction is part of the model. In classical ML, it is a separate, human-driven step. This is why DL dominates images and text — hand-designing good features for a 224×224×3 image is effectively impossible, but a CNN discovers edges → textures → parts → objects on its own.

Mathematical Formulation

Classical ML (empirical risk minimization). Given a hypothesis class $\mathcal{H}$H (e.g., all linear functions, all depth-3 trees):$$\widehat{f}=\arg \underset{f\in \mathcal{H}}{\min }\frac{1}{n}\sum _{i=1}^{n}L(y_i,f(x_i))+\lambda \mathrm{\Omega }(f)$$f^​=argf∈Hmin​n1​i=1∑n​L(yi​,f(xi​))+λΩ(f)

The hypothesis class is chosen by a human. The model is fit by the algorithm.

Deep Learning (compositional function approximation). A feed-forward network of depth $L$L computes:$$\widehat{y}=f_L\circ f_{L-1}\circ \cdots \circ f_1(x),f_{\mathrm{\ell }}(z)={\sigma }_{\mathrm{\ell }}(W_{\mathrm{\ell }}z+b_{\mathrm{\ell }})$$y^​=fL​∘fL−1​∘⋯∘f1​(x),fℓ​(z)=σℓ​(Wℓ​z+bℓ​)

with ${\sigma }_{\mathrm{\ell }}$σℓ​ a non-linear activation (ReLU, GELU, etc.). The parameters $\{W_{\mathrm{\ell }},b_{\mathrm{\ell }}{\}}_{\mathrm{\ell }=1}^L${Wℓ​,bℓ​}ℓ=1L​ are learned end-to-end by gradient descent on the loss:$$W_{\mathrm{\ell }}\leftarrow W_{\mathrm{\ell }}-\eta {\mathrm{\nabla }}_{W_{\mathrm{\ell }}}L(y,\widehat{y})$$Wℓ​←Wℓ​−η∇Wℓ​​L(y,y^​)

Why non-linearity is essential. Without $\sigma$σ, the composition collapses:$$W_2(W_{1}x)=(W_2{W}_1)x=Wx$$W2​(W1​x)=(W2​W1​)x=Wx

— the whole network is a single linear map, no matter how many layers you stack. The non-linear activation is what makes depth meaningful.

Universal Approximation Theorem (Cybenko 1989, Hornik 1991). A feed-forward network with a single hidden layer of sufficient width can approximate any continuous function on a compact domain to arbitrary accuracy. But this is an existence theorem, not a learnability one — it says nothing about how much data or how many iterations you need. Depth delivers the same expressiveness with exponentially fewer parameters, which is why modern networks are deep rather than wide.

Parameter count comparison (for intuition):

Architecture	Layers	Params (approx.)
Logistic regression on 64 features	1 linear	650
2-layer MLP (64 → 128 → 10)	3	~10k
ResNet-50 (ImageNet)	50	25.6M
GPT-3	96	175B

The jump from ~${10}^4$104 to ~${10}^{11}$1011 parameters is why DL needed new hardware (GPUs, TPUs), new data (web-scale corpora), and new training tricks (dropout, batch norm, residual connections).

---

H2: Syntax & Core Mechanics

Required Libraries & Environment

bash

pip install "numpy>=1.26" "pandas>=2.1" "scikit-learn>=1.4" "torch>=2.2" matplotlib

python

import sys, numpy, pandas, sklearn print(f"Python : {sys.version.split()[0]}") # 3.10+ print(f"NumPy : {numpy.__version__}") # 1.26.x print(f"pandas : {pandas.__version__}") # 2.1.x print(f"scikit-learn : {sklearn.__version__}") # 1.4.x import torch print(f"PyTorch : {torch.__version__}") # 2.2+ print(f"CUDA avail. : {torch.cuda.is_available()}")

The Three Paradigms Side by Side

Aspect	Rule-Based AI	Classical ML	Deep Learning
Who designs the mapping?	Human writes it	Human picks the class; data fits it	Data fits everything end-to-end
Human effort	High (rule-writing)	Medium (feature engineering)	Low (architecture + tuning)
Data needed	None	Hundreds to millions	Millions to billions
Compute needed	Trivial	CPU-friendly	GPU often required
Interpretability	High (rules are readable)	Medium–high (trees, linear)	Low (distributed weights)
Failure mode	Brittle to novel inputs	Misses interactions	Overfits, needs data, hard to debug
Best for	Well-specified logic	Tabular, structured data	Images, audio, text, video

The ML/DL Pipeline Compared

text

CLASSICAL ML PIPELINE ───────────────────────────────────────────────────────────── raw data │ ▼ [HUMAN] feature engineering ← the bottleneck │ ▼ standardized features │ ▼ [ALGORITHM] fit model (logistic regression, RF, XGBoost) │ ▼ predictions DEEP LEARNING PIPELINE ───────────────────────────────────────────────────────────── raw data │ ▼ tensor / DataLoader │ ▼ [MODEL] forward pass ← features learned inside │ (layers + non-linearities) ▼ loss │ ▼ [OPTIMIZER] backward pass ← gradient descent │ ▼ parameter update ──► repeat over epochs │ ▼ predictions

Design Choices Per Paradigm

Rule-based AI:

• Which features to extract (must be human-computable from raw input)
• Thresholds for each rule
• Rule ordering (first match wins vs. weighted vote)

Classical ML:

• Which model class (LogisticRegression, RandomForestClassifier, SVC, GradientBoostingClassifier)
• Which features to engineer
• Regularization strength (C for SVM/logreg, alpha for ridge)
• Preprocessing: scaling (needed for SVM/k-NN, not for trees), encoding, imputation

Deep Learning:

• Architecture (MLP, CNN, RNN, transformer)
• Depth and width
• Activation function (ReLU, GELU, SiLU)
• Optimizer (AdamW, SGD+momentum)
• Learning rate schedule (warmup + cosine decay)
• Batch size and gradient accumulation
• Regularization (Dropout, weight_decay, BatchNorm)
• Early stopping patience

Essential Code Skeletons

Rule-based prediction:

python

def rule_based_predict(X: np.ndarray) -> np.ndarray: """Apply hand-written if-then rules. No learning occurs here.""" features = extract_features(X) # human-designed preds = np.full(len(X), DEFAULT_CLASS) for rule in RULES: # ordered list of (mask_fn, label) mask = rule.condition(features) preds[mask] = rule.label return preds

Classical ML:

python

from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression clf = Pipeline([ ("scaler", StandardScaler()), ("clf", LogisticRegression(max_iter=1000, random_state=42)), ]) clf.fit(X_train, y_train) y_pred = clf.predict(X_test)

Deep Learning (PyTorch):

python

import torch, torch.nn as nn from torch.utils.data import DataLoader, TensorDataset model = nn.Sequential( nn.Linear(in_dim, 128), nn.ReLU(), nn.Dropout(0.2), nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, n_classes), ) optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4) criterion = nn.CrossEntropyLoss() for epoch in range(n_epochs): model.train() for xb, yb in train_loader: optimizer.zero_grad() loss = criterion(model(xb), yb) loss.backward() optimizer.step()

Note the difference: in classical ML, fit() is one call. In DL, you write the training loop. This is not incidental — it is the core reason DL is harder to productionize.

---

Full Practical Code Example with Explanation

The Experiment

We will train three classifiers on the exact same data — sklearn.datasets.load_digits (1,797 handwritten digit images, 8×8 grayscale, 10 classes) — and measure test accuracy for each. This makes the AI/ML/DL distinction empirical rather than rhetorical.

• Level 1 — Rule-based AI: hand-written if/else rules on human-designed shape features.
• Level 2 — Classical ML: logistic regression, random forest, and SVM on raw pixels.
• Level 3 — Deep Learning: a small PyTorch MLP on raw pixels.

Reproducibility note: All printed values below come from one run with random_state=42, torch.manual_seed(42), NumPy 1.26, scikit-learn 1.4, PyTorch 2.2, CPU-only. Expect the last 1–2 decimals to shift across library versions or hardware; the conclusions are stable.

Complete Self-Contained Script

python

# ============================================================================ # ai_vs_ml_vs_dl.py # Python 3.10+ | scikit-learn >= 1.4 | torch >= 2.2 # # Trains three classifiers on the same dataset: # Level 1 — Rule-based AI (hand-written if-then rules) # Level 2 — Classical ML (LogReg, RandomForest, SVM) # Level 3 — Deep Learning (PyTorch MLP) # ============================================================================ from __future__ import annotations import os import random import warnings from dataclasses import dataclass import numpy as np import pandas as pd SEED: int = 42 def set_global_seed(seed: int = SEED) -> None: """Make every RNG reachable from Python deterministic.""" os.environ["PYTHONHASHSEED"] = str(seed) os.environ["OMP_NUM_THREADS"] = "1" random.seed(seed) np.random.seed(seed) set_global_seed(SEED) warnings.filterwarnings("ignore", category=FutureWarning) import torch import torch.nn as nn from torch.utils.data import DataLoader, TensorDataset from sklearn.datasets import load_digits from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import LogisticRegression from sklearn.metrics import ( accuracy_score, classification_report, confusion_matrix, ) from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.svm import SVC # torch also needs a seed for every source of randomness torch.manual_seed(SEED) if torch.cuda.is_available(): torch.cuda.manual_seed_all(SEED) torch.use_deterministic_algorithms(False) # full determinism costs speed on CPU @dataclass(frozen=True) class Audit: n_rows: int n_cols: int n_classes: int missing: int non_finite: int def audit(X: np.ndarray, y: np.ndarray) -> Audit: """Hard integrity gates — fail loudly.""" if X.ndim != 2: raise ValueError(f"X must be 2-D, got shape {X.shape}.") if len(X) != len(y): raise ValueError("X and y have mismatched lengths.") if not np.isfinite(X).all(): raise ValueError("X contains NaN or inf.") return Audit(len(X), X.shape[1], len(np.unique(y)), int(np.isnan(X).sum()), int((~np.isfinite(X)).sum())) def section(title: str) -> None: print("\n" + "=" * 72) print(title) print("=" * 72) # ============================================================================ # STEP 1 — Load and audit the dataset # ============================================================================ section("STEP 1 — LOAD & AUDIT") digits = load_digits() X: np.ndarray = digits.data.astype(np.float64) # (1797, 64), values 0..16 y: np.ndarray = digits.target.astype(np.int64) # (1797,), labels 0..9 a = audit(X, y) print(f"shape : {X.shape}") print(f"classes : {a.n_classes} -> {sorted(np.unique(y).tolist())}") print(f"pixel value range : [{X.min():.1f}, {X.max():.1f}]") print(f"missing / non-finite : {a.missing} / {a.non_finite}") print(f"class counts : {dict(zip(*np.unique(y, return_counts=True)))}") print(f"majority baseline : {np.bincount(y).max() / len(y):.4f}") # Split ONCE. Test set stays sealed until final evaluation. X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.20, stratify=y, random_state=SEED ) print(f"train / test : {X_train.shape} / {X_test.shape}")

Expected output — Step 1

text

======================================================================== STEP 1 — LOAD & AUDIT ======================================================================== shape : (1797, 64) classes : 10 -> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] pixel value range : [0.0, 16.0] missing / non-finite : 0 / 0 class counts : {0: 178, 1: 182, 2: 177, 3: 183, 4: 181, 5: 182, 6: 181, 7: 179, 8: 174, 9: 180} majority baseline : 0.1018 train / test : (1437, 64) / (360, 64)

Class balance is near-uniform, so majority baseline is 10.18% — barely better than random guessing on 10 classes. Every model must beat this, and by a wide margin.

python

# ============================================================================ # LEVEL 1 — RULE-BASED "AI" (Symbolic, no learning) # ============================================================================ section("LEVEL 1 — RULE-BASED AI (hand-written if/then rules)") def extract_shape_features(X_raw: np.ndarray) -> dict[str, np.ndarray]: """Human-designed features from 8x8 digit images. No learning here.""" imgs = X_raw.reshape(-1, 8, 8) total_ink = imgs.sum(axis=(1, 2)) rows_with_ink = (imgs.sum(axis=2) > 2).sum(axis=1) # vertical extent cols_with_ink = (imgs.sum(axis=1) > 2).sum(axis=1) # horizontal extent top_ink = imgs[:, :4, :].sum(axis=(1, 2)) bot_ink = imgs[:, 4:, :].sum(axis=(1, 2)) left_ink = imgs[:, :, :4].sum(axis=(1, 2)) right_ink = imgs[:, :, 4:].sum(axis=(1, 2)) return { "total_ink": total_ink, "rows_with_ink": rows_with_ink, "cols_with_ink": cols_with_ink, "top_ink": top_ink, "bot_ink": bot_ink, "left_ink": left_ink, "right_ink": right_ink, } def rule_based_predict(X_raw: np.ndarray, default_class: int = 5) -> np.ndarray: """Apply ordered if-then rules. The mapping is entirely human-authored.""" f = extract_shape_features(X_raw) n = len(X_raw) preds = np.full(n, default_class, dtype=np.int64) # Rule 1 — very low ink + tall narrow shape => 1 m = (f["total_ink"] < 40) & (f["rows_with_ink"] >= 5) & (f["cols_with_ink"] <= 4) preds[m] = 1 # Rule 2 — heavy top half, light bottom half => 7 m = (f["top_ink"] > 2.0 * f["bot_ink"]) & (f["total_ink"] > 35) & (f["total_ink"] < 95) preds[m] = 7 # Rule 3 — very high ink => 8 m = f["total_ink"] > 100 preds[m] = 8 # Rule 4 — medium-high ink, wide shape => 0 m = (f["total_ink"] > 70) & (f["total_ink"] <= 100) & (f["cols_with_ink"] >= 5) preds[m] = 0 # Rule 5 — left-heavy, medium ink => 5 m = (f["left_ink"] > 1.6 * f["right_ink"]) & (f["total_ink"] > 40) & (f["total_ink"] < 90) preds[m] = 5 # Rule 6 — strong left-right asymmetry, low ink => 4 m = (f["left_ink"] > 2.0 * f["right_ink"]) & (f["total_ink"] <= 50) preds[m] = 4 return preds y_pred_rule = rule_based_predict(X_test) rule_acc = accuracy_score(y_test, y_pred_rule) print(f"Test accuracy (rule-based AI) : {rule_acc:.4f}") print(f"Train rows seen : 0 (rules were written, not learned)")

Expected output — Level 1

text

======================================================================== LEVEL 1 — RULE-BASED AI (hand-written if/then rules) ======================================================================== Test accuracy (rule-based AI) : 0.4028 Train rows seen : 0 (rules were written, not learned)

Interpretation

• 40.28% accuracy on 10 classes is 4× better than random, but far below what ML achieves. The rules do capture real signal (the “1” and “7” detectors are reasonably reliable), but every edge case breaks them.
• Crucially, the classifier never saw the training set. This is the essence of symbolic AI: the mapping lives in the engineer’s head, not in a parameter vector.
• The failure mode is brittleness. Change the image resolution, the font, or the scan brightness, and every threshold breaks. This is why expert systems lost to ML for perceptual tasks in the 1990s.
• When rule-based AI is still the right answer: tax engines, unit converters, business workflow logic, safety interlocks. If the rule is specifiable, hand-write it. Do not train a neural network to compute if age < 18.

python

# ============================================================================ # LEVEL 2 — CLASSICAL MACHINE LEARNING # ============================================================================ section("LEVEL 2 — CLASSICAL ML (scikit-learn)") candidates: dict[str, Pipeline] = { "Logistic Regression": Pipeline([ ("scaler", StandardScaler()), ("clf", LogisticRegression(max_iter=2000, random_state=SEED)), ]), "Random Forest (300)": Pipeline([ ("clf", RandomForestClassifier( n_estimators=300, n_jobs=-1, random_state=SEED)), ]), "SVM RBF (C=10)": Pipeline([ ("scaler", StandardScaler()), ("clf", SVC(C=10.0, gamma="scale", kernel="rbf", random_state=SEED)), ]), } cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED) print(f"\n{'model':<22} {'cv_mean':>9} {'cv_std':>8}") print("-" * 42) cv_scores: dict[str, float] = {} for name, pipe in candidates.items(): scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="accuracy", n_jobs=-1) cv_scores[name] = scores.mean() print(f"{name:<22} {scores.mean():>9.4f} {scores.std():>8.4f}") best_ml_name = max(cv_scores, key=cv_scores.get) # type: ignore[arg-type] best_ml = candidates[best_ml_name] best_ml.fit(X_train, y_train) y_pred_ml = best_ml.predict(X_test) ml_acc = accuracy_score(y_test, y_pred_ml) print(f"\nBest ML model (by CV) : {best_ml_name}") print(f"Test accuracy (ML) : {ml_acc:.4f}")

Expected output — Level 2

text

======================================================================== LEVEL 2 — CLASSICAL ML (scikit-learn) ======================================================================== model cv_mean cv_std ------------------------------------------ Logistic Regression 0.9527 0.0098 Random Forest (300) 0.9722 0.0071 SVM RBF (C=10) 0.9729 0.0063 Best ML model (by CV) : SVM RBF (C=10) Test accuracy (ML) : 0.9750

Interpretation

• Every classical ML model more than doubles the rule-based accuracy. SVM RBF hits 97.50% vs. the rule-based 40.28%.
• How did SVM do this? It projected the 64 raw pixels into a high-dimensional kernel space and found a maximum-margin separating surface. No human wrote a single rule. The mapping was learned from 1,437 examples.
• Cross-validation and test agree closely (CV 0.9729 vs. test 0.9750), which is strong evidence that the model generalizes rather than having been selected by luck.
• Why SVM beats logistic regression here: the decision boundary between handwritten digits is non-linear. LogReg’s linear boundary in pixel space is not flexible enough; SVM’s RBF kernel is.
• Every model still uses the raw 64 pixels. No feature engineering was done. The “feature engineering” step in classical ML pipelines becomes essential when raw inputs are large (e.g., raw text, raw audio), not when they are 64 numbers.

python

# ============================================================================ # LEVEL 3 — DEEP LEARNING (PyTorch MLP) # ============================================================================ section("LEVEL 3 — DEEP LEARNING (PyTorch MLP)") # Scale pixels to [0, 1] — improves optimization; fitted on TRAIN only. scaler_dl = StandardScaler().fit(X_train) X_train_s = scaler_dl.transform(X_train).astype(np.float32) X_test_s = scaler_dl.transform(X_test).astype(np.float32) # Tensors + DataLoaders with a seeded generator for reproducible shuffling X_train_t = torch.from_numpy(X_train_s) y_train_t = torch.from_numpy(y_train) X_test_t = torch.from_numpy(X_test_s) y_test_t = torch.from_numpy(y_test) g = torch.Generator().manual_seed(SEED) train_loader = DataLoader( TensorDataset(X_train_t, y_train_t), batch_size=64, shuffle=True, generator=g, num_workers=0, ) test_loader = DataLoader( TensorDataset(X_test_t, y_test_t), batch_size=256, shuffle=False, num_workers=0, ) class MLP(nn.Module): """A small multi-layer perceptron: 64 -> 128 -> 64 -> 10.""" def __init__(self, in_dim: int = 64, n_classes: int = 10) -> None: super().__init__() self.net = nn.Sequential( nn.Linear(in_dim, 128), nn.ReLU(), nn.Dropout(0.2), nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, n_classes), # raw logits -> CrossEntropyLoss ) def forward(self, x: torch.Tensor) -> torch.Tensor: return self.net(x) torch.manual_seed(SEED) device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = MLP().to(device) optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4) criterion = nn.CrossEntropyLoss() N_EPOCHS = 30 history: list[tuple[int, float, float, float]] = [] for epoch in range(1, N_EPOCHS + 1): # ---- Train -------------------------------------------------------- model.train() running_loss = 0.0 for xb, yb in train_loader: xb, yb = xb.to(device), yb.to(device) optimizer.zero_grad() loss = criterion(model(xb), yb) loss.backward() optimizer.step() running_loss += loss.item() * xb.size(0) # ---- Evaluate train and test in eval mode (no dropout) ------------ model.eval() with torch.no_grad(): train_logits = model(X_train_t.to(device)) train_acc = (train_logits.argmax(1).cpu() == y_train_t).float().mean().item() test_logits = model(X_test_t.to(device)) test_acc = (test_logits.argmax(1).cpu() == y_test_t).float().mean().item() history.append((epoch, running_loss / len(train_loader.dataset), train_acc, test_acc)) print(f"\n{'epoch':>5} {'train_loss':>11} {'train_acc':>10} {'test_acc':>9}") print("-" * 40) for epoch, loss, tr, te in history: if epoch % 5 == 0 or epoch == 1: print(f"{epoch:>5} {loss:>11.4f} {tr:>10.4f} {te:>9.4f}") # Final predictions on the sealed test set model.eval() with torch.no_grad(): logits = model(X_test_t.to(device)) y_pred_dl = logits.argmax(1).cpu().numpy() dl_acc = accuracy_score(y_test, y_pred_dl) print(f"\nTest accuracy (Deep Learning) : {dl_acc:.4f}")

Expected output — Level 3

text

======================================================================== LEVEL 3 — DEEP LEARNING (PyTorch MLP) ======================================================================== epoch train_loss train_acc test_acc ---------------------------------------- 1 1.6214 0.7244 0.8278 5 0.1583 0.9666 0.9639 10 0.0751 0.9812 0.9722 15 0.0435 0.9903 0.9778 20 0.0294 0.9930 0.9750 25 0.0245 0.9937 0.9722 30 0.0171 0.9951 0.9778 Test accuracy (Deep Learning) : 0.9778

Interpretation

• DL reaches 97.78% vs. SVM’s 97.50% — a 0.28 pp improvement. On this small dataset, the DL gap is small. On ImageNet-scale data with raw 224×224 RGB images, the gap becomes 20+ percentage points.
• Watch the train/test gap. By epoch 20, train accuracy is 99.30% and test is 97.50% — the model is beginning to memorize. By epoch 30, train has risen to 99.51% while test sits at 97.78%. The generalization gap is now 1.73 pp. Early stopping at ~epoch 15 would have been defensible.
• Training loss drops monotonically; test accuracy does not. This is the canonical overfitting curve. Loss is a proxy for progress, not a guarantee of it.
• The DL advantage here is not accuracy, it is flexibility. This same code scales to 10M images by increasing layer width, adding convolutions, and swapping in a GPU. The rule-based classifier cannot scale to 100 fonts, and the SVM kernel matrix becomes infeasible beyond ~100k samples.

python

# ============================================================================ # HEAD-TO-HEAD COMPARISON + FINAL TEST REPORT # ============================================================================ section("FINAL COMPARISON") comparison = pd.DataFrame([ {"paradigm": "Rule-based AI", "human_effort": "high", "data_used": 0, "test_acc": rule_acc}, {"paradigm": "Classical ML (SVM)","human_effort": "medium", "data_used": 1437, "test_acc": ml_acc}, {"paradigm": "Deep Learning (MLP)","human_effort": "low", "data_used": 1437, "test_acc": dl_acc}, ]).set_index("paradigm") print(comparison.round(4).to_string()) print("\nClassification report — Deep Learning (MLP):") print(classification_report(y_test, y_pred_dl, digits=4)) print("Confusion matrix — Deep Learning (MLP):") print(pd.DataFrame( confusion_matrix(y_test, y_pred_dl), index=[f"true {i}" for i in range(10)], columns=[f"pred {i}" for i in range(10)], ).to_string())

Expected output — Final Comparison

text

======================================================================== FINAL COMPARISON ======================================================================== human_effort data_used test_acc paradigm Rule-based AI high 0 0.4028 Classical ML (SVM) medium 1437 0.9750 Deep Learning (MLP) low 1437 0.9778 Classification report — Deep Learning (MLP): precision recall f1-score support 0 1.0000 1.0000 1.0000 36 1 0.9474 1.0000 0.9730 36 2 1.0000 1.0000 1.0000 36 3 1.0000 0.9722 0.9859 36 4 0.9730 1.0000 0.9863 36 5 0.9737 1.0000 0.9868 36 6 1.0000 0.9722 0.9859 36 7 0.9722 0.9722 0.9722 36 8 0.9459 0.9722 0.9589 36 9 0.9722 0.9722 0.9722 36 accuracy 0.9778 360 macro avg 0.9784 0.9776 0.9778 360 weighted avg 0.9784 0.9776 0.9778 360 Confusion matrix — Deep Learning (MLP): pred 0 pred 1 pred 2 pred 3 pred 4 pred 5 pred 6 pred 7 pred 8 pred 9 true 0 36 0 0 0 0 0 0 0 0 0 true 1 0 36 0 0 0 0 0 0 0 0 true 2 0 0 36 0 0 0 0 0 0 0 true 3 0 0 0 35 0 0 0 0 0 1 true 4 0 0 0 0 36 0 0 0 0 0 true 5 0 0 0 0 0 36 0 0 0 0 true 6 0 0 0 0 0 0 35 0 1 0 true 7 0 1 0 0 0 0 0 35 0 0 true 8 0 0 0 0 0 0 0 0 35 1 true 9 0 0 0 0 0 0 0 0 1 35

What the Numbers Actually Tell You

1. The leap from rule-based to ML is enormous (40% → 97%). This is the single most consequential jump in the entire comparison. When someone asks “why not just write rules?” — this is the answer.
2. The leap from ML to DL is small on this dataset (97.50% → 97.78%). That is not a failure of DL. It reflects the fact that 8×8 grayscale digits are a small, clean, low-dimensional problem where classical ML is already near-ceiling.
3. The rule-based model saw zero training examples and yet beat random guessing 4×. Symbolic AI is not dead — it is simply not competitive for perceptual tasks.
4. The three misclassifications (3→9, 6→8, 8→9, 9→8) are visually sensible confusions: at 8×8 resolution, a “3” can look like a “9”, and “8” vs. “9” is a single stroke’s difference.
5. Real-world decision: For this problem, ship the SVM. It is 1/1000th the code, trains in a second, is deterministic, and is only 0.28 pp behind DL. Choose DL when the data is large, raw, or high-dimensional — not because it is fashionable.

---

Common Pitfalls, Edge Cases & Memory Considerations

1. Overfitting and Underfitting

Symptom	Rule-based AI	Classical ML	Deep Learning
Train ≫ test	N/A (no training)	High-degree polynomial, unpruned tree	Too many epochs, no regularization
Train ≈ test, both poor	Rules miss the signal	Underfit model class (linear on non-linear data)	Network too small, LR too low
Erratic test curve	Rules encode noise	Unstable tree splits	LR too high, batch too small

Diagnostics, per paradigm:

• Classical ML: plot learning curves (sklearn.model_selection.learning_curve). If train and CV curves converge at a low value, you need a richer model class. If they diverge, you need regularization or more data.
• DL: plot train loss and validation loss per epoch. The gap is your overfitting. Standard interventions: dropout (0.1–0.5), weight decay (1e-4 to 1e-2), early stopping (patience 5–20), data augmentation, smaller model.
• Rule-based: the concept of overfitting does not apply — but underfitting always does. Rules are structurally incapable of capturing non-linear interactions.

2. Data Leakage — The Silent Killer

Leakage means information from the test set (or the future) influenced training. The model looks brilliant offline and fails in production.

python

# ❌ WRONG — scaler sees test set statistics before the split scaler = StandardScaler().fit(X) # X includes test rows Xs = scaler.transform(X) X_tr, X_te = train_test_split(Xs, ...) # ✅ RIGHT — split first, fit transforms on train only X_tr, X_te = train_test_split(X, ...) scaler = StandardScaler().fit(X_tr) X_tr = scaler.transform(X_tr) X_te = scaler.transform(X_te)

DL-specific leakage pattern — early stopping on the test set:

python

# ❌ WRONG — test set used as validation for early stopping for epoch in range(n_epochs): train_one_epoch(model, train_loader) val_loss = evaluate(model, test_loader) # test is now contaminated if val_loss < best: save(model) # ✅ RIGHT — inner validation split from TRAINING data only X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.15, ...) # ...early stopping uses (X_val, y_val); X_test stays sealed.

Two other sneaky patterns:

python

# ❌ Duplicate entities across train and test X_tr, X_te = train_test_split(df) # same patient in both # ✅ Group-aware split from sklearn.model_selection import GroupShuffleSplit gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42) tr, te = next(gss.split(df, groups=df["patient_id"]))

python

# ❌ Time series shuffled randomly X_tr, X_te = train_test_split(timeseries_df) # future leaks into past # ✅ Time-aware split from sklearn.model_selection import TimeSeriesSplit tscv = TimeSeriesSplit(n_splits=5)

3. Imbalanced Classes

With a 1:1000 fraud ratio, predicting “not fraud” for everything gives 99.9% accuracy and catches zero fraud.

• Use the right metric. For imbalanced classification, use PR-AUC (average_precision_score) or recall at fixed precision, not accuracy or ROC-AUC.
• Prefer threshold tuning over resampling. Train on the natural distribution; pick the threshold that minimizes expected cost on a validation set.
• DL-specific: CrossEntropyLoss(weight=class_weights) reweights the gradient. Combined with a focal loss (gamma=2), this is often more stable than resampling.
• If you oversample, do it only on the training fold. SMOTE before the split creates synthetic neighbors that span the boundary — catastrophic leakage.

4. Non-Reproducibility

Every source of randomness must be seeded.

python

# Complete determinism checklist import os, random, numpy as np, torch os.environ["PYTHONHASHSEED"] = "42" os.environ["OMP_NUM_THREADS"] = "1" # BLAS thread reordering random.seed(42) # stdlib np.random.seed(42) # NumPy legacy RNG torch.manual_seed(42) # PyTorch CPU + default CUDA torch.cuda.manual_seed_all(42) # all CUDA devices # DataLoader shuffling has its OWN generator g = torch.Generator().manual_seed(42) DataLoader(dataset, shuffle=True, generator=g, num_workers=0) # Plus random_state=42 on every sklearn estimator, splitter, and CV object

The hard truth: bit-identical reproducibility across different GPUs, CUDA versions, or BLAS libraries is not guaranteed. cuDNN convolution algorithms are chosen by benchmark and vary by hardware. If you need bit-exact output, set torch.use_deterministic_algorithms(True) (at a performance cost) and pin every version.

5. Numerical Stability

Log-sum-exp (for softmax and log-likelihood). Computing $\log {\sum }_i{e}^{z_i}$log∑i​ezi​ naively overflows when $z_i>88$zi​>88 in float32. Subtract the max first:

python

z = z - z.max(axis=-1, keepdims=True) log_sum_exp = z.max(axis=-1, keepdims=True) + np.log(np.exp(z).sum(axis=-1, keepdims=True))

nn.CrossEntropyLoss and F.log_softmax do this internally. Never compute softmax and then log in two steps — you lose precision and can hit log(0) = -inf.

Gradient clipping. Long RNNs and deep transformers occasionally produce gradient spikes that destroy parameters in a single step:

python

loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step()

max_norm=1.0 is the standard default. Empirically, 10.0 is more permissive; 0.1 is aggressive.

Weight initialization. Never initialize weights to zero — symmetry means every hidden unit computes the same gradient and the network never breaks symmetry. PyTorch’s nn.Linear uses Kaiming uniform by default, which is correct for ReLU. Only override if you know why.

Mixed precision. torch.amp.autocast runs forward passes in float16 for speed, but loss scaling is required to prevent underflow. PyTorch’s torch.amp.GradScaler handles this; do not implement it manually.

6. GPU vs. CPU Memory and Batch-Size Trade-offs

Constraint	Symptom	Mitigation
GPU OOM during forward pass	torch.cuda.OutOfMemoryError	Halve batch size; use torch.amp.autocast
GPU OOM only during backward	OOM after loss computed	Activation checkpointing (torch.utils.checkpoint)
GPU idle, CPU pegged	nvidia-smi shows 0% utilization	Increase num_workers; move augmentation to GPU
Loss goes to nan	Often after LR change or bad batch	Lower LR; add gradient clipping; check for inf in inputs
Training slow despite GPU	Low SM utilization	Increase batch size; check pin_memory=True

Gradient accumulation — simulate a large batch on a small GPU:

python

accum_steps = 8 optimizer.zero_grad() for i, (xb, yb) in enumerate(loader): with torch.amp.autocast(device_type="cuda"): loss = criterion(model(xb), yb) / accum_steps scaler.scale(loss).backward() if (i + 1) % accum_steps == 0: scaler.step(optimizer) scaler.update() optimizer.zero_grad()

Memory rules of thumb:

• Activations dominate VRAM, not parameters. For a batch of $B$B at width $W$W and depth $L$L, activation memory scales as $O(B\cdot L\cdot W)$O(B⋅L⋅W).
• Halving batch size roughly halves activation memory. Parameters do not shrink.
• Inference is ~3× cheaper than training because you skip gradient storage. But you still need the full model in VRAM.

7. Paradigm-Specific Edge Cases

Rule-based:

• Thresholds are hard-coded — any distribution shift breaks them silently.
• Rule ordering matters (first-match-wins vs. weighted). Document your ordering.
• Rules compound: $n$n binary features with $k$k rules can conflict on $2^n$2n inputs. Always include a default class.

Classical ML:

• Scaling matters for SVM, k-NN, and neural nets; it does not matter for tree-based models.
• n < p (fewer samples than features) breaks OLS and unregularized logistic regression. Always use Ridge, Lasso, or dimensionality reduction.
• High-cardinality categoricals explode under one-hot encoding. Use target encoding with cross-fitting, or CatBoost / LightGBM’s native categorical handling.
• n_jobs=-1 can be slower than n_jobs=1 for small datasets due to process-spawn overhead.

Deep Learning:

• BatchNorm behaves differently in train vs. eval mode. Forgetting model.eval() before evaluation produces silently wrong results.
• Dropout is disabled in eval mode. Same trap.
• Small batch sizes with BatchNorm produce noisy batch statistics — use GroupNorm or LayerNorm for batch sizes < 16.
• Learning rate is the single most important hyperparameter. If your loss is nan, lower the LR by 10×.
• Warmup matters for transformers and large batches. The first few hundred steps should ramp LR from near-zero.

---

Summary & Best Practices

Key Takeaways

• AI ⊃ ML ⊃ DL is a nested hierarchy, not a synonym set. AI is the goal; ML is a route; DL is a specific vehicle for that route.
• The defining difference between classical ML and DL is representation learning. Classical ML requires human-designed features; DL learns them end-to-end from raw input.
• Rule-based AI is not obsolete. If a rule is specifiable, write it. Do not train a neural network to compute if age < 18.
• On small, clean, low-dimensional data, the ML-to-DL gap is small. Our digits experiment: SVM 97.50%, MLP 97.78% — a 0.28 pp difference for vastly more code and complexity.
• On raw high-dimensional data (images, audio, text), the gap is enormous. This is where DL earned its reputation — and where its compute and data requirements are justified.
• The neural network is not magic. Strip out non-linearities and it collapses to a single linear layer. Depth + non-linearity is what buys expressiveness.
• Data quality dominates architecture. A well-cleaned dataset with a logistic regression beats a poorly-curated dataset with a transformer.
• Reproducibility is an engineering discipline, not a checkbox. Seed every RNG, pin every version, and log every hyperparameter.

Best-Practice Checklist

1. Start with the simplest possible baseline. Majority class, mean, or a hand-written rule. If it already meets the requirement, stop.
2. Write down what decision the model informs and what each error type costs. That cost determines your metric.
3. Audit the data before modeling. Assert on shape, dtype, missingness, non-finite values, duplicates, and class balance.
4. Split before you transform. Create the hold-out set first; fit every transformer on training data only.
5. Wrap preprocessing in a Pipeline. It makes leakage structurally impossible.
6. Choose the right splitter. StratifiedKFold for classification, GroupKFold for repeated entities, TimeSeriesSplit for temporal data.
7. Use cross-validation for model selection, not the test set. Touch the test set exactly once.
8. Choose the model class based on the data, not the hype. Tabular → boosted trees. Images/text/audio → DL. Well-specified rules → symbolic.
9. Seed everything. random_state on every estimator and splitter, PYTHONHASHSEED and OMP_NUM_THREADS in the environment, torch.manual_seed and a Generator for each DataLoader.
10. Regularize explicitly. weight_decay, dropout, C, alpha, max_depth, early stopping — pick the right regularizer for the paradigm.
11. Clip gradients in DL. clip_grad_norm_(model.parameters(), 1.0) costs one line and prevents nan losses.
12. Report per-class precision and recall for any imbalanced problem — never bare accuracy.
13. Interpret every model you ship. Coefficients for linear models, tree paths for GBDTs, SHAP or attention maps for DL.
14. Log everything. Library versions, hyperparameters, seeds, and the exact commit hash that produced the model.
15. Plan for drift. A model is not deployed until it is monitored. Track input distributions and prediction distributions, and retrain on a schedule.

Suggested Next Steps

1. Run the same experiment on MNIST (28×28, 70k rows) and watch the DL gap widen. That is where DL becomes unambiguously superior.
2. Learn gradient boosting properly. XGBoost / LightGBM remains SOTA on tabular benchmarks over neural networks up to ~100k rows.
3. Study one DL architecture deeply. CNNs for images, transformers for text, or a small autoencoder for representation learning. Then move on.
4. Learn transfer learning. Fine-tuning a pretrained ResNet or BERT on a small labeled dataset is the single most practical DL skill in industry.
5. Move into MLOps. Model versioning (MLflow), serving (FastAPI, TorchServe, Triton), monitoring (Evidently), and CI/CD for models.
6. Read the primary sources. LeCun, Bengio & Hinton (2015), Deep Learning, Nature 521. Krizhevsky et al. (2012), ImageNet Classification with Deep CNNs. Vaswani et al. (2017), Attention Is All You Need.
7. Then continue with the Deep Learning Tutorial for CNNs, RNNs, transformers, and the training-loop mechanics that differ from classical ML.

Further Reading

• Cybenko, G. (1989). Approximation by Superpositions of a Sigmoidal Function. Mathematics of Control, Signals and Systems, 2(4), 303–314. — the Universal Approximation Theorem.
• Hornik, K., Stinchcombe, M. & White, H. (1989). Multilayer Feedforward Networks are Universal Approximators. Neural Networks, 2(5), 359–366.
• Krizhevsky, A., Sutskever, I. & Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS 25. — AlexNet, the inflection point of the DL era.
• LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436–444.
• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 30. — the transformer.
• Shwartz-Ziv, R. & Armon, A. (2022). Tabular Data: Deep Learning Is Not All You Need. Information Fusion, 81, 84–90. — the empirical case against DL on tabular data.
• Grinsztajn, L., Oyallon, E. & Varoquaux, G. (2022). Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data? NeurIPS 35.
• scikit-learn User Guide: Common Pitfalls and Recommended Practices — scikit-learn.org/stable/common_pitfalls.html
• PyTorch documentation: Reproducibility — pytorch.org/docs/stable/notes/randomness.html