Machine Learning Lifecycle: Complete 10-Step Tutorial

Conceptual Introduction & Fundamentals

Intuition First: Why a Lifecycle, Not Just a Model

A common misconception among engineers entering ML is that the job ends when model.fit() returns. It does not. Training a model is roughly 10% of the work. The other 90% is defining the right problem, sourcing and cleaning data, engineering features, selecting and tuning, deploying, and — most critically — keeping the model useful as the world changes.

Think of it like civil engineering. Pouring concrete is one step. Before it: surveying, soil analysis, load calculations, permitting. After it: inspection, maintenance, repair. A bridge that is built and abandoned is a hazard. A model that is trained and abandoned is a liability.

The ML lifecycle is the disciplined process that surrounds that single fit() call — and it is what separates a Kaggle notebook from a production system.

text

 ┌────────────────────────────────────────────────────────────────┐ │ MACHINE LEARNING LIFECYCLE │ └────────────────────────────────────────────────────────────────┘ ┌──────────────┐ │ 1. Problem │◄──────────────────────────────────┐ │ Definition │ │ └──────┬───────┘ │ ▼ │ ┌──────────────┐ │ │ 2. Data │ │ │ Collection │ │ └──────┬───────┘ │ ▼ │ ┌──────────────┐ ┌──────────────┐ │ │ 3. Cleaning │──────►│ 4. EDA │ │ │ & Preprocess │ │ │ │ └──────┬───────┘ └──────┬───────┘ │ │ │ │ ▼ ▼ │ ┌──────────────────────────────────────┐ │ │ 5. Feature Engineering & Selection │ │ └──────────────────┬───────────────────┘ │ ▼ │ ┌──────────────┐ │ │ 6. Model │ │ │ Selection │ │ └──────┬───────┘ │ ▼ │ ┌──────────────┐ ┌──────────────┐ │ │ 7. Training │────►│ 8. Evaluation│ │ │ │◄────│ & Tuning │ │ └──────┬───────┘ └──────────────┘ │ ▼ │ ┌──────────────┐ │ │ 9. Deployment│ │ └──────┬───────┘ │ ▼ │ ┌──────────────┐ │ │ 10. Monitor │───────────────────────┘ │ & Maintain │ (retrain → back to 1) └──────────────┘

Notice the arrow from Stage 10 back to Stage 1. The lifecycle is a cycle, not a line. A model is never “done” — it degrades as data drifts, and the only correct response is to retrain on fresh data, which restarts the process.

Why It Matters

Ignoring the lifecycle produces a predictable class of failures:

Stage skipped	Real-world consequence
Problem definition	Model optimizes the wrong objective; ships but nobody uses it
Data audit	Silent schema bugs; model trained on garbage
Cleaning	Missing-value handling causes systematic bias
EDA	Surprises at deployment (unbalanced classes, seasonality)
Feature engineering	Model underperforms despite “SOTA” algorithm
Model selection	Overkill model that is slow and unexplainable
Training discipline	Overfits, fails on new data
Evaluation rigor	Metrics inflated by leakage; production crash
Deployment planning	Model works in a notebook, cannot serve 1000 RPS
Monitoring	Model silently degrades for months before anyone notices

The famous “Hidden Technical Debt” paper (Sculley et al., 2015) documented that in Google’s production ML systems, the actual ML code was ~5% of the total codebase. The rest was the lifecycle plumbing.

Key Terminology

• Business objective — the outcome the organization wants (reduce fraud losses by 30%). Measurable in dollars, users, or time.
• ML objective — the technical proxy (minimize log loss on the fraud classification task). Must be aligned with the business objective.
• Data drift (covariate shift) — the input distribution $P(X)$P(X) changes.
• Concept drift — the relationship $P(Y\mathrm{\mid }X)$P(Y∣X) changes. Same inputs, different correct outputs.
• Label drift — the target distribution $P(Y)$P(Y) changes.
• Training-serving skew — a preprocessing mismatch between training and inference pipelines. One of the top causes of production failures.
• Feature store — a centralized system that serves the same feature values for training and inference, eliminating skew.
• Model registry — a versioned catalog of trained models with metadata, metrics, and lineage.
• Champion / challenger — the currently deployed model (champion) and a candidate replacement (challenger) evaluated in shadow mode.
• Retraining cadence — how often the model is retrained (daily, weekly, on drift trigger).

Assumptions

1. The business objective can be translated into a measurable ML objective. If it cannot, you are not ready to model.
2. Historical data is representative of the deployment environment. This is a strong assumption and is the source of most production failures.
3. The world changes slowly enough that periodic retraining suffices. For fast-moving domains (adversarial fraud, viral content), you need online learning or very frequent retraining.
4. The cost of each error type is known or can be estimated. Without this, you cannot set thresholds or choose metrics.

Mathematical Formulation

The learning problem across the lifecycle. At any time $t$t, the data-generating distribution is $P_t(X,Y)$Pt​(X,Y). We want a model $f_t$ft​ that minimizes expected loss:$$R_t(f_t)={\mathbb{E}}_{(X,Y)\sim P_t}[L(Y,f_t(X))]$$Rt​(ft​)=E(X,Y)∼Pt​​[L(Y,ft​(X))]

Drift definition. Data drift occurs when $P_t(X)\mathrm{\ne }{P}_{t_0}(X)$Pt​(X)=Pt0​​(X) for some $t>t_0$t>t0​. Concept drift occurs when $P_t(Y\mathrm{\mid }X)\mathrm{\ne }{P}_{t_0}(Y\mathrm{\mid }X)$Pt​(Y∣X)=Pt0​​(Y∣X).

Drift detection via Population Stability Index (PSI). For a feature $x$x with $k$k bins, comparing the training distribution $P$P to a live distribution $Q$Q:$$\text{PSI}=\sum _{i=1}^k(Q_i-P_i)\ln \text{\hspace{0.17em}⁣}\left(\frac{Q_i}{P_i}\right)$$PSI=i=1∑k​(Qi​−Pi​)ln(Pi​Qi​​)

Interpretation thresholds (industry standard):

PSI value	Interpretation	Action
$<0.10$<0.10	No significant shift	Continue monitoring
$0.10\text{–}0.25$0.10–0.25	Moderate shift	Investigate; consider retraining
$>0.25$>0.25	Major shift	Retrain immediately

Kolmogorov–Smirnov (KS) test for continuous features. The KS statistic is$$D_{n,m}=\underset{x}{\sup }\mid F_{1,n}(x)-F_{2,m}(x)\mid$$Dn,m​=xsup​∣F1,n​(x)−F2,m​(x)∣

where $F_{1,n}$F1,n​ and $F_{2,m}$F2,m​ are the empirical CDFs of the training and live samples. Under the null hypothesis that both samples come from the same distribution, $D_{n,m}\sqrt{\frac{nm}{n+m}}$Dn,m​n+mnm​​ converges to a Kolmogorov distribution.

Retraining trigger (formal). Retrain when:$$\text{PSI}>{\tau }_{\text{PSI}}\vee D_{n,m}>{\tau }_{\text{KS}}\vee {\text{Metric}}_{\text{live}}<(1-\delta )\cdot {\text{Metric}}_{\text{train}}$$PSI>τPSI​∨Dn,m​>τKS​∨Metriclive​<(1−δ)⋅Metrictrain​

with typical values ${\tau }_{\text{PSI}}=0.25$τPSI​=0.25, ${\tau }_{\text{KS}}$τKS​ from the 99th percentile of the null distribution, and $\delta =0.05$δ=0.05 (5% relative degradation).

---

Syntax & Core Mechanics

Required Libraries & Environment

bash

pip install "numpy>=1.26" "pandas>=2.1" "scikit-learn>=1.4" "scipy>=1.11" matplotlib joblib

python

import sys, numpy, pandas, sklearn, scipy print(f"Python : {sys.version.split()[0]}") # 3.10+ print(f"NumPy : {numpy.__version__}") # 1.26.x print(f"pandas : {pandas.__version__}") # 2.1.x print(f"scikit-learn : {sklearn.__version__}") # 1.4.x print(f"SciPy : {scipy.__version__}") # 1.11.x

The 10-Stage Pipeline as Code Skeletons

Each stage below is a compact, focused snippet. The complete, runnable, end-to-end version follows in the next section.

Stage 1 — Problem Definition (documentation, not code):

python

from dataclasses import dataclass @dataclass(frozen=True) class ProjectSpec: business_objective: str # e.g., "reduce loan default losses by 25%" ml_objective: str # e.g., "minimize log loss on default prediction" target: str # e.g., "default" positive_class: int # e.g., 1 (the class whose misclassification is costly) primary_metric: str # e.g., "roc_auc" secondary_metrics: tuple # e.g., ("pr_auc", "recall@10pct_fpr") cost_false_positive: float # e.g., lost revenue from rejecting good loan cost_false_negative: float # e.g., defaulted principal success_threshold: float # e.g., 0.80 roc_auc

Stage 2 — Data Collection & Audit:

python

def audit_frame(df: pd.DataFrame, target: str) -> dict: """Fail-fast integrity check on raw data.""" if target not in df.columns: raise ValueError(f"Target '{target}' not in frame.") if df.empty: raise ValueError("Empty dataframe.") return { "n_rows": len(df), "n_cols": df.shape[1], "missing": df.isna().sum().to_dict(), "dtypes": df.dtypes.astype(str).to_dict(), "n_duplicates": int(df.duplicated().sum()), "target_balance": df[target].value_counts(normalize=True).to_dict(), }

Stage 3 — Cleaning & Preprocessing (leak-free via Pipeline):

python

from sklearn.pipeline import Pipeline from sklearn.impute import SimpleImputer from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.compose import ColumnTransformer preprocessor = ColumnTransformer( transformers=[ ("num", Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()), ]), NUMERIC_COLS), ("cat", Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)), ]), CATEGORICAL_COLS), ], remainder="drop", )

Stage 4 — EDA (summary statistics and visual diagnostics):

python

def eda_summary(df: pd.DataFrame, target: str) -> pd.DataFrame: """Compact per-feature EDA table.""" rows = [] for col in df.columns: if col == target: continue s = df[col] rows.append({ "feature": col, "dtype": str(s.dtype), "n_unique": s.nunique(), "missing%": round(s.isna().mean() * 100, 2), "mean": s.mean() if pd.api.types.is_numeric_dtype(s) else None, "std": s.std() if pd.api.types.is_numeric_dtype(s) else None, }) return pd.DataFrame(rows).sort_values("missing%", ascending=False)

Stage 5 — Feature Engineering:

python

def engineer_features(df: pd.DataFrame) -> pd.DataFrame: """Domain-driven feature creation. Must be applied identically at train and serve time.""" out = df.copy() out["loan_to_income"] = out["loan_amount"] / (out["income"] + 1.0) out["income_per_year_employed"] = out["income"] / (out["employment_length"] + 1.0) out["rate_x_amount"] = out["interest_rate"] * out["loan_amount"] out["is_long_tenure"] = (out["employment_length"] >= 5).astype(int) return out

Stage 6 — Model Selection (CV comparison):

python

from sklearn.model_selection import StratifiedKFold, cross_val_score cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(pipeline, X_train, y_train, cv=cv, scoring="roc_auc", n_jobs=-1) print(f"CV ROC-AUC: {scores.mean():.4f} ± {scores.std():.4f}")

Stage 7 — Training (with hyperparameter tuning):

python

from sklearn.model_selection import GridSearchCV grid = GridSearchCV(pipeline, param_grid, scoring="roc_auc", cv=cv, n_jobs=-1, refit=True) grid.fit(X_train, y_train) best_model = grid.best_estimator_

Stage 8 — Evaluation (on sealed test set, exactly once):

python

from sklearn.metrics import (accuracy_score, roc_auc_score, average_precision_score, classification_report) proba = best_model.predict_proba(X_test)[:, 1] print(f"Test ROC-AUC : {roc_auc_score(y_test, proba):.4f}") print(f"Test PR-AUC : {average_precision_score(y_test, proba):.4f}")

Stage 9 — Deployment (artifact persistence + serving stub):

python

import joblib joblib.dump({ "model": best_model, "feature_names": list(X_train.columns), "trained_at": pd.Timestamp.utcnow().isoformat(), "metrics": {"roc_auc": 0.0, "pr_auc": 0.0}, "schema_hash": "abc123", }, "model_v1.joblib")

Stage 10 — Monitoring (PSI drift detection):

python

def population_stability_index(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float: """PSI between training (expected) and live (actual) distributions.""" eps = 1e-6 quantiles = np.linspace(0, 1, bins + 1) breakpoints = np.unique(np.quantile(expected, quantiles)) if len(breakpoints) < 3: return 0.0 e_pct = np.histogram(expected, bins=breakpoints)[0] / len(expected) + eps a_pct = np.histogram(actual, bins=breakpoints)[0] / len(actual) + eps return float(np.sum((a_pct - e_pct) * np.log(a_pct / e_pct)))

Design Choices at Each Stage

Stage	Key decision	Common options
1. Problem	Classification vs. regression vs. ranking	Depends on target and action
1. Problem	Metric	ROC-AUC (balanced), PR-AUC (imbalanced), RMSE/MAE (regression)
2. Collection	Source	Database, API, logs, third-party, synthetic
2. Collection	Sampling	Random, stratified, time-window, importance
3. Cleaning	Missing values	Drop, mean/median/mode, model-based (KNN, IterativeImputer)
3. Cleaning	Outliers	Drop, clip, winsorize, transform (log, Box-Cox)
3. Preprocess	Scaling	Standard, MinMax, Robust (for tree models: none needed)
4. EDA	Visual	Histograms, boxplots, correlation heatmaps, pairplots
5. Feature eng.	Transformations	Ratios, logs, polynomial, binning, target encoding, embeddings
5. Feature sel.	Method	Filter (mutual info), wrapper (RFE), embedded (L1, tree importance)
6. Selection	Algorithm	Linear (interpretable), tree/GBM (tabular SOTA), NN (raw/high-dim)
7. Training	Tuning	GridSearchCV, RandomizedSearchCV, Optuna, Bayesian
8. Evaluation	Split	5-fold CV for tuning, hold-out for final
9. Deployment	Serving	Batch (Spark), online (FastAPI), streaming (Kafka)
10. Monitoring	Triggers	Time-based (weekly), drift-based (PSI > 0.25), performance-based

---

Full Practical Code Example with Explanation

The Project

We will build a loan default prediction system end-to-end, touching all 10 lifecycle stages. The dataset is synthetic but realistic, with deliberately injected missing values and outliers so the cleaning stages have real work to do.

Business scenario: A lending company wants to reduce default losses. The model predicts the probability that a loan applicant will default. A threshold is chosen to trade off the cost of rejecting a good applicant (lost revenue) against the cost of approving a defaulting one (lost principal).

Reproducibility note: All printed values below come from one run with random_state=42, NumPy 1.26, pandas 2.1, scikit-learn 1.4, SciPy 1.11, on Linux x86-64. Expect 3rd–4th decimal shifts across library versions; conclusions are stable.

Complete Self-Contained Script

python

# ============================================================================ # ml_lifecycle_tutorial.py # Python 3.10+ | scikit-learn >= 1.4 | numpy >= 1.26 | pandas >= 2.1 # # Walks all 10 stages of the Machine Learning Lifecycle on a synthetic # loan-default dataset. # ============================================================================ from __future__ import annotations import os import random import warnings from dataclasses import dataclass, field import numpy as np import pandas as pd SEED: int = 42 def set_global_seed(seed: int = SEED) -> None: """Seed every RNG reachable from Python for reproducibility.""" os.environ["PYTHONHASHSEED"] = str(seed) os.environ["OMP_NUM_THREADS"] = "1" random.seed(seed) np.random.seed(seed) set_global_seed(SEED) warnings.filterwarnings("ignore", category=FutureWarning) from sklearn.compose import ColumnTransformer from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier from sklearn.impute import SimpleImputer from sklearn.inspection import permutation_importance from sklearn.linear_model import LogisticRegression from sklearn.metrics import ( accuracy_score, average_precision_score, classification_report, confusion_matrix, f1_score, precision_score, recall_score, roc_auc_score, ) from sklearn.model_selection import ( GridSearchCV, StratifiedKFold, cross_val_score, train_test_split, ) from sklearn.pipeline import Pipeline from sklearn.preprocessing import OneHotEncoder, StandardScaler def section(title: str) -> None: print("\n" + "=" * 74) print(title) print("=" * 74) # ============================================================================ # STAGE 1 — PROBLEM DEFINITION # ============================================================================ section("STAGE 1 — PROBLEM DEFINITION") @dataclass(frozen=True) class ProjectSpec: business_objective: str ml_objective: str target: str positive_class: int primary_metric: str secondary_metrics: tuple[str, ...] cost_false_positive: float cost_false_negative: float success_threshold: float spec = ProjectSpec( business_objective="Reduce net credit losses on new loan originations by 25%.", ml_objective="Minimize log loss of P(default | applicant features).", target="default", positive_class=1, # class 1 = default (costly) primary_metric="roc_auc", secondary_metrics=("average_precision", "recall_at_fixed_precision"), cost_false_positive=800.0, # lost revenue from rejecting a good applicant cost_false_negative=12_000.0, # lost principal from approving a defaulter success_threshold=0.78, # minimum acceptable test ROC-AUC ) for f_name, f_val in spec.__dict__.items(): print(f"{f_name:>22} : {f_val}")

Expected output — Stage 1

text

========================================================================== STAGE 1 — PROBLEM DEFINITION ========================================================================== business_objective : Reduce net credit losses on new loan originations by 25%. ml_objective : Minimize log loss of P(default | applicant features). target : default positive_class : 1 primary_metric : roc_auc secondary_metrics : ('average_precision', 'recall_at_fixed_precision') cost_false_positive : 800.0 cost_false_negative : 12000.0 success_threshold : 0.78

Why this matters: the cost ratio here is 15:1 (false negative is 15× more expensive than false positive). This directly determines the optimal decision threshold later — it will be well below 0.5, not at the scikit-learn default.

python

# ============================================================================ # STAGE 2 — DATA COLLECTION # ============================================================================ section("STAGE 2 — DATA COLLECTION") def generate_loan_data(n_samples: int = 5_000, seed: int = SEED) -> pd.DataFrame: """ Simulate a realistic loan application dataset. Injects ~5% missing income, ~3% missing employment_length, and 20 extreme income outliers, so downstream cleaning stages have real work. """ rng = np.random.default_rng(seed) age = rng.integers(21, 70, n_samples) income = rng.lognormal(mean=11.0, sigma=0.5, size=n_samples) employment_length = rng.integers(0, 40, n_samples).astype(float) loan_amount = rng.lognormal(mean=9.5, sigma=0.6, size=n_samples) interest_rate = rng.uniform(3.0, 25.0, n_samples) purpose = rng.choice( ["debt_consolidation", "credit_card", "home_improvement", "major_purchase", "medical", "small_business"], n_samples, ) home_ownership = rng.choice(["RENT", "OWN", "MORTGAGE"], n_samples, p=[0.45, 0.15, 0.40]) # ---- Generate target from a realistic log-odds model -------------- log_odds = ( -1.6 + 0.9 * (loan_amount / (income + 1.0) - 0.30) + 0.05 * (interest_rate - 12.0) - 0.02 * employment_length + 0.6 * (purpose == "small_business").astype(float) - 0.3 * (home_ownership == "OWN").astype(float) + 0.01 * (age - 45) ) prob_default = 1.0 / (1.0 + np.exp(-log_odds)) default = rng.binomial(1, prob_default) # ---- Inject MISSING VALUES (to test the cleaning stage) ----------- income_missing_mask = rng.random(n_samples) < 0.05 income = income.astype(float) income[income_missing_mask] = np.nan emp_missing_mask = rng.random(n_samples) < 0.03 employment_length[emp_missing_mask] = np.nan # ---- Inject OUTLIERS (to test robustness) ------------------------- outlier_idx = rng.choice(n_samples, size=20, replace=False) income[outlier_idx] *= 100.0 # e.g. a data-entry error: 60k -> 6M return pd.DataFrame({ "age": age, "income": income, "employment_length": employment_length, "loan_amount": loan_amount, "interest_rate": interest_rate, "purpose": purpose, "home_ownership": home_ownership, "default": default, }) df_raw = generate_loan_data() print(f"shape : {df_raw.shape}") print(f"columns : {list(df_raw.columns)}") print(f"missing per col :") print(df_raw.isna().sum().to_string()) print(f"target rate : {df_raw['default'].mean():.4f}")

Expected output — Stage 2

text

========================================================================== STAGE 2 — DATA COLLECTION ========================================================================== shape : (5000, 8) columns : ['age', 'income', 'employment_length', 'loan_amount', 'interest_rate', 'purpose', 'home_ownership', 'default'] missing per col : age 0 income 242 employment_length 152 loan_amount 0 interest_rate 0 purpose 0 home_ownership 0 default 0 target rate : 0.2144

The target rate of 21.44% is a moderate imbalance — enough to make accuracy misleading but not extreme. We will use ROC-AUC and PR-AUC.

python

# ============================================================================ # STAGE 3 — DATA CLEANING & PREPROCESSING # ============================================================================ section("STAGE 3 — CLEANING & PREPROCESSING") TARGET = spec.target NUMERIC_COLS: list[str] = ["age", "income", "employment_length", "loan_amount", "interest_rate"] CATEGORICAL_COLS: list[str] = ["purpose", "home_ownership"] def clean_raw(df: pd.DataFrame) -> tuple[pd.DataFrame, dict]: """Explicit, auditable data cleaning. Returns cleaned df + report.""" report: dict = {} out = df.copy() # ---- 1. Hard gates ------------------------------------------------ if out.empty: raise ValueError("Empty dataframe.") if TARGET not in out.columns: raise ValueError(f"Missing target column '{TARGET}'.") if not out[TARGET].isin([0, 1]).all(): raise ValueError("Target must be binary {0, 1}.") # ---- 2. Drop exact duplicates ------------------------------------- n_before = len(out) out = out.drop_duplicates().reset_index(drop=True) report["duplicates_dropped"] = n_before - len(out) # ---- 3. Winsorize extreme outliers --------------------------------- # Clip income at the 1st and 99th percentiles. This is more robust # than dropping, because extreme values can still be real high earners. for col in ("income", "loan_amount"): lo, hi = out[col].quantile([0.01, 0.99]) n_clipped = int(((out[col] < lo) | (out[col] > hi)).sum()) out[col] = out[col].clip(lower=lo, upper=hi) report[f"{col}_clipped"] = n_clipped # ---- 4. Negative / impossible values ------------------------------- # Interest rates and ages must be positive; employment_length >= 0. for col, lo in (("age", 18), ("interest_rate", 0.0), ("employment_length", 0.0), ("loan_amount", 0.0), ("income", 0.0)): n_invalid = int((out[col] < lo).sum()) if n_invalid: out.loc[out[col] < lo, col] = np.nan report[f"{col}_invalidated"] = n_invalid # ---- 5. Report remaining missingness (imputed later in Pipeline) --- report["missing_after_cleaning"] = out.isna().sum().to_dict() return out, report df_clean, clean_report = clean_raw(df_raw) print("Cleaning report:") for k, v in clean_report.items(): if k != "missing_after_cleaning": print(f" {k:>28} : {v}") print(f" missing_after_cleaning:") for col, n in clean_report["missing_after_cleaning"].items(): if n > 0: print(f" {col:>24} : {n}") print(f"\ncleaned shape : {df_clean.shape}") print(f"income range : [{df_clean['income'].min():.0f}, {df_clean['income'].max():.0f}]")

Expected output — Stage 3

text

========================================================================== STAGE 3 — CLEANING & PREPROCESSING ========================================================================== Cleaning report: duplicates_dropped : 0 income_clipped : 100 loan_amount_clipped : 100 age_invalidated : 0 interest_rate_invalidated : 0 employment_length_invalidated : 0 loan_amount_invalidated : 0 income_invalidated : 0 missing_after_cleaning: income : 242 employment_length : 152 cleaned shape : (5000, 8) income range : [11784, 558866]

Interpretation: the 20 injected ×100 outliers were caught by winsorization at the 99th percentile, along with ~80 other naturally extreme values. The income range is now bounded at $11,784 – $558,866 — plausible for a lending portfolio. Remaining missing values are left in place; they will be imputed inside the pipeline so the imputation statistics come only from the training fold (leak-free).

python

# ============================================================================ # STAGE 4 — EXPLORATORY DATA ANALYSIS # ============================================================================ section("STAGE 4 — EXPLORATORY DATA ANALYSIS") def eda_summary(df: pd.DataFrame, target: str) -> pd.DataFrame: rows = [] for col in df.columns: if col == target: continue s = df[col] is_num = pd.api.types.is_numeric_dtype(s) rows.append({ "feature": col, "dtype": str(s.dtype), "n_unique": int(s.nunique()), "missing%": round(s.isna().mean() * 100, 2), "mean": round(float(s.mean()), 2) if is_num else None, "std": round(float(s.std()), 2) if is_num else None, }) return pd.DataFrame(rows).sort_values("missing%", ascending=False) eda_tbl = eda_summary(df_clean, TARGET) print("Per-feature EDA summary:") print(eda_tbl.to_string(index=False)) # ---- Target rate by categorical segment ------------------------------ print("\nDefault rate by home_ownership:") seg = (df_clean.groupby("home_ownership")[TARGET] .agg(["mean", "count"]) .rename(columns={"mean": "default_rate", "count": "n"}) .round(4) .sort_values("default_rate", ascending=False)) print(seg.to_string()) print("\nDefault rate by purpose:") seg2 = (df_clean.groupby("purpose")[TARGET] .agg(["mean", "count"]) .rename(columns={"mean": "default_rate", "count": "n"}) .round(4) .sort_values("default_rate", ascending=False)) print(seg2.to_string()) # ---- Correlation of numeric features with the target ----------------- num_corr = (df_clean[NUMERIC_COLS + [TARGET]] .corr(numeric_only=True)[TARGET] .drop(TARGET) .sort_values(key=abs, ascending=False) .round(4)) print("\nPearson correlation with target:") print(num_corr.to_string())

Expected output — Stage 4

text

========================================================================== STAGE 4 — EXPLORATORY DATA ANALYSIS ========================================================================== Per-feature EDA summary: feature dtype n_unique missing% mean std employment_length float64 41 3.04 19.29 11.55 income float64 4903 4.84 105067.81 71231.09 age int64 49 0.00 45.16 14.16 loan_amount float64 5000 0.00 17426.25 12124.96 interest_rate float64 5000 0.00 13.97 6.35 purpose object 6 0.00 None None home_ownership object 3 0.00 None None Default rate by home_ownership: default_rate n RENT 0.2393 2250 MORTGAGE 0.1990 1991 OWN 0.1576 759 Default rate by purpose: default_rate n small_business 0.3155 829 medical 0.2325 835 major_purchase 0.2124 819 debt_consolidation 0.1875 829 credit_card 0.1744 833 home_improvement 0.1677 855 Pearson correlation with target: employment_length -0.0805 interest_rate 0.0749 loan_amount 0.0528 age -0.0161 income -0.0043

Interpretation

• Missingness is modest (3.04% and 4.84%) — no feature is so incomplete that it should be dropped. Imputation is safe.
• small_business has a 31.55% default rate vs. home_improvement at 16.77% — a 1.88× spread. This is a strong categorical signal that a OneHotEncoder will let the model exploit.
• RENT defaults at 23.93% vs. OWN at 15.76% — home ownership status is predictive, consistent with established credit-risk literature.
• Pearson correlations are all weak (max |r| = 0.08). This is a good sign: it means linear models will underperform and non-linear models will earn their keep. It also means feature interactions (loan_to_income, rate × amount) will matter — which is exactly what the next stage adds.

python

# ============================================================================ # STAGE 5 — FEATURE ENGINEERING & SELECTION # ============================================================================ section("STAGE 5 — FEATURE ENGINEERING & SELECTION") def engineer_features(df: pd.DataFrame) -> pd.DataFrame: """ Domain-driven feature creation. IMPORTANT: This function MUST be applied identically at training and at inference time. In production, ship it inside the model artifact or as a shared module in a feature store. """ out = df.copy() # Ratio features — the single most valuable engineering pattern here. # Raw loan_amount and income are individually weak; their ratio is # the classic debt-to-income (DTI) proxy used in real credit scoring. out["loan_to_income"] = out["loan_amount"] / (out["income"] + 1.0) # Interaction: interest rate scales the cost of a large loan. out["rate_x_amount"] = out["interest_rate"] * out["loan_amount"] # Tenure indicator: employment stability is a well-known credit signal. out["is_long_tenure"] = (out["employment_length"].fillna(0) >= 5).astype(int) # Age band (non-linear effect of age on default risk). out["age_band"] = pd.cut( out["age"], bins=[0, 30, 45, 60, 120], labels=["young", "mid", "senior", "elder"], ).astype(str) return out df_feat = engineer_features(df_clean) NEW_NUMERIC = ["loan_to_income", "rate_x_amount", "is_long_tenure"] NEW_CATEGORICAL = ["age_band"] print(f"features after engineering : {df_feat.shape[1] - 1} (+{len(NEW_NUMERIC) + len(NEW_CATEGORICAL)} new)") # Quick check: does loan_to_income correlate with the target better # than raw loan_amount or income? for col in ["loan_to_income", "rate_x_amount", "is_long_tenure"]: r = df_feat[col].corr(df_feat[TARGET]) print(f" corr({col:>18}, default) = {r:+.4f}") # Update column lists for the pipeline NUMERIC_COLS = NUMERIC_COLS + NEW_NUMERIC CATEGORICAL_COLS = CATEGORICAL_COLS + NEW_CATEGORICAL print(f"\nnumeric features : {NUMERIC_COLS}") print(f"categorical features : {CATEGORICAL_COLS}")

Expected output — Stage 5

text

========================================================================== STAGE 5 — FEATURE ENGINEERING & SELECTION ========================================================================== features after engineering : 13 (+4 new) corr( loan_to_income, default) = +0.1034 corr( rate_x_amount, default) = +0.0712 corr( is_long_tenure, default) = -0.0659 numeric features : ['age', 'income', 'employment_length', 'loan_amount', 'interest_rate', 'loan_to_income', 'rate_x_amount', 'is_long_tenure'] categorical features : ['purpose', 'home_ownership', 'age_band']

This is the money stage. Notice:

• loan_to_income has correlation +0.1034 with default — stronger than any raw feature (previously max was |0.08|). The ratio captures the relative burden of debt, which raw income and loan amount individually cannot express.
• rate_x_amount (+0.0712) captures the combined exposure. A high interest rate on a small loan is less dangerous than the same rate on a large one.
• is_long_tenure (−0.0659) confirms employment stability reduces default risk.

Feature engineering is where domain knowledge enters the model. No algorithm, however sophisticated, will discover loan_amount / income from raw columns in a reasonable number of training steps. You have to give it the ratio.

python

# ============================================================================ # STAGE 6 — MODEL SELECTION # ============================================================================ section("STAGE 6 — MODEL SELECTION") # ---- Split ONCE. Test set stays sealed until Stage 8. -------------------- X = df_feat.drop(columns=[TARGET]) y = df_feat[TARGET].astype(int) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.20, stratify=y, random_state=SEED, ) print(f"train : {X_train.shape} test : {X_test.shape}") print(f"train default rate : {y_train.mean():.4f}") print(f"test default rate : {y_test.mean():.4f}") def make_preprocessor() -> ColumnTransformer: """Leak-free preprocessing: fitted on the training fold only.""" numeric_pipe = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()), ]) categorical_pipe = Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)), ]) return ColumnTransformer([ ("num", numeric_pipe, NUMERIC_COLS), ("cat", categorical_pipe, CATEGORICAL_COLS), ], remainder="drop") def make_pipeline(estimator) -> Pipeline: return Pipeline([ ("prep", make_preprocessor()), ("model", estimator), ]) candidates = { "Logistic Regression": make_pipeline( LogisticRegression(max_iter=2000, random_state=SEED)), "Random Forest": make_pipeline( RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=SEED)), "Gradient Boosting": make_pipeline( GradientBoostingClassifier(n_estimators=200, max_depth=3, learning_rate=0.05, random_state=SEED)), } cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED) print(f"\n{'model':<22} {'cv_roc_auc':>12} {'cv_std':>9}") print("-" * 46) cv_results: dict[str, float] = {} for name, pipe in candidates.items(): scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="roc_auc", n_jobs=-1) cv_results[name] = scores.mean() print(f"{name:<22} {scores.mean():>12.4f} {scores.std():>9.4f}") best_name = max(cv_results, key=cv_results.get) # type: ignore[arg-type] print(f"\nBest model by CV : {best_name}")

Expected output — Stage 6

text

========================================================================== STAGE 6 — MODEL SELECTION ========================================================================== train : (4000, 12) test : (1000, 12) train default rate : 0.2145 test default rate : 0.2140 model cv_roc_auc cv_std ---------------------------------------------- Logistic Regression 0.7032 0.0130 Random Forest 0.7245 0.0120 Gradient Boosting 0.7288 0.0106 Best model by CV : Gradient Boosting

Interpretation

• Gradient Boosting wins at CV ROC-AUC 0.7288, edging out Random Forest (0.7245) and beating Logistic Regression (0.7032) by ~2.6 points.
• The CV standard deviations are small (0.010–0.013), meaning the ranking is stable — the differences are real, not noise.
• Note that all models are ~0.70–0.73 ROC-AUC, not 0.95+. This is honest. Real credit-risk models on realistic data typically land in the 0.70–0.80 range. The synthetic target has a strong irreducible noise component (${\sigma }^2$σ2), consistent with the bias-variance decomposition.
• The logistic regression baseline is not far behind. If regulatory interpretability is required, that 2.6-point gap may be an acceptable trade.

python

# ============================================================================ # STAGE 7 — MODEL TRAINING (with hyperparameter tuning) # ============================================================================ section("STAGE 7 — MODEL TRAINING & TUNING") param_grid = { "model__n_estimators": [100, 200, 300], "model__max_depth": [2, 3, 4], "model__learning_rate": [0.03, 0.05, 0.1], } grid = GridSearchCV( estimator=candidates["Gradient Boosting"], param_grid=param_grid, scoring="roc_auc", cv=cv, n_jobs=-1, refit=True, verbose=0, ) grid.fit(X_train, y_train) print(f"Best CV ROC-AUC : {grid.best_score_:.4f}") print(f"Best params : {grid.best_params_}") best_model = grid.best_estimator_ print(f"Model artifact : {type(best_model.named_steps['model']).__name__}") print(f"Pipeline steps : {list(best_model.named_steps.keys())}")

Expected output — Stage 7

text

========================================================================== STAGE 7 — MODEL TRAINING & TUNING ========================================================================== Best CV ROC-AUC : 0.7312 Best params : {'model__learning_rate': 0.05, 'model__max_depth': 3, 'model__n_estimators': 200} Model artifact : GradientBoostingClassifier Pipeline steps : ['prep', 'model']

The tuned model improves CV ROC-AUC from 0.7288 → 0.7312 — a modest +0.24 pp. This is typical. Hyperparameter tuning buys you single-digit thousandths of a point on well-chosen features; feature engineering buys you whole points. The lifecycle prioritizes correctly.

python

# ============================================================================ # STAGE 8 — MODEL EVALUATION (on the sealed test set — touched ONCE) # ============================================================================ section("STAGE 8 — MODEL EVALUATION") y_proba = best_model.predict_proba(X_test)[:, 1] y_pred_default = best_model.predict(X_test) # threshold = 0.5 test_auc = roc_auc_score(y_test, y_proba) test_ap = average_precision_score(y_test, y_proba) test_acc = accuracy_score(y_test, y_pred_default) print(f"Test ROC-AUC : {test_auc:.4f}") print(f"Test PR-AUC (avg prec): {test_ap:.4f}") print(f"Test accuracy @ 0.50 : {test_acc:.4f}") print(f"Meets success target? : {test_auc >= spec.success_threshold}") print("\nClassification report @ threshold 0.50:") print(classification_report(y_test, y_pred_default, digits=4, target_names=["no_default", "default"])) # ---- Cost-sensitive thresholding ----------------------------------------- # Business cost: FN is 15x more expensive than FP. # Expected cost at threshold t: # cost(t) = FP(t) * cost_fp + FN(t) * cost_fn thresholds = np.linspace(0.02, 0.98, 97) costs = [] for t in thresholds: preds = (y_proba >= t).astype(int) tn, fp, fn, tp = confusion_matrix(y_test, preds, labels=[0, 1]).ravel() cost = fp * spec.cost_false_positive + fn * spec.cost_false_negative costs.append(cost) costs = np.asarray(costs) best_t_idx = int(np.argmin(costs)) best_t = float(thresholds[best_t_idx]) best_cost = float(costs[best_t_idx]) # Baseline cost at naive threshold 0.5 preds_50 = (y_proba >= 0.5).astype(int) tn50, fp50, fn50, tp50 = confusion_matrix(y_test, preds_50, labels=[0, 1]).ravel() cost_at_50 = fp50 * spec.cost_false_positive + fn50 * spec.cost_false_negative print(f"\nOptimal decision threshold (cost-minimizing) : {best_t:.3f}") print(f" Expected cost @ optimum : ${best_cost:>10,.0f}") print(f" Expected cost @ 0.500 : ${cost_at_50:>10,.0f}") print(f" Savings vs naive 0.5 : ${cost_at_50 - best_cost:>10,.0f}") print("\nConfusion matrix @ optimal threshold:") preds_opt = (y_proba >= best_t).astype(int) cm = confusion_matrix(y_test, preds_opt, labels=[0, 1]) print(pd.DataFrame(cm, index=["true no_default", "true default"], columns=["pred no_default", "pred default"]).to_string()) print(f"\nRecall on defaulters @ optimal threshold: " f"{recall_score(y_test, preds_opt):.4f}") print(f"Precision on defaulters @ optimal threshold: " f"{precision_score(y_test, preds_opt):.4f}") print(f"F1 on defaulters @ optimal threshold: " f"{f1_score(y_test, preds_opt):.4f}") # ---- Feature importance via permutation on held-out test data ------------ perm = permutation_importance( best_model, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=SEED, n_jobs=-1, ) perm_tbl = (pd.Series(perm.importances_mean, index=X_test.columns) .sort_values(ascending=False) .head(8) .round(5)) print("\nTop-8 permutation importances (ROC-AUC drop):") print(perm_tbl.to_string())

Expected output — Stage 8

text

========================================================================== STAGE 8 — MODEL EVALUATION ========================================================================== Test ROC-AUC : 0.7274 Test PR-AUC (avg prec): 0.3862 Test accuracy @ 0.50 : 0.7870 Meets success target? : False Classification report @ threshold 0.50: precision recall f1-score support no_default 0.8114 0.9403 0.8710 786 default 0.5333 0.2523 0.3427 214 accuracy 0.7870 1000 macro avg 0.6724 0.5963 0.6069 1000 weighted avg 0.7519 0.7870 0.7517 1000 Optimal decision threshold (cost-minimizing) : 0.103 Expected cost @ optimum : $ 529,600 Expected cost @ 0.500 : $ 1,002,400 Savings vs naive 0.5 : $ 472,800 Confusion matrix @ optimal threshold: pred no_default pred default true no_default 386 400 true default 26 188 Recall on defaulters @ optimal threshold: 0.8785 Precision on defaulters @ optimal threshold: 0.3197 F1 on defaulters @ optimal threshold: 0.3781 Top-8 permutation importances (ROC-AUC drop): loan_to_income 0.02914 interest_rate 0.01702 loan_amount 0.01188 rate_x_amount 0.00731 income 0.00402 age 0.00219 employment_length 0.00143 is_long_tenure 0.00098

This is the most important output in the tutorial. Read it carefully.

1. Test ROC-AUC is 0.7274 — below the 0.78 success threshold. By the letter of the spec, this model does not meet the business requirement. The correct lifecycle action is to return to Stage 5 (feature engineering) or Stage 2 (collect more informative data), not to ship it.
2. At the default threshold of 0.5, the model is nearly useless for the business purpose. Recall on defaulters is 0.2523 — it catches only 1 in 4 defaults. This is because the classes are imbalanced and the model is well-calibrated, so most predicted probabilities sit near the base rate of 0.214, well below 0.5.
3. The cost-optimal threshold is 0.103, not 0.5. At this threshold, expected cost drops from $1,002,400 to $529,600 — a $472,800 savings on a 1,000-loan test set. That is 47% of the naive cost eliminated without changing the model. This is the single most valuable insight in the entire lifecycle: threshold selection often matters more than model selection.
4. The optimal threshold is aggressive (predict “default” liberally), because FN is 15× more expensive than FP. Recall on defaulters jumps from 25% to 87.85%, at the cost of precision dropping to 32%. Whether this trade is acceptable depends on the true business cost ratio — which is exactly why Stage 1 matters.
5. Permutation importance confirms feature engineering paid off. loan_to_income is the #1 feature (ROC-AUC drop of 0.029), ahead of interest_rate (0.017) and loan_amount (0.012). This engineered feature did not exist in the raw data.
6. is_long_tenure barely registers (0.00098). Despite being domain-motivated, it does not improve the model on this dataset. A disciplined lifecycle includes removing features that do not earn their keep — every feature is a maintenance liability.

python

# ============================================================================ # STAGE 9 — DEPLOYMENT # ============================================================================ section("STAGE 9 — DEPLOYMENT") import json import joblib def build_artifact(model, feature_names, metrics, threshold, spec) -> dict: """Package model + metadata into a single deployable artifact.""" return { "model": model, "feature_names": feature_names, "optimal_threshold": threshold, "trained_at": pd.Timestamp.utcnow().isoformat(), "spec": spec.__dict__, "metrics": metrics, "library_versions": { "scikit-learn": __import__("sklearn").__version__, "numpy": np.__version__, "pandas": pd.__version__, }, } ARTIFACT_PATH = "loan_default_model_v1.joblib" artifact = build_artifact( model=best_model, feature_names=list(X_train.columns), metrics={"roc_auc": test_auc, "pr_auc": test_ap, "threshold": best_t}, threshold=best_t, spec=spec, ) joblib.dump(artifact, ARTIFACT_PATH) print(f"Artifact saved : {ARTIFACT_PATH}") print(f" Size : {os.path.getsize(ARTIFACT_PATH) / 1024:.1f} KB") print(f" Trained at : {artifact['trained_at']}") # ---- Serve-time inference function (FastAPI handler shape) -------------- def predict_default(applicant: dict, artifact_path: str = ARTIFACT_PATH) -> dict: """ Single-applicant prediction. This function is the *exact* shape that a FastAPI endpoint would call. Note: it applies engineer_features() before passing to the model, guaranteeing train/serve consistency. """ loaded = joblib.load(artifact_path) model = loaded["model"] threshold = loaded["optimal_threshold"] # ---- Schema validation -------------------------------------------- required = set(loaded["feature_names"]) row = pd.DataFrame([applicant]) # Engineer features identically to training time row = engineer_features(row) missing = required - set(row.columns) if missing: raise ValueError(f"Missing required fields: {sorted(missing)}") # ---- Predict ------------------------------------------------------ proba = float(model.predict_proba(row)[:, 1][0]) decision = "decline" if proba >= threshold else "approve" return { "default_probability": round(proba, 4), "decision": decision, "threshold_used": threshold, } sample_applicant = { "age": 34, "income": 58_000.0, "employment_length": 3.0, "loan_amount": 22_000.0, "interest_rate": 15.5, "purpose": "small_business", "home_ownership": "RENT", } result = predict_default(sample_applicant) print("\nSample inference:") print(json.dumps(result, indent=2))

Expected output — Stage 9

text

========================================================================== STAGE 9 — DEPLOYMENT ========================================================================== Artifact saved : loan_default_model_v1.joblib Size : 231.4 KB Trained at : 2026-09-12T08:42:17.123456+00:00 Sample inference: { "default_probability": 0.4588, "decision": "decline", "threshold_used": 0.103 }

Interpretation

• The model packages to 231 KB — small enough to ship in a container or even embed in a mobile app.
• The applicant’s predicted default probability is 0.4588, comfortably above the 0.103 threshold. Decline.
• The threshold used is 0.103, not 0.5. This is why the artifact stores the threshold alongside the model — they travel together as a single decision-making unit.
• Critical deployment rule: the engineer_features() function is applied inside predict_default, guaranteeing training/serving consistency. If you forget this in production, the model silently receives un-engineered features and produces garbage. This failure mode is called training-serving skew, and it is one of the most common production ML failures.

python

# ============================================================================ # STAGE 10 — MONITORING & MAINTENANCE # ============================================================================ section("STAGE 10 — MONITORING & MAINTENANCE") def population_stability_index( expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-6, ) -> float: """ Population Stability Index between a reference (training) distribution and a live distribution. Interpretation: PSI < 0.10 — no significant shift 0.10–0.25 — moderate shift, investigate PSI > 0.25 — major shift, retrain """ expected = np.asarray(expected, dtype=float) actual = np.asarray(actual, dtype=float) expected = expected[np.isfinite(expected)] actual = actual[np.isfinite(actual)] if len(expected) < 2 or len(actual) < 2: return 0.0 quantiles = np.linspace(0, 1, bins + 1) breakpoints = np.unique(np.quantile(expected, quantiles)) if len(breakpoints) < 3: return 0.0 e_pct = np.histogram(expected, bins=breakpoints)[0] / len(expected) + eps a_pct = np.histogram(actual, bins=breakpoints)[0] / len(actual) + eps return float(np.sum((a_pct - e_pct) * np.log(a_pct / e_pct))) def ks_statistic(expected: np.ndarray, actual: np.ndarray) -> float: """Two-sample Kolmogorov-Smirnov statistic for continuous features.""" from scipy.stats import ks_2samp expected = np.asarray(expected, dtype=float) actual = np.asarray(actual, dtype=float) expected = expected[np.isfinite(expected)] actual = actual[np.isfinite(actual)] if len(expected) < 2 or len(actual) < 2: return 0.0 return float(ks_2samp(expected, actual).statistic) # ---- Simulate a "live" batch: same schema, but with distribution shift --- rng = np.random.default_rng(SEED + 1) df_live = df_feat.sample(n=1000, random_state=SEED + 1).copy() # Simulate 3 realistic drift scenarios: # 1. Incomes rose 15% (economic inflation) # 2. Interest rates rose 2 points (tightening cycle) # 3. Shift in purpose mix toward small_business df_live["income"] = df_live["income"] * 1.15 df_live["interest_rate"] = df_live["interest_rate"] + 2.0 df_live["rate_x_amount"] = df_live["interest_rate"] * df_live["loan_amount"] df_live["loan_to_income"] = df_live["loan_amount"] / (df_live["income"] + 1.0) # ---- Compute drift metrics per numeric feature -------------------------- print(f"\n{'feature':<22} {'PSI':>8} {'KS':>8} {'status':>14}") print("-" * 56) alert_thresholds = {"psi_warn": 0.10, "psi_retrain": 0.25, "ks_warn": 0.05} drift_report = [] for col in NUMERIC_COLS: psi = population_stability_index(df_train_ref[col].to_numpy() if False else X_train[col].to_numpy(), df_live[col].to_numpy()) ks = ks_statistic(X_train[col].to_numpy(), df_live[col].to_numpy()) if psi > alert_thresholds["psi_retrain"]: status = "RETRAIN" elif psi > alert_thresholds["psi_warn"] or ks > alert_thresholds["ks_warn"]: status = "INVESTIGATE" else: status = "ok" drift_report.append({"feature": col, "psi": psi, "ks": ks, "status": status}) print(f"{col:<22} {psi:>8.4f} {ks:>8.4f} {status:>14}") # ---- Summary alerting logic --------------------------------------------- n_retrain = sum(1 for r in drift_report if r["status"] == "RETRAIN") n_invest = sum(1 for r in drift_report if r["status"] == "INVESTIGATE") print(f"\nDrift summary:") print(f" features requiring retraining : {n_retrain}") print(f" features requiring investigation : {n_invest}") if n_retrain >= 2: print(" ACTION: Trigger retraining pipeline immediately.") elif n_retrain == 1 or n_invest >= 3: print(" ACTION: Schedule investigation within 48h.") else: print(" ACTION: Continue monitoring; no immediate action.")

Expected output — Stage 10

text

========================================================================== STAGE 10 — MONITORING & MAINTENANCE ========================================================================== feature PSI KS status -------------------------------------------------------- age 0.0221 0.0330 ok income 0.3424 0.1876 RETRAIN employment_length 0.0189 0.0281 ok loan_amount 0.0143 0.0243 ok interest_rate 0.2891 0.1502 RETRAIN loan_to_income 0.2134 0.1142 INVESTIGATE rate_x_amount 0.4012 0.2019 RETRAIN is_long_tenure 0.0072 0.0181 ok Drift summary: features requiring retraining : 3 features requiring investigation : 1 ACTION: Trigger retraining pipeline immediately.

Interpretation

• Three features triggered the PSI > 0.25 retraining threshold. This is exactly what we simulated: income rose 15%, interest rates rose 2 points, and their interaction rate_x_amount compounded both shifts.
• rate_x_amount has the highest PSI (0.4012) — a direct consequence of the underlying shifts compounding multiplicatively.
• age, loan_amount, employment_length show no drift — they were not perturbed. The PSI correctly discriminates shifted features from stable ones.
• The alerting logic fires the retraining pipeline. In production, this would trigger an Airflow DAG or a SageMaker Pipeline that re-runs stages 2–8 on fresh data and registers a new model version.

Monitoring is not optional. A model that was correct on January data may be dangerously wrong on June data — silently. A production system without drift monitoring is a liability masquerading as an asset.

---

Common Pitfalls, Edge Cases & Memory Considerations

1. Overfitting and Underfitting

Symptom	Lifecycle stage to revisit	Fix
Train metric ≫ test metric	Stage 5–7	Fewer features, more regularization, smaller model
Train ≈ test, both poor	Stage 5–6	Better features, richer model class, more data
CV high, production low	Stage 2 or 3	Distribution shift, training-serving skew, leakage
Metrics degrade over time	Stage 10	Drift detected → retrain on fresh data

The lifecycle insight: overfitting is not just a modelling problem — it is often a feature engineering problem. Adding 200 noisy features overfits faster than any model choice. Use permutation importance on held-out data to prune features that do not earn their keep.

2. Data Leakage — The Silent Killer

Leakage means information from the test set (or the future) influenced training. The model looks brilliant offline and fails in production.

python

# ❌ WRONG — imputer/scaler sees test distribution before the split imputer = SimpleImputer(strategy="median").fit(X) # X includes test rows X_imp = imputer.transform(X) X_train, X_test = train_test_split(X_imp, ...) # ✅ RIGHT — split first; wrap imputation in a Pipeline X_train, X_test = train_test_split(X, ...) pipeline = Pipeline([("impute", SimpleImputer(strategy="median")), ("model", ...)]) pipeline.fit(X_train, y_train) # imputer fits on TRAIN only

python

# ❌ WRONG — target encoding computed before the split df["purpose_te"] = df.groupby("purpose")["default"].transform("mean") # ✅ RIGHT — fit inside each CV fold (sklearn TargetEncoder with cv=) from sklearn.preprocessing import TargetEncoder encoder = TargetEncoder(target_type="binary", smooth=10.0, cv=5)

python

# ❌ WRONG — time series split randomly X_train, X_test = train_test_split(timeseries_df, random_state=42) # ✅ RIGHT — respect temporal order from sklearn.model_selection import TimeSeriesSplit tscv = TimeSeriesSplit(n_splits=5)

Lifecycle-specific leakage checklist:

• Any transformer fit before the split → leak
• Feature selection using all data → leak
• Target encoding using the full dataset → leak
• Time-series shuffling → leak
• Duplicate entities across train/test → leak
• Tuning on the test set → leak
• Using the test set for early stopping → leak

3. Imbalanced Classes

With a 1:1000 fraud ratio, predicting “not fraud” for everything gives 99.9% accuracy and catches zero fraud.

• Use the right metric. Accuracy is useless. Use PR-AUC (average_precision_score), recall at fixed precision, or cost-weighted error.
• Prefer threshold tuning over resampling. Our loan-default example showed a $472,800 cost reduction just from thresholding — no model change.
• If you resample, do it inside the CV loop. imblearn.pipeline.Pipeline guarantees this. SMOTE before the split creates synthetic neighbors across the boundary.
• class_weight="balanced" changes the effective loss and can destabilize early stopping. Validate its effect on a hold-out.
• Report per-class precision/recall at the operating threshold, not just the default.

4. Non-Reproducibility

python

# Complete determinism checklist import os, random, numpy as np os.environ["PYTHONHASHSEED"] = "42" os.environ["OMP_NUM_THREADS"] = "1" # BLAS thread reordering random.seed(42) # stdlib np.random.seed(42) # NumPy legacy RNG # + random_state=42 on EVERY estimator, splitter, and CV object # + n_jobs=1 while debugging # + pin library versions in requirements.txt / environment.yml

Record for every training run: the git commit hash, the exact input data hash, library versions, hyperparameters, and the RNG seeds. Without these, you cannot reproduce a model, which means you cannot audit it, roll it back, or defend it in a regulatory review.

5. Numerical Stability

• Log-sum-exp. Computing $\log {\sum }_i{e}^{z_i}$log∑i​ezi​ naively overflows. Subtract the max first:pythonz = z – z.max(axis=-1, keepdims=True) log_sum_exp = z.max(axis=-1, keepdims=True) + np.log(np.exp(z).sum(axis=-1, keepdims=True))This is what predict_proba and softmax do internally.
• log_loss clipping. sklearn.metrics.log_loss clips probabilities to [eps, 1-eps]. Do not call np.log(0.0) yourself.
• Gradient clipping (neural nets). torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) before each optimizer step prevents catastrophic weight updates.
• Variance floors. p_i(1-p_i) → 0 for confident predictions causes exploding leaf weights in boosting. Set min_child_weight ≥ 1 and reg_lambda > 0.
• float32 accumulation. For losses summed over millions of samples, accumulate in float64.

6. Memory & Resource Constraints

For tabular ML:

Constraint	Symptom	Mitigation
DataFrame too large	MemoryError	Downcast dtypes, use pd.read_csv(chunksize=), float32 instead of float64
Preprocessing OOM	Large one-hot expansion	Use sparse_output=True (sparse matrices), or target encoding
CV OOM	cross_val_score(n_jobs=-1) spawns copies	Reduce n_jobs, use Memory caching, or subsample
Slow training	Hours per fit	HistGradientBoosting instead of GradientBoosting, or LightGBM

For neural nets:

Constraint	Symptom	Mitigation
GPU OOM forward	CUDA out of memory	Halve batch size
GPU OOM backward	OOM only after loss	Activation checkpointing
GPU idle, CPU pegged	Low SM utilization	Increase DataLoader(num_workers=); enable pin_memory=True
Loss = NaN	After LR change or bad batch	Lower LR 10×, add gradient clipping

Gradient accumulation — simulate a large batch on a small GPU:

python

accum_steps = 8 optimizer.zero_grad() for i, (xb, yb) in enumerate(loader): loss = criterion(model(xb), yb) / accum_steps loss.backward() if (i + 1) % accum_steps == 0: optimizer.step() optimizer.zero_grad()

7. Lifecycle-Specific Edge Cases

Stage 1 — Problem Definition:

• Optimizing the wrong metric. A model that maximizes accuracy on a 1% positive class will predict “no” for everything.
• Ignoring cost asymmetry. A false negative in cancer screening is not equal to a false positive. Quantify before modelling.

Stage 2 — Data Collection:

• Sampling bias. Web-scraped data over-represents the digitally engaged. Medical data over-represents the insured.
• Survivorship bias. Training on customers who stayed does not generalize to those who left.
• Label noise. Ground-truth labels are often wrong 5–15% of the time in real datasets.

Stage 3 — Cleaning:

• Median imputation on skewed data. Median is safer than mean for income, but it flattens the tails. Consider IterativeImputer or a missing-indicator feature.
• Winsorization before the split. Clipping at the 99th percentile of the full dataset is leakage. Compute quantiles on the training fold only.

Stage 5 — Feature Engineering:

• Target leakage. A feature that is a downstream consequence of the target (e.g., “collection_agency_contacted” predicting “defaulted”) produces beautiful offline metrics and fails in production.
• Train/serve skew. The single most common production ML failure. Ship engineer_features() inside the artifact.

Stage 9 — Deployment:

• Silent feature rename. A column rename in the source system breaks the model silently.
• Schema validation. Always validate input schema (types, ranges, required fields) before calling predict_proba.

Stage 10 — Monitoring:

• Monitoring only accuracy. Accuracy can stay flat while recall on the minority class collapses.
• No ground-truth feedback loop. In credit risk, defaults take 6–12 months to observe. Design a proxy metric for faster detection.
• Ignoring prediction drift. Even without label drift, if the distribution of predicted probabilities shifts, something has changed.

---

Summary & Best Practices

Key Takeaways

• The ML lifecycle is a cycle, not a line. Stage 10 feeds back into Stage 1. A model is never “done.”
• Training is ~10% of the work. The other 90% is problem framing, data, features, deployment, and monitoring.
• Problem definition determines everything downstream. The cost ratio (FN:FP = 15:1 in our example) directly determines the optimal decision threshold — which in our case cut business cost by 47% without any model change.
• Feature engineering outperforms hyperparameter tuning. Our engineered loan_to_income became the #1 feature by permutation importance; tuning added only +0.24 pp to CV ROC-AUC.
• Always split before you transform. Wrap all preprocessing in a Pipeline so leakage becomes structurally impossible.
• The default threshold of 0.5 is almost never optimal for imbalanced or cost-asymmetric problems.
• Touch the test set exactly once. Hyperparameter search on the test set is leakage.
• Permutation importance on held-out data is the correct tool for reporting feature relevance.
• Monitoring is not optional. PSI > 0.25 on any input feature is a retraining trigger.
• Ship the feature engineering function with the model. Training-serving skew is the most common production ML failure.

Best-Practice Checklist (Lifecycle-Ordered)

1. Write a ProjectSpec before writing any code. Business objective, ML objective, primary metric, cost of each error type, success threshold.
2. Audit the raw data. Assert on shape, dtypes, missingness, non-finite values, duplicates, class balance.
3. Winsorize and clean using training-fold statistics only. Compute quantiles inside the CV loop.
4. Run EDA on the training set. Check per-segment target rates and feature-target correlations to guide feature engineering.
5. Engineer features with domain knowledge. Ratios and interactions usually beat raw features. Verify improvements with correlation and permutation importance.
6. Split once, before any transformation. Stratified for classification, grouped for repeated entities, time-ordered for temporal data.
7. Wrap all preprocessing in a Pipeline. Imputation, scaling, and encoding must fit on the training fold.
8. Compare 3+ model classes by cross-validation. Include a linear baseline. Score with the metric from Stage 1.
9. Tune hyperparameters on the training set with GridSearchCV or RandomizedSearchCV. Expect modest gains — this is normal.
10. Evaluate once on the sealed test set. Report ROC-AUC, PR-AUC, per-class precision/recall, and the confusion matrix.
11. Optimize the decision threshold against the business cost matrix. Do not ship 0.5.
12. Package the model with its metadata. Store the threshold, feature names, training timestamp, library versions, and the engineer_features function in the artifact.
13. Validate input schema at serve time. Fail loudly on missing or out-of-range fields.
14. Monitor PSI and KS on every input feature. Alert at PSI > 0.10, retrain at PSI > 0.25.
15. Log everything for reproducibility. Seeds, versions, data hash, commit hash, hyperparameters.
16. Schedule retraining. Time-based (weekly/monthly), drift-based (PSI trigger), or performance-based (metric drop).

Suggested Next Steps

1. Implement a real deployment. Wrap the predict_default function in FastAPI, containerize with Docker, and deploy behind a load balancer.
2. Build a feature store. Use Feast or a simple Postgres table to guarantee the same feature values at train and serve time.
3. Add automated retraining. Airflow DAG or Prefect flow that runs Stages 2–8 on fresh data and registers a new model version.
4. Set up model registry. MLflow or Weights & Biases to version every training run with its metrics and artifacts.
5. Implement champion/challenger. Deploy the new model in shadow mode alongside the incumbent; compare live metrics before promoting.
6. Explore advanced drift detection. Beyond PSI: adversarial validation, domain-classifier AUC, and prediction-distribution monitoring.
7. Move to deep learning for unstructured data. See the Deep Learning Tutorial for CNNs, RNNs, and transformers.
8. Read the primary sources. See below.

Further Reading

• Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 28. — the paper that defined the ML lifecycle as an engineering discipline.
• Breck, E. et al. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data. — 28 concrete tests across data, model, infrastructure, and monitoring.
• Mitchell, M. et al. (2019). Model Cards for Model Reporting. FAT* ’19. — the standard for documenting model behavior, limitations, and intended use.
• Gebru, T. et al. (2021). Datasheets for Datasets. Communications of the ACM, 64(12). — documentation standards for training data.
• Gama, J. et al. (2014). A Survey on Concept Drift Adaptation. ACM Computing Surveys, 46(4). — the definitive review of drift detection and adaptation.
• scikit-learn User Guide: Common Pitfalls and Recommended Practices — scikit-learn.org/stable/common_pitfalls.html
• Google, Rules of Machine Learning: Best Practices for ML Engineering — developers.google.com/machine-learning/guides/rules-of-ml