Module 8 — Honest Evaluation
Every model in this course is governed by a complexity hyperparameter — k in k-NN, depth in a decision tree, the degree of a polynomial. Increasing model complexity drives training accuracy toward 100%. This closing module addresses the central methodological insight that follows: improvements in training accuracy beyond a certain point degrade generalization performance. Learning to measure generalization rigorously is what distinguishes a genuine model from one whose apparent performance is an artifact of overfitting to its training data.
The fundamental distinction: memorization is not learning
The objective of a model is to generalize — to perform accurately on observations it has not previously seen. However, the model is fit using only the training data, so a sufficiently flexible model can attain near-perfect training accuracy by memorizing each observation, including its noise component. This phenomenon is overfitting: high performance on the training set, poor performance on new data. The only valid empirical test of generalization is performance on data that was withheld from the training process.
Training versus test error: the characteristic U-curve
Below, a noisy nonlinear dataset is fit by a polynomial whose flexibility (degree) you control. Observe the two quantities as the degree is increased:
- Training error — computed on the points used to fit the polynomial. This quantity decreases monotonically: greater flexibility always permits a closer fit to the training data.
- Test error — computed on held-out points not used in fitting. This quantity initially decreases, reaches a minimum, then increases as the polynomial begins to fit noise rather than signal.
The minimum of the test-error curve identifies the optimal model complexity. Complexity below this value yields underfitting — the model lacks the flexibility to represent the underlying pattern. Complexity above this value yields overfitting — the model captures random variation in the training sample that does not generalize.
This activity needs JavaScript. The lesson below still covers everything.
Bias and variance: the two components of generalization error
Each side of the U-curve corresponds to a distinct source of error. Bias is error arising from a model too restricted to represent the underlying function: the model is systematically wrong in the same direction. Variance is error arising from a model so flexible that its fitted form is highly sensitive to the particular training sample: the model captures random fluctuations rather than signal. Bias and variance cannot, in general, be minimized simultaneously; selecting a model's complexity is therefore an exercise in balancing the two. This is the bias–variance trade-off, the unifying principle underlying every complexity hyperparameter encountered in this course.
Cross-validation: a more reliable estimate of generalization
A single train/test split yields an estimate of generalization that is subject to substantial sampling variability. k-fold cross-validation mitigates this variability: the dataset is partitioned into \( k \) disjoint folds, the model is trained on \( k - 1 \) folds and evaluated on the remaining fold, and this procedure is repeated for each fold. The reported performance is the mean of the \( k \) per-fold scores. Every observation serves as test data exactly once, producing a more reliable estimate of generalization than any single split.
from sklearn.model_selection import train_test_split, cross_val_score X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2) # withhold 20% for evaluation model.fit(X_tr, y_tr) model.score(X_te, y_te) # the valid estimate of generalization cross_val_score(model, X, y, cv=5) # 5-fold: mean of 5 held-out scoresimport numpy as np from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split, cross_val_score X, y = make_classification(n_samples=400, n_features=8, n_informative=5, random_state=0) # One split: train on 80%, judge on the held-out 20% X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0) model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr) print("accuracy on TRAINING data (optimistic estimate) =", round(model.score(X_tr, y_tr), 3)) print("accuracy on HELD-OUT data (unbiased estimate) =", round(model.score(X_te, y_te), 3)) # 5-fold cross-validation: averaging over five held-out partitions scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5) print("5-fold scores =", np.round(scores, 3)) print("cross-val mean =", round(scores.mean(), 3), "+/-", round(scores.std(), 3)) # A picture of what the numbers mean ------------------------------------------ import matplotlib.pyplot as plt train_acc = model.score(X_tr, y_tr) # optimistic — judged on data it learned test_acc = model.score(X_te, y_te) # unbiased — judged on unseen data fig, (axL, axR) = plt.subplots(1, 2, figsize=(9, 3.6)) # Left — the optimism gap: training accuracy is inflated vs the honest estimate bars = axL.bar(["Training\n(optimistic)", "Held-out\n(unbiased)"], [train_acc, test_acc], color=["#f59e0b", "#2563eb"]) for b, v in zip(bars, [train_acc, test_acc]): axL.text(b.get_x() + b.get_width() / 2, v + 0.01, f"{v:.3f}", ha="center", va="bottom", fontsize=10) axL.annotate("", xy=(1, test_acc), xytext=(1, train_acc), arrowprops=dict(arrowstyle="<->", color="#dc2626", lw=1.5)) axL.text(1.18, (train_acc + test_acc) / 2, f"optimism\ngap = {train_acc - test_acc:.3f}", color="#dc2626", fontsize=9, va="center") axL.set_ylim(0, 1.08); axL.set_ylabel("accuracy") axL.set_title("Single split: why training score lies") # Right — five held-out estimates scattering around their mean (mean +/- 1 std) folds = np.arange(1, len(scores) + 1) m, s = scores.mean(), scores.std() axR.bar(folds, scores, color="#2563eb", alpha=0.75) axR.axhline(m, color="#dc2626", lw=2, label=f"mean = {m:.3f}") axR.fill_between([0.4, len(scores) + 0.6], m - s, m + s, color="#dc2626", alpha=0.12, label=f"+/- 1 std = {s:.3f}") axR.set_xticks(folds); axR.set_xlim(0.4, len(scores) + 0.6); axR.set_ylim(0, 1.08) axR.set_xlabel("fold"); axR.set_ylabel("accuracy") axR.set_title("5-fold cross-validation: a steadier estimate") axR.legend(fontsize=8, loc="lower right") plt.tight_layout() plt.show()
The score produced by .score() on held-out data — and never on the training data — is the valid estimate of generalization performance. Cross-validation produces a more stable estimate of the same quantity. Click Run it yourself to compare the optimistic training-set score against the unbiased held-out score.
When you run it, the program prints these numbers and draws the chart below them — the left panel shows the optimism gap between the inflated training score and the honest held-out score; the right panel shows the five cross-validation folds scattering around their mean (the red band is \( \text{mean} \pm 1 \) standard deviation), making clear why averaging over folds gives a steadier estimate than any single split:
accuracy on TRAINING data (optimistic estimate) = 0.869 accuracy on HELD-OUT data (unbiased estimate) = 0.838 5-fold scores = [0.862 0.912 0.812 0.925 0.775] cross-val mean = 0.858 +/- 0.057
Project — synthesis across the course
The following questions integrate material from across the course: the modeling workflow, each model family, and the evaluation methodology by which all models are judged.
This activity needs JavaScript.