Module 13 · Project 7 · Due the date on Canvas · Ethics, Privacy & Dashboard Detective · adaptive competency DV21
Fairness & base-rate mathematics
Mission progress0%
Begin by evaluating base-rate fairness and Simpson's paradox.
Module 13 · Step 1 of 9 · Watch8–10 min
Simpson's paradox, conditional probabilities, and denominator preservation. Compare subgroup performance against aggregate metrics to prevent mathematically distorted conclusions.
Subgroup rate\(P(A|G_i) = x_i / n_i\)
Pooled rate\(P(A) = \sum x_i / \sum n_i\)
Impact Ratio\(DIR = \frac{P(A|G_1)}{P(A|G_2)}\)
- Why a pooled comparison can reverse. A pooled rate is a weighted average of the subgroup rates, weighted by each group's share of students in each subgroup. Program A has 10 students in an easy course (9 pass, 90%) and 90 in a hard one (54 pass, 60%): pooled, \( 63/100 = 63\% \). Program B has 90 in the easy course (72 pass, 80%) and 10 in the hard one (5 pass, 50%): pooled, \( 77/100 = 77\% \). A wins inside both courses and loses pooled, because its students sit mostly in the hard course. That is Simpson's paradox; report the subgroup rates with their denominators before any pooled claim.
- Every rate names its numerator and denominator. Publish the count and who counts in the denominator, such as which students were eligible, the time window and any exclusions. A tile reading “91% completion” without them cannot be checked.
- Association is not cause. When students chose whether to take part, an observational comparison cannot show cause: the students who opted in may differ in preparation, motivation or time. Caption the difference as an association, and name selection and prior preparation as reasons it may not be causal.
Fairness & base-rate lab loading…
Classifier threshold, error trade-offs & subgroup fairness
A risk-score model is only a dashboard away from a decision. Move the threshold and watch the confusion matrix, the ROC operating point, and — the part aggregate metrics hide — how the same threshold treats the two student groups differently.
The classifier threshold explorer needs JavaScript.