Module 7 — Learning Analytics

Dashboards, leading indicators, and responsible data · about 30 minutes.

You now understand how an adaptive system tracks learner knowledge internally. But that data doesn't stay inside the algorithm — it surfaces as dashboards, progress reports, and intervention alerts for teachers, administrators, and learners themselves. Learning analytics is the field concerned with measuring, collecting, analyzing, and reporting data about learners and learning contexts, with the goal of understanding and improving learning.

That goal sounds unambiguously good. And it can be — but only if the right metrics are being measured, interpreted honestly, and used ethically. In practice, dashboards are plagued by metrics that look impressive and mean nothing, and by data practices that erode the trust of the learners they're supposed to serve. This module is about how to tell the difference.

Leading vs. lagging indicators

One of the most important distinctions in any data system — and one that is routinely ignored in ed-tech dashboards — is between leading indicators and lagging indicators.

• A leading indicator shows up early and predicts a later outcome. It gives you a window to act before the outcome is determined. Examples: a student's weekly retrieval-practice streak, mastery on Unit 1 within the first three days of a course, percentage of skill checks answered correctly on the first attempt in week 2.
• A lagging indicator can only be measured after the outcome has occurred. It confirms whether you succeeded, but it's too late to change course. Examples: final exam score, course completion rate, certification awarded.

A good learning dashboard surfaces leading indicators early enough for intervention. If the only metric on a teacher's screen is final exam scores, the system is a rearview mirror — useful for post-mortem analysis, useless for helping the students in front of you right now. The mastery estimates that BKT produces (Module 6) are valuable precisely because they are a leading indicator: a skill below threshold at mid-unit predicts poor performance on the unit assessment, and a teacher who sees that signal can act.

See it: the intervention window

Here is a cohort of students. Their week-1 mastery is a leading indicator; their final score is the lagging outcome it predicts (notice the points already trend upward — the early signal really does forecast the result). Now play the dashboard designer: choose which week you first see the at-risk signal and how hard you intervene. Watch how many students you can pull above the pass line — and what happens when the only number you have is the final score in week 6.

Vanity metrics vs. actionable metrics

Not every metric that goes up and to the right is a good metric. Vanity metrics look impressive in a product demo or investor deck but don't connect to learning outcomes and don't guide decisions. Actionable metrics are directly tied to a decision: if this number goes down, I know what to do about it.

The classic vanity metrics in ed-tech are:

• Total logins — measures access, not engagement with content; a student can log in and immediately log out.
• Total page views — measures exposure, not processing; passive scrolling leaves no learning trace.
• Total minutes spent — time-on-task is a weak proxy for learning when the task is passive. Durable learning requires effortful retrieval (Module 2), and minutes reading ≠ minutes retrieving.
• Number of registered users — a marketing number, not a learning number.

Contrast those with genuinely actionable metrics:

• Percentage of students below mastery threshold on skill X — tells a teacher exactly which skill to reteach and which students need support.
• Items overdue for spaced review per student — tells the system (and the student) exactly what to practice today.
• Days since last retrieval-practice session — an early warning sign that spacing is breaking down before forgetting has occurred.
• First-attempt accuracy on new skills vs. review items — distinguishes encoding strength from retrieval fluency.

The test for actionability is simple: if this metric changes, what decision changes? If the answer is "none," it is a vanity metric, regardless of how prominently it is displayed.

Classify the metrics: Leading or Lagging?

Sort each metric into the correct category. Think carefully — some that feel like early signals are actually outcome measures.

Classify the metrics: Actionable or Vanity?

Now classify each metric by whether it connects to a decision a teacher or learner could make right now.

Ethics and privacy in learning analytics

Learning analytics involves collecting detailed behavioral data about people — often people who have little practical choice about whether to participate, in contexts (schools, required training programs) where the power asymmetry between institution and learner is large. That combination creates serious ethical obligations.

The core duty: use data to help, not to surveil

The primary ethical obligation in learning analytics is straightforward: collect learner data only to improve learning for that learner, with their knowledge and consent, and protect it accordingly. Data collected for one purpose (academic support) must not be repurposed for another (employer screening, behavior tracking unrelated to learning) without explicit, renewed consent. The moment data becomes a tool of surveillance rather than support, the educator-learner relationship is broken.

Minimization, transparency, and consent

Good data practice in learning analytics follows three principles:

• Minimize collection. Collect only what is necessary to improve the learning experience. Every additional data point is a potential privacy risk if stored, breached, or misused. Ask before collecting: "What decision will this data enable, and is that decision worth the risk?"
• Be transparent. Learners should know what is being collected, why, how long it is kept, and who has access. Opacity breeds distrust, and distrust undermines the collaborative relationship that makes adaptive learning work.
• Obtain meaningful consent. In institutional settings, "consent" often defaults to a checkbox buried in terms-of-service. Meaningful consent is informed, specific, and freely given — meaning the learner must have a genuine alternative if they decline.

Algorithmic bias

A less-discussed but serious concern: the models that drive adaptive systems — including BKT — are trained on historical data, and historical data reflects existing inequalities. If a model learns from data generated by students who were systematically under-resourced, it may produce lower prior estimates (\( p(L_0) \)) for students from similar backgrounds, creating a self-fulfilling prophecy: lower priors lead to more remedial content, less challenge, and slower progression. Responsible deployment of learning analytics requires ongoing audit for differential outcomes across demographic groups, and a commitment to updating models when disparate impact is detected.

Next: Does It Actually Work? Evaluating Ed-Tech →