Module 9 — Limits & Continuity

Pillar 3 · Optimization · best taken before Module 6 · hands-on · about 25 minutes.

Module 6 defines the derivative as the slope of a function at a single point. That definition rests on something the course has not yet stated: a limit — the value a function approaches as its input closes in on a point. And a limit is only well behaved when the function has no break there. This module supplies both ideas, so that the gradient rule reads as a consequence rather than a formula to memorize.

This module is numbered 9 because it was added after the original eight, but it is placed in Pillar 3 for a reason: work through it before Module 6 and the derivative arrives already explained.

The payoff is not only notational. The reason a neural network can be trained at all is that it is built, deliberately, out of continuous pieces — and the reason a model is trained on log loss while being graded on accuracy is that accuracy is discontinuous. Both facts follow from this module.

What this module is not. There are no epsilon–delta proofs here, and no symbolic limit laws to memorize. The goal is the practitioner's version: what continuity guarantees, how to detect a break, and what a break costs a model.

The limit: where a function is heading

The limit of \( f \) at \( a \), written \( \lim_{x \to a} f(x) = L \), says that the output gets arbitrarily close to \( L \) as the input gets close to \( a \) — from either side. Notice what the statement never mentions: the value \( f(a) \) itself. The limit describes the approach, not the arrival, and the two can disagree. They can even exist independently: a function can have a perfectly good limit at a point where it is not defined at all.

The classic case is \( f(x) = \dfrac{x^2 - 1}{x - 1} \) at \( x = 1 \). Substituting gives \( \dfrac{0}{0} \), which is not a number, so \( f(1) \) does not exist. Yet everywhere else \( f(x) = x + 1 \), so as \( x \) approaches \( 1 \) the output approaches \( 2 \). The graph is the line \( y = x + 1 \) with a single point punched out of it.

Choose a function below and walk the input toward the marked point from both sides. Watch the two columns close in on the same number — or fail to.

This activity needs JavaScript. The limit is the value f(x) approaches as x nears a point from both sides; it can exist even where f(a) itself is undefined, as with (x²−1)/(x−1) at x = 1, whose limit is 2.

Continuity: the three-part test

A function is continuous at \( x = a \) when three conditions all hold, in this order:

\[ \text{(i) } f(a) \text{ is defined} \qquad \text{(ii) } \lim_{x \to a} f(x) \text{ exists} \qquad \text{(iii) } \lim_{x \to a} f(x) = f(a) \]

The order matters, because each condition presupposes the one before it. Condition (iii) is the substantive one: it says the arrival matches the approach. When a function fails the test, the useful answer is never "it is discontinuous" — it is which condition failed, because that names the kind of break and tells you whether it can be repaired.

"Continuous" is often taught as "you can draw it without lifting your pencil." That picture is a helpful start and an unreliable finish: it says nothing about a function defined by cases, and nothing about a point where the formula is undefined. Run the test instead.

Run all three conditions on each function below. The widget reports each condition separately, names the failure, and classifies the break.

This activity needs JavaScript. A function is continuous at a when f(a) is defined, the limit at a exists, and the two are equal. A removable break is a hole, a jump has different one-sided limits, and an infinite break grows without bound.

Piecewise functions: everything happens at the seam

A piecewise function is defined by one rule on part of its domain and another rule elsewhere. Each rule may be perfectly continuous on its own stretch, and the function as a whole can still break — because continuity of the pieces says nothing about the point where they meet. The only place worth checking is the seam, and the check is the same three-part test: the left-hand limit, the right-hand limit, and the assigned value must all agree.

Below, the branches meet at \( x = 1 \) and one branch carries an adjustable constant \( k \). Move it until the seam closes.

This activity needs JavaScript. A piecewise function is continuous at its seam only when the left limit, the right limit, and the assigned value all agree — solve for the constant that makes them match.

AI anchor — ReLU is a piecewise function that was built to be continuous The most widely used activation function in deep learning is \( \mathrm{ReLU}(x) = \max(0, x) \) — piecewise, with the seam at \( x = 0 \): it is \( 0 \) to the left and \( x \) to the right. Its two branches agree at the seam, both giving \( 0 \), so it passes the three-part test there. That is not luck. An activation with a jump at \( 0 \) would make the network's output leap in response to an arbitrarily small change in its input, and — as the last section shows — would leave gradient descent with nothing to follow.

Continuity survives composition

Sums, products, quotients (where the denominator is nonzero), and compositions of continuous functions are continuous. This closure property is what makes reasoning about large models tractable at all: no one verifies continuity of a hundred-layer network directly. Instead each layer is built from continuous pieces — a matrix multiply, a bias addition, an activation — and closure carries the property through the entire stack.

The logistic regression of Module 8 is the small version of the same argument. The chain

\[ \mathbf{x} \;\longmapsto\; z = \mathbf{w}\cdot\mathbf{x} + b \;\longmapsto\; \hat{y} = \sigma(z) \;\longmapsto\; L(\hat{y}, y) \]

is a composition of a linear map, the sigmoid, and the log loss. Every stage is continuous, so the loss is a continuous function of the weights — which is precisely the property gradient descent needs. Break any one stage and the guarantee is gone for the whole chain.

The Intermediate Value Theorem

Continuity is not merely the absence of a defect; it buys a guarantee. The Intermediate Value Theorem states that if \( f \) is continuous on the closed interval \( [a, b] \) and \( N \) is any value between \( f(a) \) and \( f(b) \), then \( f(c) = N \) for at least one \( c \) in \( (a, b) \). In the common case: if a continuous function is negative at one endpoint and positive at the other, it must cross zero somewhere in between.

Two cautions. The theorem is an existence statement — it promises a crossing, not its location, and not that there is only one. And it needs continuity on the whole interval: a function that jumps across the target value can step over it without ever attaining it.

This activity needs JavaScript. If a continuous function has opposite signs at the two ends of an interval, it must cross zero somewhere between them — repeatedly halving the interval brackets that crossing. A function with a jump can step over zero without ever attaining it.

Continuous is not the same as smooth

Continuity rules out breaks. It does not deliver a slope. A function is differentiable at a point when it has a single well-defined tangent there — and differentiability is the stronger condition: every differentiable function is continuous, but not every continuous function is differentiable.

The standard counterexample is \( f(x) = |x| \) at \( x = 0 \) — equivalently \( \mathrm{ReLU} \), which is the same corner with the left branch flattened. The graph has no break, so the function is continuous. But approaching \( 0 \) from the left the slope is \( -1 \), and from the right it is \( +1 \). Two different answers means no single tangent line, so the derivative at \( 0 \) does not exist. A corner is continuous and non-differentiable at once.

This is a live implementation detail rather than a technicality. Deep-learning frameworks are asked for the derivative of ReLU at exactly \( 0 \) constantly, and they return \( 0 \) — a subgradient, chosen by convention because the derivative genuinely does not exist there. Knowing that the value is a convention, not a computation, is the difference between understanding the framework and trusting it blindly.

Why training needs a continuous objective

Here is where the module pays for itself. Gradient descent reads the slope of the loss and steps downhill. That procedure needs the loss to respond to a small change in the weights by a small change in value — which is exactly what continuity provides, and what a jump destroys.

Consider a classifier built on a hard threshold: output \( 1 \) when the score exceeds \( 0 \), otherwise \( 0 \). That step function is flat on both sides of the threshold and jumps across it. Its derivative is \( 0 \) everywhere it exists, and undefined at the jump. A flat slope tells gradient descent nothing about which direction improves the model, so training stalls immediately — not because the optimizer is weak, but because the objective supplies no signal.

The fix is to replace the discontinuous objective with a continuous surrogate. The sigmoid \( \sigma(z) = \frac{1}{1 + e^{-z}} \) is the step function's continuous stand-in: it delivers the same S-shaped decision behavior with a nonzero slope everywhere, so there is always a direction to step. Compare them below.

This activity needs JavaScript. A hard step function is flat on both sides and jumps at the threshold, so its slope is 0 or undefined and gradient descent gets no direction to move. The sigmoid is the continuous surrogate: it has a usable slope everywhere.

AI anchor — why the metric and the objective are different functions Accuracy — the fraction of examples classified correctly — is piecewise constant in the weights. Nudge a weight slightly and, almost always, every prediction keeps its label and accuracy does not move at all; then at one exact setting a prediction flips and accuracy jumps by a whole example. Flat almost everywhere, discontinuous at the flips: it has no usable gradient anywhere. This is why no model is trained on accuracy. Training minimizes a continuous surrogate — the log loss of Module 8, which responds smoothly to every weight — while accuracy is reported as the metric humans care about. The gap between the objective a model optimizes and the metric it is judged on is a direct consequence of continuity.

Check your understanding

Seven questions across limits, the continuity test, composition, the IVT, and what all of it means for training.

This activity needs JavaScript.

Why this matters next Module 6 defines the derivative as the slope at a point — a limit of the difference quotient, which exists only where the function is well behaved. You now have both halves of that sentence. The same ideas return in Module 8, where the sigmoid and the log loss appear as the continuous pieces that make one gradient step possible, and in Course 4, where every activation function is chosen with the tradeoff of this module in mind.

Summary: a limit is the value a function approaches at a point; continuity at \( a \) means \( f(a) \) is defined, the limit exists, and the two are equal; breaks come in three kinds — removable, jump, and infinite — and only the removable kind can be repaired. Continuity survives sums, products, and composition, which is why a deep network is continuous end to end; it guarantees the crossing the Intermediate Value Theorem promises; it is weaker than differentiability, as the corner in \( \mathrm{ReLU} \) shows; and it is the property that makes a loss function trainable at all.

Next: Derivatives & Gradient Descent →