← All Neural Networks & Deep Learning modules

Module 3 — The Forward Pass

Putting neurons together · hands-on · about 25 minutes.

Module 1 established that a single neuron cannot represent XOR ("output 1 when exactly one input is 1") because XOR requires a non-linearly-separable decision boundary, and a single neuron implements only a linear one. The resolution is to compose neurons into a layered architecture. This module introduces such an architecture — a two-layer neural network — and the operation that propagates data through it from input to output, termed the forward pass.

Architecture: input, hidden, and output layers

A neural network is a sequence of neuron layers. Intermediate layers are termed hidden layers — designated as such because their activations are computed internally and are neither the network's input nor its output. Each hidden neuron receives the inputs and produces an activation; the output neuron receives the activations of the hidden layer and produces the final prediction. Data flows strictly in one direction: forward, from input to output.

Layer 1 (hidden) — two neurons each compute a linear decision boundary over the input space. One approximates "is at least one input active?"; the other approximates "is it not the case that both inputs are active?".
Layer 2 (output) — a single neuron computes the logical conjunction of the two hidden-layer outputs. "At least one active, and not both active" is precisely the XOR function.

Move the probe point below. The activation of each neuron is displayed as the inputs propagate forward through the network, and the output neuron produces the correct classification — a decision boundary not representable by any single linear neuron. Try it: drag the sliders or tap a corner button, and watch the brightness ripple from input to output.

This activity needs JavaScript. The lesson below still covers everything.

The forward pass as inference

Executing the forward pass on a trained network — supplying an input and reading the resulting output — is the operation referred to as inference. Every classification of an image, every next-token prediction by a language model, is a single forward pass through a substantially larger network with the same fundamental structure. The component absent from this presentation is the procedure by which the weights are learned — addressed in the following three modules on gradient descent, backpropagation, and depth.

A forward pass in Keras

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense

model = Sequential([
    Dense(2, activation='tanh', input_shape=(2,)),  # hidden layer: 2 neurons
    Dense(1, activation='sigmoid')                  # output neuron
])
y = model.predict(X)   # the forward pass — input flows in, answer comes out

This corresponds to the 2-2-1 architecture displayed in the activity above. .predict() executes the forward pass; the network's weights are determined by the training procedure.

AI anchor — every prediction is computed by a forward pass The forward pass is the inference step common to all of AI. A vision model classifying a medical image, a recommender system ranking content, a language model selecting its next token — each is a single forward pass through a deep network, structurally identical to the propagation demonstrated here but with millions or billions of neurons rather than three. A thorough understanding of this small example transfers directly to large-scale models.

Check your understanding

Answer a short set of questions on the forward pass.

This activity needs JavaScript.

Why this matters next The network in this module was supplied with pre-computed weights. Module 4 addresses the central question: how does a network learn these weights? You will adjust the learning rate and observe the loss curve descend — or diverge — as gradient descent trains a network from a random initialization.

Summary: a neural network composes neurons into a sequence of layers, and the forward pass propagates an input through each layer in succession. The hidden layer constructs intermediate features that the output neuron combines, enabling the network to represent non-linear decision boundaries — such as XOR — that no single neuron can.

Next: Gradient Descent in Practice →