Review and practice

What the exam will probably look like

The format is 12 multiple-choice questions plus 2 or 3 problems you solve by hand. The review sheet contained exactly three problems, and two of them are the same types as Quiz 2. Expect the same three with new numbers.

Problem type 1 · Computational graph and backprop

The method, every time

1. Break the expression into single operations. One node per operation, innermost first. Give each intermediate a name (a, b, c…).
2. Forward pass, left to right. Write each value on its edge.
3. Backward pass, right to left. Start with 1 at the output. At each node:
   gradient on an input = upstream gradient × local derivative.
4. Add at branches. If a variable feeds several nodes, its gradient is the sum over all those paths.
5. Sanity-check with algebra if you have a minute: simplify the function for the active branch and differentiate directly.

Local derivatives to know cold

Worked example 1 · the review-sheet problem

Forward. Name the pieces: \(a = x\cdot y\), \(b = x/y\), \(c = \min(b, x)\), \(f = a \cdot c\).

Backward. Start with \(\partial f/\partial f = 1\) and walk right to left.

Add the branches.

Worked example 2 · the Quiz 2 problem

Forward: \(1\cdot 2 = 2\), \(0.5 \cdot 1 = 0.5\), sum \(= 2.5\), plus bias \(= 3.5\), ReLU \(= 3.5\). Backward: ReLU passes 1 because \(3.5 > 0\); both add nodes pass 1 along; each multiply node hands every input the other input.

Worked example 3 · the sigmoid graph from the MLP slides

The slides walk through \(f(x, w) = \dfrac{1}{1 + e^{-(w_0 x_0 + w_1 x_1 + w_2)}}\) with \(w_0 = 2,\; x_0 = -1,\; w_1 = -3,\; x_1 = -2,\; w_2 = -3\), one node at a time. Rework it once with the gradients hidden.

• \(1/x\) node: local \(= -1/1.37^2 = -0.53\). So \(1.00 \times -0.53 = -0.53\).
• \(+1\) node: local \(= 1\). Still \(-0.53\).
• \(\exp\) node: local \(= e^{-1} = 0.37\). So \(-0.53 \times 0.37 = -0.20\).
• \(\times(-1)\) node: local \(= -1\). So \(+0.20\).
• Both add nodes pass \(0.20\) to every input, so \(\partial f/\partial w_2 = 0.20\).
• Multiply nodes swap: \(w_0\) gets \(x_0 \times 0.20 = -0.20\), \(x_0\) gets \(w_0 \times 0.20 = 0.40\), \(w_1\) gets \(x_1 \times 0.20 = -0.40\), \(x_1\) gets \(w_1 \times 0.20 = -0.60\).

Practice · do these on paper

Problem type 2 · Weight-matrix shapes

The check that catches mistakes: write the multiplication and make sure the inner dimensions agree. \((5 \times 8)(8 \times 9) \rightarrow 5 \times 9\).

Worked example · the review-sheet problem

The input to layer 2 is the output of layer 1, which has 9 values per sample, plus the bias 1. The final output for the batch is 5 × 3. Total parameters: 72 + 30 = 102.

Worked example · the Quiz 2 problem

Three layers with 12, 8 and 4 neurons, 10 samples of dimension 6, \(\text{net} = XW\), bias at each layer.

Output for the whole batch: 10 × 4. ReLU and softmax change values, not dimensions. Each row holds 4 probabilities that sum to 1.

Practice

Problem type 3 · Softmax and cross-entropy

The method

1. Softmax turns the raw outputs \(z\) into probabilities: \[ \hat{y}_k = \frac{e^{z_k}}{\sum_j e^{z_j}} \]
2. Cross-entropy compares them with the target \(y\): \[ L = -\sum_k y_k \,\ln(\hat{y}_k) \]
3. If the target is one-hot, every term but one is multiplied by 0, so \(L = -\ln(\hat{y}_{\text{true class}})\).

Worked example · the review-sheet problem

Step 1, softmax. \(e^1 = 2.718\), \(e^2 = 7.389\), sum \(= 10.107\).

Step 2, the loss. It depends on the desired output:

Practice

Other by-hand problems that could appear

Less likely than the three above, but each one is worked in the notes or was set as homework.

Perceptron update rule

\(\hat{y} = 1\) if \(z \ge 0\), else 0. For each example: \(w \leftarrow w + \eta\,(y - \hat{y})\,x\) and \(b \leftarrow b + \eta\,(y - \hat{y})\). Correct prediction means no change.

The notes train an AND gate from \(w = (0, 0),\; b = 0,\; \eta = 1\). Epoch 1:

After epoch 1 the model is \(w = (1, 1),\; b = 0\). Test it: (0, 0) gives \(z = 0\), so ŷ = 1, which is wrong, so training continues. Note that \(z = 0\) counts as class 1.

One gradient-descent step for linear regression

Data from the notes: \((x, y) = (1, 2)\) and \((2, 4)\). Model \(z = wx + b\), start at \(w = 0,\; b = 0\), using the notes’ gradients \(\partial L/\partial w = \sum (z_i - y_i)\,x_i\) and \(\partial L/\partial b = \sum (z_i - y_i)\).

• Predictions: \(z = 0, 0\). Errors: \(-2,\; -4\). Starting MSE \(= (4 + 16)/2 = 10\).
• \(\partial L/\partial w = (-2)(1) + (-4)(2) = -10\). \(\partial L/\partial b = -6\).
• With \(\alpha = 0.5\): \(w = 5,\; b = 3\). New predictions 8 and 13, MSE \(= 58.5\). The loss went up: overshoot.
• With \(\alpha = 0.1\): \(w = 1,\; b = 0.6\). New predictions 1.6 and 2.6, MSE \(= 1.06\). The loss went down.

Forward pass through a small network

A hidden layer lets a network solve XOR. One set of weights that works: \(h_1 = \text{ReLU}(x_1 + x_2)\), \(h_2 = \text{ReLU}(x_1 + x_2 - 1)\), \(y = h_1 - 2h_2\).

Single neuron with a step activation

The notebook exercises: design AND and OR, and evaluate a given neuron.

• AND: \(w = (1, 1),\; b = -1.5\). Only (1, 1) gives \(z \ge 0\).
• OR: \(w = (1, 1),\; b = -0.5\). Only (0, 0) gives \(z < 0\).
• Evaluate \(z = 2x_1 - x_2 - 1\): (1, 0) → \(z = 1\) → 1.   (0, 2) → \(z = -3\) → 0.   (2, 1) → \(z = 2\) → 1.

Rewrite a loop with NumPy

Quiz 1 asked for this. The pattern is always: drop the index and apply the expression to the whole array.

# loop version
for i in range(len(A)):
    B[i] = A[i] ** 2 + 2 * A[i] + 1

# vectorized
B = A ** 2 + 2 * A + 1          # note ** for power, not ^

For by-hand convolution and pooling problems, see the CNN page.

Revision · NumPy, broadcasting, ufuncs, vectorization

• NumPy is an open-source Python package for multidimensional arrays, designed for scientific computing. Main class: ndarray.
• All elements of an array must be the same type. An ndarray is not a Python list.

Universal functions (ufuncs) such as np.sin, np.cos, np.exp, np.sqrt operate element-wise on an array and return an array.

Broadcasting

Broadcasting is how NumPy does arithmetic between arrays of different shapes. Two rules:

1. If the arrays have different numbers of dimensions, prepend 1s to the shape of the smaller one.
2. An axis of size 1 is stretched (copied) to match the other array along that axis.

After that, every axis must match, otherwise it is an error. Line the shapes up from the right:

Vectorization

Applying an operation to the whole array at once instead of looping. In the notes, np.dot on 10 million elements took about 9 ms and the Python loop about 3.2 s.

Revision · Measurement scales

Each scale keeps everything from the one before and adds one property.

• Identity: each value has a unique meaning. Order: values can be ranked. Equal intervals: units along the scale are equal. True zero: zero means none of the quantity.
• Nominal and ordinal are the two categorical types.
• Interval has no true zero, so ratios mean nothing. Is 40 °C twice as hot as 20 °C? Convert to Fahrenheit (104 vs 68) and the “twice” is gone.
• Why the scale matters: it decides which operations and statistics are valid. Pick the wrong one and you do arithmetic that has no meaning.

Revision · Machine learning and the neuron

• AI ⊃ Machine learning ⊃ Deep learning. AI also includes rule-based systems, search and planning, and robotics.
• Machine learning (Arthur Samuel, 1959): the ability to learn without being explicitly programmed. Machines learn patterns from data and make decisions on their own.

Biological neuron ↔ artificial neuron

Anatomy of one neuron: inputs \(x_i\), weights \(w_i\) (importance of each input), bias \(b\) (shifts the activation), weighted sum \(z = w_1x_1 + \dots + w_nx_n + b\), activation function \(f\), output \(y = f(z)\).

Revision · Single-layer networks

One layer of neurons connected directly to the inputs, feed-forward. They all compute \(z = w^{\top}x + b\) and differ only in the activation and the loss.

Key formulas

• Decision boundary is \(z = 0\): a line for 2 inputs, a plane for 3. The bias shifts it; with no bias it must pass through the origin.
• Gradients: linear regression uses \((z - y)\,x\); logistic regression uses \((\hat{y} - y)\,x\) with \(\hat{y} = \sigma(z)\). Same form, different prediction.
• Logistic decision rule: class 1 if \(\hat{y} \ge 0.5\), which is the same as \(z \ge 0\).
• Perceptron: the step function is not differentiable, so no gradient descent. It converges only if the data is linearly separable. An epoch is one full pass over the training set.
• Why softmax for multiclass: K separate sigmoids do not sum to 1, which makes no sense when exactly one class is true. Softmax makes the outputs positive, summing to 1, and competing.
• One-hot encoding: class k of K becomes a vector of 0s with a 1 in position k. Then \(L = -\ln(\hat{y}_{\text{true}})\). It punishes confident wrong predictions heavily.
• Prediction for multiclass: compute softmax, take the argmax.
• Chain rule: \(\dfrac{dy}{dx} = \dfrac{df}{dg}\cdot\dfrac{dg}{dx}\). Example: \(y = \sin(x^2) \Rightarrow \cos(x^2)\cdot 2x\).
• Can solve: linearly separable problems (AND, OR, linear and logistic regression). Cannot solve: XOR.

Revision · Multi-layer networks and activations

Five limitations of a single-layer network

1. It can only model linear functions (works for AND and OR, fails for XOR).
2. It cannot capture complex patterns: no curves, clusters or complex boundaries.
3. Limited feature extraction: no hidden layers, so no intermediate representations.
4. Limited capacity: few parameters.
5. Sensitive to noisy data: generalises poorly on noisy or overlapping data.

Hidden layers

A hidden layer is any layer between the input and the output layer. It contains inputs, weights, a bias and an activation function, and performs intermediate computations. Hidden layers matter because they introduce non-linearity, learn hierarchical features, increase expressiveness, enable complex decision boundaries and make deep learning possible.

Activation functions

Vanishing gradient: backprop multiplies one local derivative per layer. Sigmoid and tanh are flat for large \(|z|\), so those derivatives are close to 0, and after many layers the product is almost nothing. The early layers stop learning. ReLU’s derivative is exactly 1 for positive inputs, so the gradient survives.

From the MLP slides

• Linear classifier: \(f(x) = Wx\). 2-layer network: \(f(x) = W_2\,\text{act}(W_1 x)\). 3-layer: \(f(x) = W_3\,\text{act}(W_2\,\text{act}(W_1x))\). “2-layer” means two layers of weights.
• CIFAR-10: \(32\times32\times3 = 3072\) inputs, 10 classes. 2-layer: 3072 → 128 → 10, “learn 128 shared templates instead of 10 separate ones”. 3-layer: 3072 → 512 → 128 → 10.
• MLP = fully connected network.
• Traditional vision pipeline: hand-designed features (such as a histogram of gradients), then a linear classifier. Neural networks learn the features automatically, and learn features and classifier end-to-end, using multiple layers.

Types of layered architectures

Revision · Back-propagation theory

• Training updates each layer with SGD: \(W_k \leftarrow W_k - \alpha\,\dfrac{\partial e}{\partial W_k}\). Backprop is how you get \(\partial e/\partial W_k\) for every layer.
• It is the chain rule, applied from the output back to the input.
• For any layer: \(\dfrac{\partial e}{\partial W_k} = \underbrace{\dfrac{\partial e}{\partial h_k}}_{\text{from the higher layer}} \cdot \underbrace{\dfrac{\partial h_k}{\partial W_k}}_{\text{current layer}}\)
• Each layer does two jobs: compute \(\partial h_k/\partial W_k\) to update its own weights, and compute \(\partial h_k/\partial h_{k-1}\) to pass the gradient down: \(\dfrac{\partial e}{\partial h_{k-1}} = \dfrac{\partial e}{\partial h_k}\cdot\dfrac{\partial h_k}{\partial h_{k-1}}\).
• If a layer’s output feeds two places, the incoming gradients are added.

The slides end with “derive the 2-layer case yourself”. The middle row is that derivation: \(\partial e/\partial h_2 = -2(y - h_2)\), \(\partial h_2/\partial h_1 = W_2\), \(\partial h_1/\partial W_1 = x\), and you multiply the three.

Traps: where the notes can mislead you

I checked every worked number in the course files. These are the places where a note and the correct answer disagree, or where two parts of the course use different conventions.

Short-answer bank

Quiz 1 was ten 3-point short answers, and its grading key gave marks for specific key ideas. Say the answer out loud, then open it. The words in bold are the ones that earn the marks.

Asked on Quiz 1

Likely new ones

MCQ drill

62 questions across every topic except CNNs (those are on the CNN page). Each one shows the reason as soon as you answer. Your progress is saved in this browser.

Answer key for the review sheet’s 15 MCQs

You said you got nearly all of these. Use this to confirm the ones you were unsure about.