Before you start
- Suggested time: 50 minutes. About 12 minutes for Part A, 35 for Part B, 3 to check.
- Part A gives no feedback until you press “Grade my exam” at the bottom, like a real paper.
- For Part B, write full working: the graph, every forward value, every gradient.
Part A · Multiple choice (12 questions)
One choice per question. You can change an answer until you grade.
Part B · Problems
Draw the computational graph. Show all forward values and the backward partial derivative at each node. Compute the numerical values of \(\partial f/\partial x\) and \(\partial f/\partial y\).
Solution and marking
Forward. \(a = x + y = 8\); \(b = x\cdot y = 12\); \(c = y/x = 3\); \(d = \min(12, 3) = 3\) (c is selected); \(f = a \cdot d = 24\).
Backward. \(\partial f/\partial a = d = 3\); \(\partial f/\partial d = a = 8\). The min sends 8 to \(c\) and 0 to \(b\).
| Path | Local derivative | × upstream | Value |
|---|---|---|---|
| x through \(a = x + y\) | 1 | × 3 | 3 |
| y through \(a\) | 1 | × 3 | 3 |
| x, y through \(b = x\cdot y\) | y, x | × 0 | 0 |
| x through \(c = y/x\) | \(-y/x^2 = -1.5\) | × 8 | −12 |
| y through \(c\) | \(1/x = 0.5\) | × 8 | 4 |
Check. The active branch is \(f = (x + y)\dfrac{y}{x} = y + \dfrac{y^2}{x}\). So \(\partial f/\partial x = -y^2/x^2 = -9\) and \(\partial f/\partial y = 1 + 2y/x = 7\).
- Graph drawn with one operation per node2
- All forward values shown: 8, 12, 3, 3, 242
- Min routes the gradient to y/x only, and branch gradients are added2
- ∂f/∂x = −92
- ∂f/∂y = 72
Consider a three-layer neural network with:
- Layer 1: 10 neurons · Layer 2: 6 neurons · Layer 3: 2 neurons
- Training set: 8 samples, each of dimension 4
- \(\text{net} = XW\), with the bias included in the first row of each weight matrix
(a) What are the shapes of the three weight matrices? (b) The last layer uses softmax. What is the shape of the output for the entire batch? (c) How many trainable parameters does the network have?
Solution and marking
| Layer | Rows = inputs + 1 | Columns = neurons | Shape | Parameters |
|---|---|---|---|---|
| 1 | 4 + 1 = 5 | 10 | 5 × 10 | 50 |
| 2 | 10 + 1 = 11 | 6 | 11 × 6 | 66 |
| 3 | 6 + 1 = 7 | 2 | 7 × 2 | 14 |
(b) 8 × 2: one row per sample, two probabilities per row summing to 1. (c) 50 + 66 + 14 = 130.
Check for layer 1: X (8×5) · W₁ (5×10) = 8×10.
- W₁ is 5 × 10 (bias counted)2
- W₂ is 11 × 6 and W₃ is 7 × 22
- Output is 8 × 21
- 130 parameters1
A network with two nodes in its last layer produces the output \(\begin{bmatrix}2\\0\end{bmatrix}\) for some input. The desired output is \(\begin{bmatrix}0\\1\end{bmatrix}\).
(a) Apply softmax. (b) Compute the cross-entropy loss using the natural log. (c) What would the loss be if the desired output were \(\begin{bmatrix}1\\0\end{bmatrix}\) instead, and what does the comparison tell you?
Solution and marking
(a) \(e^2 = 7.389\), \(e^0 = 1\), sum \(= 8.389\). \(\hat{y} = [0.881,\; 0.119]\).
(b) \(L = -\ln(0.119) =\) 2.127. Shortcut: \(\ln\hat{y}_2 = 0 - \ln 8.389 = -2.127\).
(c) \(L = -\ln(0.881) =\) 0.127. The same output costs about 17 times more when it is wrong: cross-entropy punishes confident wrong predictions heavily and rewards confident correct ones.
- Softmax gives [0.881, 0.119]2
- Loss is 2.127 (about 2.13)2
- Loss 0.127 for the other target, with the explanation2
Using the 4×4 input and the 2×2 filter below (no padding, slide one step at a time, no flipping):
\[ g = \begin{bmatrix} 2 & 0 & 1 & 3 \\ 4 & 1 & 0 & 2 \\ 1 & 5 & 2 & 2 \\ 0 & 3 & 6 & 1 \end{bmatrix} \qquad f = \begin{bmatrix} 1 & 0 \\ -1 & 0 \end{bmatrix} \](a) Compute the activation map \(f * g\). (b) Apply 2×2 max pooling to the original input \(g\).
Solution and marking
(a) Output size 4 − 2 + 1 = 3. With this filter each output is (top-left) − (bottom-left) of the patch, i.e. each entry minus the one below it.
\[ f * g = \begin{bmatrix} 2-4 & 0-1 & 1-0 \\ 4-1 & 1-5 & 0-2 \\ 1-0 & 5-3 & 2-6 \end{bmatrix} = \begin{bmatrix} -2 & -1 & 1 \\ 3 & -4 & -2 \\ 1 & 2 & -4 \end{bmatrix} \](b) Blocks {2, 0, 4, 1}, {1, 3, 0, 2}, {1, 5, 0, 3}, {2, 2, 6, 1}:
\[ \text{max pool} = \begin{bmatrix} 4 & 3 \\ 5 & 6 \end{bmatrix} \]- 3×3 activation map correct2
- 2×2 max-pooled output correct2
Grade it
Tick the marking boxes inside each Part B solution for the steps you got right, then grade.