Start here: the whole idea in one minute
You already know the multi-layer perceptron. Each neuron takes every input, multiplies each one by its own weight, adds them up, and applies an activation. A CNN changes exactly one thing:
Instead of one weight per input pixel, a neuron owns a tiny grid of weights (a filter, for example 3×3) and slides it across the image, taking a dot product at every position. Same filter, reused everywhere.
Everything else is what you have already studied: activation functions, a loss, backpropagation with the chain rule, and gradient descent. So there are only three new words to learn tonight:
Convolution
Slide a small filter over the input. At each stop, multiply matching numbers and add them up.
Activation map
The grid of numbers one filter produces. It lights up wherever the filter’s pattern appears.
Pooling
Shrink an activation map by keeping only the max (or average) of each small block. Nothing to learn.
Part 1 · What deep learning is
The notebook opens with deep learning in general before it reaches CNNs. This part is pure recall, which makes it easy multiple-choice material.
Deep learning is a modern evolution of neural networks that emphasises (1) very deep networks, (2) large unstructured datasets such as images, text and audio, and (3) modern compute such as GPUs.
Why deep models are powerful
- Expressivity. A shallow network can represent any function, but it may need a very large hidden layer. A deep network represents complex mappings with fewer parameters.
- Representation learning. Each layer maps its input to a more abstract feature space, and those features are learned from data. Abstraction increases layer by layer.
- Scales with data. Classical algorithms such as linear regression stop improving after a certain dataset size. Deep models keep improving as you add data.
- Scales with hardware. Neural networks are mostly linear algebra, so GPUs give order-of-magnitude speedups.
- Neuroscience link. Deep networks compute in a hierarchy similar to the visual cortex.
Challenges and their remedies
| Challenge | Remedy named in the notes |
|---|---|
| Vanishing gradients in very deep networks | Normalization (BatchNorm and variants), skip connections (ResNet) |
| Prone to overfitting | Regularization, data augmentation, close monitoring of training |
| Needs a massive amount of data | Transfer learning from pre-trained models |
Pros: state of the art in many areas, combine with many algorithms (supervised and unsupervised), lots of specialised software and hardware. Cons: slow and hard to train, and they need a lot of data. Successes mentioned: speech recognition, machine translation, image generation (fake faces), and GPT-style text generation.
Why not just feed images to an MLP?
You could flatten an image and use fully connected layers. The MLP slides did exactly that for CIFAR-10: a 32×32×3 image becomes 3072 numbers. Three problems show up:
- Too many weights. 3072 inputs into 128 hidden neurons is already 393,216 weights. A 224×224×3 photo has 150,528 inputs, so 1000 neurons would need about 150 million weights in the first layer alone.
- Flattening destroys structure. Neighbouring pixels belong together, but a flattened vector treats pixel 5 and pixel 2005 the same way.
- No reuse. An edge in the top-left and the same edge in the bottom-right would have to be learned twice, with separate weights.
Convolution fixes all three with two ideas that appear in the multi-layer handout’s description of CNNs:
Local receptive fields
Each output looks at a small patch (say 3×3), not the whole image.
Shared weights
The same small filter is used at every position, so a feature is learned once and found anywhere.
“Convolutions work best on inputs containing attributes that are interesting, but whose location in the input is not important.” Images are the natural example. Conv layers can learn similar functions to fully connected layers with far fewer parameters.
Part 2a · Convolution in one dimension
Start in 1-D, as the notebook does. You have a short filter \(f\) with \(n\) values and a longer signal \(g\) with \(m\) values.
\[ (f * g)[p] \;=\; \sum_{t=1}^{n} f[t]\; g[p+t] \]In words: put the filter on top of the signal starting at position \(p\), multiply each filter value by the signal value under it, and add the products. That is a local dot product. Then shift the filter one step and repeat.
- Output length \(= m - n + 1\). The filter fits in that many positions. (7 − 3 + 1 = 5.)
- Meaning: \((f*g)[p]\) is large when the signal near \(p\) looks like the filter. It measures how much \(f\) is “found” in \(g\).
- Notes example result: filter [1, 0, −1] on [0, 1, 2, −1, 1, 3, 0] gives
[−2, 2, 1, −4, 1], using \(1\cdot g_p + 0\cdot g_{p+1} - 1\cdot g_{p+2}\).
What a filter can do
| Filter | Effect | Why |
|---|---|---|
[−1, 1] | Edge detection | Output is 0 where the signal is flat and ±1 where it jumps. On audio it would find the boundaries of silence. |
| Small bell curve that sums to 1 | Smoothing | Each output is a weighted average of neighbours, so the kinks are smoothed out. |
Switch the preset in the demo above to see both. So a convolution either changes the behaviour of a signal (smoothing) or extracts a characteristic from it (edges). The machine-learning move is to stop choosing the filter by hand and learn the filter values.
What we just did is technically called cross-correlation. True mathematical convolution uses \(g[p-t]\), which flips the filter. Deep learning uses the un-flipped version and still calls it “convolution”.
This matters for one detail in the notebook: the code cell conv_naive([1,0,-1], [0,1,2,-1,1,3,0]) prints [2, −2, −1, 4, −1], the opposite sign of the formula’s answer, because that code flips the filter. On the exam, slide and dot-product without flipping (the formula and the picture in the notes) unless the question explicitly says to flip, and write one line saying which definition you used.
Part 2b · Convolution in two dimensions
For images the signal \(g\) is a grid and the filter \(f\) is a small grid. Slide the filter left to right, top to bottom. At each stop multiply the nine overlapping pairs and add them. Each stop gives one number of the output.
An \(N \times N\) input with an \(F \times F\) filter (no padding, moving one step at a time) gives an output of size \((N - F + 1) \times (N - F + 1)\). So 5×5 with a 3×3 filter gives 3×3. This is the animated picture in the notes.
In a neural network the two roles are:
- \(g\) = the input from the previous layer (the image, or earlier activation maps).
- \(f\) = the learnable weights. A filter is a “feature” of the image: an edge, or later on something more complex.
The notes’ example: Sobel edge filters
| Filter | Values | High activation at |
|---|---|---|
| Sobel X | [[1, 0, −1], [2, 0, −2], [1, 0, −1]] | Vertical edges (brightness changes left ↔ right) |
| Sobel Y | [[1, 2, 1], [0, 0, 0], [−1, −2, −1]] | Horizontal edges (brightness changes top ↔ bottom) |
Check it in the demo: the vertical-edge image gives a strong response with Sobel X and all zeros with Sobel Y, and the horizontal-edge image does the reverse. The notebook also combines them as the gradient magnitude \(\sqrt{g_x^2 + g_y^2}\), which lights up on edges of any direction.
How to remember which is which: Sobel X has a left column and a right column with opposite signs, so it compares left against right. A left-versus-right difference is what a vertical edge is.
The convolutional layer
Compare the two definitions side by side. They have the same shape on purpose.
| Fully connected layer | Convolutional layer | |
|---|---|---|
| Formula | \(f(x) = \sigma(Wx)\) | \(f(x) = \text{conv}(W, x)\) |
| What runs in parallel | \(p\) neurons | \(p\) convolutions (filters) |
| Each \(w_k\) is | the weight vector of neuron \(k\) | the weights of filter \(k\) |
| Each one outputs | one number | one activation map |
- Each filter produces its own activation map.
- A different filter produces a different activation map.
- All the maps from one layer are stacked together into a multi-channel output.
The three attributes of a conv layer
| Attribute | Meaning |
|---|---|
| # input channels | How many feature maps come in. 1 for grayscale, 3 for RGB, or whatever the previous layer produced. |
| # output channels | How many feature maps go out. This equals the number of filters in the layer. |
| kernel_size | The size of each filter, for example 3×3 or 5×5. |
Multiple input channels
If the input has several channels (an RGB image has 3), each filter is not flat. It has the same depth as the input: a “3×3” filter on RGB is really 3×3×3 = 27 weights. It still slides only across width and height, the dot product now sums over all channels too, and it still produces one activation map.
The written notes only use the plain case (no padding, one step at a time), where the output size is \(N - F + 1\). The calculator also handles padding and stride in case they come up: \(\text{output} = (N + 2P - F)/S + 1\). Counting parameters as \((F \cdot F \cdot C_{in} + 1) \cdot K\) is a common exam extension of “convolution layers have much smaller parameters”.
Pooling
A pooling layer applies a fixed, pre-defined operation over small regions of its input. It usually comes right after a convolution (and its activation).
| Type | For each region, return… | Note |
|---|---|---|
| Max pooling | the largest value | The most common type |
| Average pooling | the mean value |
- It has no learnable weights.
- It is applied to each activation map separately, so the number of channels does not change.
- It reduces the spatial size. A 2×2 pool halves width and height (4×4 → 2×2).
- The LeNet paper called it subsampling.
Why pool? Two purposes
- Slight translation invariance. Inside a pooling window the exact location of a feature is not critical. Nudge the feature by a pixel and the max is the same, which makes detection more robust.
- More abstract representations. Together with convolutions, shrinking the maps lets later layers summarise larger areas of the image.
Part 3 · Putting it together: the CNN
A convolutional neural network is a composition of \(L\) layers, \(f(x) = f_L \circ f_{L-1} \circ \dots \circ f_1(x)\), where the layers contain convolutions. The final layer is often a fully connected output layer.
Typically a CNN is made of consecutive convolution + activation + pooling layers grouped into blocks. The multi-layer handout writes the basic structure as:
Trace the shapes once (LeNet-style)
Being able to follow sizes through a network is the CNN version of the weight-matrix-shape problem you already practised.
| Layer | What it does | Output size | Rule used |
|---|---|---|---|
| Input | grayscale image | 32 × 32 × 1 | |
| Conv | 6 filters, 5×5 | 28 × 28 × 6 | 32 − 5 + 1 = 28; 6 filters → 6 channels |
| Pool | 2×2 | 14 × 14 × 6 | halve width and height; channels unchanged |
| Conv | 16 filters, 5×5 | 10 × 10 × 16 | 14 − 5 + 1 = 10 |
| Pool | 2×2 | 5 × 5 × 16 | halve again |
| Flatten | unroll into a vector | 400 | 5 · 5 · 16 |
| Fully connected | ordinary MLP layers | 120 → 84 → 10 | 10 outputs = 10 digit classes |
Notice the pattern: width and height shrink while the number of channels grows. The early layers hold fine spatial detail; the later layers hold many abstract features.
The algorithm box
| Type | Supervised learning (regression and classification) |
|---|---|
| Model family | Compositions of convolutional, pooling and other types of layers |
| Objective function | Any differentiable objective |
| Optimizer | Gradient descent |
Training is the same backpropagation you know. The filter values are just weights, so the chain rule gives each one a gradient and gradient descent updates it.
Two famous CNNs
| LeNet | AlexNet | |
|---|---|---|
| Who / when | LeCun, Bottou, Bengio, Haffner, 1998 | Krizhevsky, Sutskever, Hinton, 2012 |
| Task | Handwritten digit recognition (MNIST) | ImageNet image classification |
| Output | 10 classes (digits 0–9) | 1000 classes |
| Structure | Several conv and max-pooling (“subsampling”) layers, then a fully connected MLP | Multiple conv layers followed by max pooling; originally split across two GPUs because of hardware limits |
| Why it matters | Inspired the CNN architectures that followed | Started a new era on ImageNet; deeper CNNs later exceeded human performance |
CNNs and feature learning
Before neural networks, computer vision relied on hand-crafted features. Two names to recognise: SIFT (Scale-Invariant Feature Transform) and HOG (Histogram of Oriented Gradients). Designing such features is hard, and they may not be the best ones for a particular task. This is the “traditional computer vision pipeline” from the MLP slides: extract features by hand, then train a linear classifier.
A CNN removes the hand-crafting. It extracts features at different levels of abstraction on its own, learned end-to-end with the classifier.
What each depth learns (Zeiler and Fergus, 2013)
| Layers | Responds to | Level |
|---|---|---|
| Layer 1 | Coloured edges and blobs | Low |
| Layers 2–3 | Object parts, such as the wheels of a car or the beak of a bird | Mid |
| Layers 4–5 | Larger object parts and even entire objects | High |
Pros and cons of CNNs
- Pro: they encode visual information efficiently and are used as universal visual feature extractors. It is common to start from an ImageNet-pretrained CNN.
- Pro: they are building blocks in larger pipelines and can be combined with other networks (for example a recurrent network) for complex tasks.
- Con: their main drawbacks are computational and data requirements.
By-hand problems
Do these on paper before opening the solutions. If the exam has a CNN calculation, it will look like one of these.
Filter \(f = [1, 2, 1]\), signal \(g = [1, 3, 0, 2, 4, 1]\). Compute \(f * g\) (slide and dot-product, no flipping). How long is the output?
Solution
Length: 6 − 3 + 1 = 4.
- p = 0: 1·1 + 2·3 + 1·0 = 7
- p = 1: 1·3 + 2·0 + 1·2 = 5
- p = 2: 1·0 + 2·2 + 1·4 = 8
- p = 3: 1·2 + 2·4 + 1·1 = 11
Answer: [7, 5, 8, 11]. This filter is symmetric, so flipping it would change nothing.
Slide the 2×2 filter over the 4×4 image (no padding, one step at a time). Give the activation map.
\[ g = \begin{bmatrix} 1 & 2 & 0 & 1 \\ 0 & 1 & 3 & 2 \\ 2 & 1 & 0 & 1 \\ 1 & 0 & 2 & 3 \end{bmatrix} \qquad f = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \]Solution
Size: 4 − 2 + 1 = 3, so the output is 3×3. With this filter each output is (top-left) − (bottom-right) of the 2×2 patch.
- Row 0: 1 − 1 = 0, 2 − 3 = −1, 0 − 2 = −2
- Row 1: 0 − 1 = −1, 1 − 0 = 1, 3 − 1 = 2
- Row 2: 2 − 0 = 2, 1 − 2 = −1, 0 − 3 = −3
Apply 2×2 max pooling, then 2×2 average pooling, to
\[ \begin{bmatrix} 1 & 3 & 2 & 4 \\ 5 & 6 & 1 & 2 \\ 7 & 2 & 9 & 0 \\ 1 & 4 & 3 & 8 \end{bmatrix} \]Solution
Split into four non-overlapping 2×2 blocks: {1, 3, 5, 6}, {2, 4, 1, 2}, {7, 2, 1, 4}, {9, 0, 3, 8}.
\[ \text{max} = \begin{bmatrix} 6 & 4 \\ 7 & 9 \end{bmatrix} \qquad \text{average} = \begin{bmatrix} 3.75 & 2.25 \\ 3.5 & 5 \end{bmatrix} \]This is the default example in the pooling demo above.
A 28×28 grayscale image goes through a conv layer with 8 filters of size 5×5 (no padding, one step at a time), then 2×2 max pooling. (a) What is the output size after the conv layer? (b) After pooling? (c) How many learnable parameters does each layer have, counting one bias per filter?
Solution
- (a) 28 − 5 + 1 = 24, and 8 filters give 8 channels: 24 × 24 × 8.
- (b) Pooling halves width and height and keeps channels: 12 × 12 × 8.
- (c) Conv: (5 · 5 · 1 + 1) · 8 = 208. Pooling: 0.
A 100×100 RGB image is the input. Compare the number of weights in (a) a fully connected layer with 100 neurons and (b) a conv layer with 100 filters of size 3×3.
Solution
- (a) Inputs = 100 · 100 · 3 = 30,000. Weights = 30,000 · 100 = 3,000,000 (plus 100 biases).
- (b) Each filter has 3 · 3 · 3 = 27 weights. 27 · 100 = 2,700 (plus 100 biases).
Roughly a thousand times fewer, because each filter is small and reused at every position.
CNN cheat sheet
- Deep learning = very deep networks + large unstructured data + GPUs.
- Deep beats shallow because it represents complex functions with fewer parameters, and layers learn increasingly abstract features.
- Vanishing gradients → BatchNorm, skip connections (ResNet). Overfitting → regularization, data augmentation. Little data → transfer learning.
- Convolution = slide a filter, take a local dot product. Large output = the filter’s pattern is present there.
- Deep learning’s “convolution” is really cross-correlation (no flip).
- Output size (plain case) = N − F + 1. 1-D: m − n + 1.
- [−1, 1] detects edges. A bell-shaped filter smooths.
- Sobel X → vertical edges. Sobel Y → horizontal edges.
- Filter = learnable weights. Input g = output of the previous layer.
- One filter → one activation map. Maps are stacked into channels.
- Conv layer attributes: # input channels, # output channels (= # filters), kernel size.
- Use convolutions when a feature matters but its location does not. Far fewer parameters than fully connected.
- Pooling: max (most common) or average; no learnable weights; applied to each map separately; shrinks spatial size.
- Pooling gives slight translation invariance and more abstract representations.
- CNN = blocks of conv → activation → pool, then fully connected, then output.
- CNN: supervised, any differentiable objective, trained by gradient descent.
- LeNet (1998, LeCun): digits, 10 classes. AlexNet (2012): ImageNet, 1000 classes, two GPUs.
- Hand-crafted features before CNNs: SIFT, HOG.
- Layer 1: edges and blobs. Layers 2–3: object parts. Layers 4–5: whole objects.
- Main drawbacks: computation and data.
CNN quiz
27 questions covering every heading of the CNN notebook. Aim for 24 or more, then use “Retry missed”.