deep learning for computer vision, part 5: convolutional networks

A filter is a tiny grid of weights, here 3 × 3. This one has a column of −1 on the left, 0 in the middle and +1 on the right. Laid over a 3 × 3 patch of the image, it multiplies each pixel by its weight and adds them up.

We use the photo’s brightness, minus its average brightness, so each pixel is a number around 0. Where the patch is the same all over, the −1s and +1s cancel and the sum is about 0. Where it gets brighter from left to right, the +1s win and the sum is positive; where it gets darker, negative. So this filter finds vertical edges.

The ringed cell on the right is where it responds most strongly in this photo: 1.74, beside the police car’s front wheel, where the dark tyre gives way to the light body. The building’s columns light up too. Use the arrows above to try other test photos, and the menu below to try other filters.

Convolution is that one sum, at every position. Keep scrolling to slide the window across the photo, one row at a time. Each position gives one number of the output.

The output is a feature map: an image of where the filter found what it looks for. Red where it matched, blue where the opposite pattern was.

The key idea is that every position uses the same 9 weights. A fully connected layer would need 1,024 weights for each output; this needs 9 for all of them. That’s weight sharing. And each output looks only at its own 3 × 3 neighbourhood, its receptive field.

The dashed ring around the photo is zero padding: one border of zeros, so the window can be centred on the edge pixels too, and the output stays 32 × 32. Because we subtracted the average brightness first, a zero is an average grey. With raw brightness from 0 to 1, the padding would be black, and every bright photo would get a strong false edge all the way round.

A layer has many filters, each producing its own map. Here are four hand-made edge filters: vertical, horizontal and the two diagonals. Each map has been passed through ReLU, as in part 4, so only the positive responses remain.

Stacked together, the maps are the layer’s output: a block 32 × 32 × 4, which the next layer reads as an image with 4 channels. A real network’s first layer has dozens of filters, each spanning all 3 colour channels: a 3 × 3 filter on a colour photo has 3 × 3 × 3 = 27 weights.

Each map shows a different part of the picture. The vertical map picks out the building’s columns and the edges of the wheels; the horizontal map, the outline of the roof and the bottom of the body; the two diagonal maps, the two slopes of the roof.

None of this classifies anything yet. A network learns its own filters, and learns what to make of their maps, as we’ll see below.

Two settings change the output’s size. Padding \(P\) adds a border of zeros. Stride \(S\) is how far the window moves each time. For an input \(W\) pixels wide and a filter \(F\) wide, the output is

pixels wide. Try the menus below:

• padding 1, stride 1: \((32 - 3 + 2)/1 + 1 = 32\), the same size as the input;
• padding 0, stride 1: \((32 - 3)/1 + 1 = 30\), so the map shrinks by 2 every layer;
• padding 1, stride 2: \((32 - 3 + 2)/2 + 1 = 16.5\), which doesn’t divide evenly. Libraries round down to 16, quietly ignoring the last row and column of the padding.

Stride 2 halves the size: a cheap way to shrink the maps, which some networks use instead of pooling.

Max pooling shrinks a map by keeping only the largest value in each 2 × 2 block. Keep scrolling to watch it, one row of blocks at a time.

The 32 × 32 map becomes 16 × 16: a quarter of the numbers, with the strong responses kept. Pooling has no weights at all.

Shrinking matters because each later layer’s 3 × 3 filter then covers a bigger part of the original photo. Stack enough conv and pool layers and the last ones see the whole picture.

Move the photo right with the slider, and look at the feature map: it moves right by the same amount. The ringed response keeps its exact value, just a few columns over. Because the filter is the same everywhere, a shifted input gives a shifted output. That’s translation equivariance.

Except at the edges. The black strip the move leaves behind is itself a strong edge, and the map lights up along it.

That matters more than you might hope. Move all 100 test photos 4 pixels right, and the network below drops from 65% to 52%; part 2’s linear classifier drops from 67% to 58%. A convnet reuses what it learned at every position, but nothing makes the whole network ignore a black strip it never saw in training. Real networks learn that from data, as “Going deeper” explains.

We don’t hand-make filters in practice: they’re learned. The network here is small, but it’s a real convnet:

1. 8 filters of 3 × 3 × 3 on the colour photo, with padding 1, then ReLU;
2. a 2 × 2 max pool, giving 16 × 16 × 8;
3. 16 filters of 3 × 3 × 8, then ReLU;
4. the largest value of each of those 16 maps (a global max pool), giving 16 numbers;
5. a linear classifier from those 16 numbers to 4 class scores.

It’s trained exactly like part 4’s network: softmax loss, backpropagation, Adam. During the backward pass, each max routes its gradient to the one position that won (part 4’s max gate), and from there into the filter weights below it.

Like part 2’s classifier, it was trained before you opened the page: 60 passes over the 320 training photos, each photo flipped left to right half the time, which is a free way to get more data (a car facing left is still a car). Its settings were picked on the validation photos, where it gets 74%.

The top row shows its first layer’s filters as tiny colour pictures, with their maps on the current photo. Some respond to edges, some to colours. On the test photos it gets 65%: 23 of 25 cars and 20 frogs, but only 8 planes.

Compare it to part 2’s linear classifier, top. That model needs a full 32 × 32 colour template per class: 12,292 parameters, and 67% of the test photos right.

The convnet, bottom, has 1,460 parameters, an eighth as many: 224 in the first layer, 1,168 in the second, and 68 in the linear layer. It gets 65%.

On 100 test photos those two numbers are too close to separate: one photo is one percentage point. On all 4,000 CIFAR-10 test photos of these four classes, the same two models get 59.7% (linear) and 67.5% (convnet). And with more training data the gap grows: trained on 1,000 photos per class instead of 80, the same convnet design reaches 76.9% on those 4,000 photos, and a linear classifier 64.8%.

An eighth of the parameters, about the same accuracy here, and better with more data, because the layer’s design matches what photos are like: local patterns that can appear anywhere. That built-in assumption is called an inductive bias.