What a neural network is: neurons, layers, weights
3Blue1Brown · 19 min · 6 sections
The structure of the simplest neural network, built up from scratch to recognize handwritten digits: what a neuron holds, how layers pass activations on through weights, biases and the sigmoid, and how the whole thing is one function written with matrices. Learning comes in the next lecture.
Mind map
- Neurons
- Hold a number from 0 to 1
- That number is the activation
- Layers
- 784 input pixels
- 2 hidden layers of 16
- 10 outputs, one per digit
- Weights and bias
- Weighted sum of the previous layer
- Bias sets the threshold
- Squishing
- Sigmoid maps to (0, 1)
- ReLU in modern networks
- Matrix form
- Fast in libraries
- Why layers
- Edges, then patterns, then digits
- Same idea in speech
Why recognizing a digit is hard
A 28×28 pixel image of a 3 is easy for the brain to read, even though the exact pixel values differ completely from one 3 to the next.
Writing a program that takes the 784 pixels and outputs the digit goes from trivial to dauntingly difficult. The goal: build a network that learns to recognize handwritten digits.
Summary Brains recognize a sloppy 3 effortlessly, but writing a program that does it is dauntingly difficult.
Neurons and layers
- Neuron
- A thing that holds a number between 0 and 1.
- Activation
- The number inside a neuron. For an input neuron it is the grayscale value of its pixel, 0 for black up to 1 for white.
The network in this lecture:
- Input layer: 784 neurons, one per pixel of the 28×28 image.
- Hidden layers: two layers of 16 neurons each, an arbitrary choice.
- Output layer: 10 neurons, one per digit; the brightest one is the network's answer.
Activations in one layer determine the activations of the next, loosely like groups of biological neurons firing cause others to fire.
Summary Neurons hold activations between 0 and 1, arranged in an input layer of 784, two hidden layers of 16 and an output layer of 10.
Why a layered structure could work
Digits break into parts:
- A 9 is a loop up top and a line on the right.
- An 8 is a loop up top and another loop down low.
- A loop itself breaks into little edges.
The hope: second-layer neurons pick up little edges, third-layer neurons pick up loops and lines, and the output combines them into digits.
The same layering fits other tasks: speech goes from raw audio to sounds, syllables, words and phrases.
Summary The hope is that layers build up abstraction: pixels into edges, edges into patterns like loops and lines, patterns into digits.
Key takeaways
- A neuron is just a number between 0 and 1, its activation.
- Each layer computes from the layer before.
- Weights say which pattern a neuron picks up; the bias says how strong it must be.
- The digit network has about 13,000 parameters, and learning means finding them.
- Modern networks mostly use ReLU instead of the sigmoid.
Glossary
- Activation
- The number between 0 and 1 a neuron holds.
- Bias
- A number added to the weighted sum that shifts when the neuron becomes active.
- Hidden layer
- A layer between the input and the output layer.
- ReLU
- Rectified linear unit, ; easier to train than the sigmoid.
- Sigmoid
- The logistic curve that squishes any number into the range 0 to 1.
- Weight
- The number on a connection that scales the activation it carries.
Mentioned
- Essence of linear algebra, chapter 3 · For a visual understanding of matrix–vector multiplication
- Lisha Li · PhD in the theory of deep learning, Amplify Partners
Questions for review
- Why does the input layer have exactly 784 neurons?
- What would the weights of a neuron that detects a horizontal edge look like?
- What does the bias change about when a neuron becomes active?