The Perceptron: one of the simplest ANN architectures (ANN = Artificial Neural Networks)
Figure 10-4. TLU (threshold logic unit): an artificial neuron that computes a weighted sum of its inputs $w^Tx$, plus a bias term b, then applies a step function
Most commond step function is Heaviside step function, sometimes sign function is used.
$$ \text { heaviside }(z)= \begin{cases}0 & \text { if } z<0 \\ 1 & \text { if } z \geq 0\end{cases} $$
$$ \operatorname{sgn}(z)=\begin{array}{ll}-1 & \text { if } z<0 \\0 & \text { if } z=0 \\+1 & \text { if } z>0\end{array} $$
How is a perceptron trained? → follows Hebb’s rule. “Cells that fire together, wire together” (the connection weight between two neurons tends to increase when they fire simultaneously.)
perceptrons has limit (eg. cannot solve XOR problem) → use multiplayer perceptron (MLP)
perceptrons do not output a class probability → use logistic regression instead.
When an ANN contains a deep stack of hidden layers → deep neural network (DNN)
Thời xưa, máy tính chưa mạnh → train MLPs is a problem kể cả khi dùng gradient descent.
→ Backpropagation : an algo to minimize the cost function of MLPs.
- Forward propagation: from X to compute the cost J
- Backward propagation: compute derivaties and optimize the params → update params
→ Read this note.