Test 3. Deep Learning

An artificial neuron computes a weighted sum of inputs plus a bias, then applies an activation:

• Inputs $x_i$: feature values.
• Weights $w_i$: scale each input’s contribution.
• Bias $b$: allows shifting the activation threshold.
• Activation $\varphi$: nonlinearly transforms $z$ into the neuron’s output $y$.

In a forward pass, each layer’s outputs become the next layer’s inputs:

1. Hidden layer:$$>z^{(1)}=W^{(1)}x+b^{(1)},a^{(1)}=\varphi (z^{(1)})$$
2. Output layer:$$>z^{(2)}=W^{(2)}{a}^{(1)}+b^{(2)},\hat{y}=\psi (z^{(2)})$$

• $W^{(l)},b^{(l)}$: weights and biases of layer $l$.
• $\varphi ,\psi$: activation functions (e.g., ReLU in hidden, softmax in output).

• Sigmoid: $\sigma (z)=1/(1+e^{-z})$, range $(0,1)$.
  • Pros: smooth, interpretable as probability.
  • Cons: vanishing gradients for $|z|\gg 0$.
• Tanh: $\tanh (z)=(e^z-e^{-z})/(e^z+e^{-z})$, range $(-1,1)$.
  • Pros: zero‑centered.
  • Cons: still suffers vanishing gradients.
• ReLU: $max(0,z)$, range $[0,\mathrm{\infty })$.
  • Pros: sparse activation, mitigates vanishing gradient.
  • Cons: “dying ReLU” (units stuck at zero).
• Leaky ReLU: $max(\alpha z,z)$ with small $\alpha >0$, range $(-\mathrm{\infty },\mathrm{\infty })$.
  • Pros: alleviates dying ReLU.
  • Cons: adds a small negative slope hyperparameter.
• Choice: ReLU/Leaky ReLU in deep nets for faster training; sigmoid/tanh in output layers for bounded outputs.

• Update rule:$$>w\leftarrow w-\eta \frac{\partial L}{\partial w}$$where $\eta$ is the learning rate.
• Variants:
  • Batch GD: use all $n$ examples to compute gradient (stable but slow).
  • Stochastic GD: update per example (noisy but fast).
  • Mini‑batch GD: update per batch of size $m$ (balance of noise and speed).
• Hyperparameters:
  • Learning rate $\eta$
  • Batch size $m$
  • Momentum, decay schedules

1. Forward pass: compute and cache activations $a^{(l)}$ and pre‑activations $z^{(l)}$.
2. Output error:
   ${\delta }^{(L)}={\mathrm{\nabla }}_{a}L\odot {\varphi }^{\prime }(z^{(L)})$.
3. Backpropagate: for $l=L-1,\dots ,1$,$$>{\delta }^{(l)}=(W^{(l+1)T}{\delta }^{(l+1)})\odot {\varphi }^{\prime }(z^{(l)})$$
4. Gradient:
   $\frac{\partial L}{\partial W^{(l)}}={\delta }^{(l)}{a}^{(l-1)T},\frac{\partial L}{\partial b^{(l)}}={\delta }^{(l)}$.
5. Update weights via gradient descent.

• Vanishing gradients: gradients shrink exponentially as they backpropagate through many layers, slowing or stalling learning in early layers.
• Causes:
  • Activation derivatives $<1$ (e.g. sigmoid, tanh).
  • Poor weight initialization.
• Mitigations:
  • Use ReLU or its variants.
  • He or Xavier initialization.
  • Batch normalization.
  • Residual connections (ResNets).

For each epoch:

1. Shuffle training data.
2. For each batch:
   • Load inputs $X$ and labels $y$.
   • Forward pass: compute $\hat{y}=f(X)$.
   • Compute loss: $L(\hat{y},y)$.
   • Backward pass: compute gradients ${\mathrm{\nabla }}_{\theta }L$.
   • Update parameters $\theta \leftarrow \theta -\eta {\mathrm{\nabla }}_{\theta }L$.
3. Validation: evaluate on hold‑out set.
4. Logging: track metrics, adjust learning rate or early stop.

• Convolutional layer: applies learnable filters $K$ over input feature maps via sliding-window dot products:$$>(I\ast K)(i,j)=\sum _{u,v}I(i+u,j+v)K(u,v)$$
• Receptive field: region of input influencing a given output unit.
• Pooling layer: down‑samples feature maps (e.g. max or average) to reduce spatial size and introduce invariance.
• Spatial exploitation: weight sharing (same filter across locations) and local connectivity capture translational features efficiently.