Deep Learning Architectures
Exploring specialized neural network architectures designed for specific types of data and tasks.
Why Deep Learning?
Deep learning refers to neural networks with many layers (typically more than 3). The "deep" in deep learning signifies the depth of the network—the number of layers between input and output. Deep architectures can learn hierarchical representations that simple shallow networks cannot.
However, depth alone isn't enough. Specialized architectures have been developed to handle different types of data structures: convolutional networks for spatial data, recurrent networks for sequential data, and more.
Convolutional Neural Networks (CNNs)
CNNs are designed for processing grid-like data, especially images. They use convolution operations that apply learnable filters across the input, detecting local patterns like edges, textures, and shapes.
Key Components
- Convolutional Layers: Apply filters (kernels) that slide over the input, computing dot products. Each filter learns to detect specific features (edges, textures, patterns).
- Pooling Layers: Downsample spatial dimensions, reducing computation and providing translation invariance. Max pooling selects maximum values, average pooling computes averages.
- Feature Maps: Output of convolutional layers. Each filter produces one feature map, highlighting where that feature appears in the input.
- Stride & Padding: Stride controls filter movement step size. Padding adds zeros around borders to control output dimensions.
Why CNNs Work for Images
- Local connectivity: Each neuron connects only to a small region (receptive field)
- Parameter sharing: Same filter used across entire image, drastically reducing parameters
- Translation invariance: Can detect features regardless of position in image
- Hierarchical features: Early layers detect edges, later layers detect complex objects
Famous CNN Architectures
- LeNet-5 (1998): Early CNN for digit recognition
- AlexNet (2012): Won ImageNet, sparked deep learning revolution
- VGGNet (2014): Demonstrated power of depth with small filters
- ResNet (2015): Introduced skip connections, enabling 100+ layer networks
- Inception/GoogLeNet (2014): Multi-scale processing with inception modules
- EfficientNet (2019): Optimized scaling of network dimensions
Recurrent Neural Networks (RNNs)
RNNs are designed for sequential data where order matters: time series, text, speech, video. Unlike feedforward networks, RNNs have loops that allow information to persist, maintaining a "memory" of previous inputs.
How RNNs Work
h_t = f(W_hh * h_(t-1) + W_xh * x_t + b_h)
y_t = W_hy * h_t + b_y
- h_t: Hidden state at time t (the "memory")
- x_t: Input at time t
- y_t: Output at time t
- W_hh, W_xh, W_hy: Weight matrices (shared across all time steps)
The hidden state is updated at each time step based on the current input and previous hidden state, allowing the network to maintain context over time.
⚠️ The Vanishing Gradient Problem (Again)
Basic RNNs struggle with long sequences due to vanishing gradients during backpropagation through time (BPTT). Gradients exponentially decay as they propagate backwards, making it difficult to learn long-term dependencies.
This limitation led to the development of more sophisticated architectures: LSTMs and GRUs.
Long Short-Term Memory (LSTM)
LSTMs, introduced by Hochreiter and Schmidhuber in 1997, address the vanishing gradient problem through a sophisticated gating mechanism that controls information flow.
LSTM Architecture
LSTMs have a cell state (long-term memory) and three gates that control information:
- Forget Gate: Decides what information to discard from cell state: f_t = σ(W_f · [h_(t-1), x_t] + b_f)
- Input Gate: Decides what new information to store: i_t = σ(W_i · [h_(t-1), x_t] + b_i)
- Output Gate: Decides what to output based on cell state: o_t = σ(W_o · [h_(t-1), x_t] + b_o)
The cell state acts as a "highway" that allows gradients to flow backwards without vanishing, enabling the network to learn dependencies spanning hundreds of time steps.
Gated Recurrent Unit (GRU)
GRUs, introduced by Cho et al. in 2014, are a simpler alternative to LSTMs with fewer parameters:
- Reset Gate: Controls how much past information to forget
- Update Gate: Controls how much new information to add
GRUs are computationally more efficient than LSTMs and often perform comparably on many tasks. The choice between LSTM and GRU often comes down to empirical performance on specific problems.
Modern Architectural Innovations
ResNet: Skip Connections
Residual connections allow gradients to flow directly through the network, enabling training of very deep networks (100+ layers).
y = F(x) + x
Batch Normalization
Normalizes layer inputs, stabilizing training, allowing higher learning rates, and reducing sensitivity to initialization.
Dropout Regularization
Randomly drops neurons during training, preventing co-adaptation and improving generalization. Essential for preventing overfitting.
Attention Mechanisms
Allow models to focus on relevant parts of input, revolutionizing NLP and becoming the foundation for transformers.
Choosing the Right Architecture
- Images/Spatial Data: Use CNNs (ResNet, EfficientNet, Vision Transformers)
- Sequential Data (pre-2017): Use LSTMs or GRUs
- Text/Language (modern): Use Transformers (BERT, GPT)
- Tabular Data: Often start with gradient boosting (XGBoost, LightGBM)
- Multi-modal: Combine architectures or use unified models (CLIP, Flamingo)
Key Takeaways
- →CNNs exploit spatial structure through convolutions, pooling, and parameter sharing
- →RNNs process sequential data but struggle with long-term dependencies
- →LSTMs and GRUs solve vanishing gradients through gating mechanisms
- →ResNet's skip connections enable training of very deep networks
- →Architecture choice should match your data structure and task requirements