Neural Networks Fundamentals
Understanding the building blocks of artificial intelligence: how neural networks learn from data and make predictions.
What is a Neural Network?
A neural network is a computational model inspired by the structure of biological neurons in the brain. At its core, it's a system of interconnected nodes (neurons) organized in layers that process information through weighted connections. Neural networks learn by adjusting these weights based on data, enabling them to recognize patterns and make predictions.
Despite the biological inspiration, modern neural networks are fundamentally mathematical functions that map inputs to outputs through a series of transformations. Understanding this mathematical foundation is crucial for engineering effective AI systems.
The Perceptron: The Building Block
The perceptron, introduced by Frank Rosenblatt in 1958, is the simplest form of a neural network. It's a single neuron that takes multiple inputs, applies weights to them, sums them up, and passes the result through an activation function.
Mathematical Formulation
y = f(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)
y = f(∑(wᵢxᵢ) + b)
- x₁, x₂, ..., xₙ: Input features
- w₁, w₂, ..., wₙ: Weights (learnable parameters)
- b: Bias term (learnable parameter)
- f: Activation function
- y: Output
The weights determine the importance of each input, while the bias allows the neuron to shift the activation function. These parameters are learned during training through optimization algorithms.
Activation Functions
Activation functions introduce non-linearity into neural networks, enabling them to learn complex patterns. Without activation functions, a neural network would simply be a linear transformation, no matter how many layers it has.
Sigmoid
σ(x) = 1 / (1 + e⁻ˣ)
Output range: (0, 1). Used for binary classification and probability outputs.
Tanh
tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)
Output range: (-1, 1). Zero-centered, often better than sigmoid.
ReLU
ReLU(x) = max(0, x)
Most popular. Computationally efficient, helps with vanishing gradient problem.
Softmax
softmax(xᵢ) = eˣⁱ / ∑eˣʲ
Used for multi-class classification. Outputs probability distribution.
⚠️ The Vanishing Gradient Problem
Sigmoid and tanh can suffer from vanishing gradients during backpropagation. Their derivatives become very small for extreme input values, causing gradients to approach zero in deep networks. This makes training difficult. ReLU and its variants (Leaky ReLU, ELU, GELU) were developed to address this issue.
Multi-Layer Perceptrons (MLPs)
A Multi-Layer Perceptron is a feedforward neural network with one or more hidden layers between the input and output layers. Each layer consists of multiple neurons, and each neuron in one layer is connected to all neurons in the next layer.
Network Architecture
- Input Layer: Receives the raw features (e.g., pixels, sensor readings, word embeddings)
- Hidden Layers: Learn increasingly abstract representations of the data
- Output Layer: Produces the final prediction (classification, regression, etc.)
The power of deep neural networks comes from composing simple transformations across many layers. Each layer learns to extract features at different levels of abstraction:
- Early layers learn low-level features (edges, textures, simple patterns)
- Middle layers combine low-level features into mid-level concepts (shapes, object parts)
- Later layers learn high-level semantic representations (complete objects, concepts)
Forward Propagation
Forward propagation is the process of computing the output of a neural network given an input. Data flows forward through the network, layer by layer, until reaching the output.
Algorithm Steps
- Initialize: Start with input data x and current layer parameters (weights W, biases b)
- Compute pre-activation: z = Wx + b (linear transformation)
- Apply activation: a = f(z) where f is the activation function
- Repeat: Use output a as input to the next layer
- Final output: The activation of the last layer is the network's prediction
In matrix notation for a batch of inputs, forward propagation for one layer is: A = f(WX + b), where X is a matrix of inputs, W is the weight matrix, b is the bias vector, and f is applied element-wise.
Loss Functions
Loss functions (or cost functions) measure how well the network's predictions match the actual targets. The goal of training is to minimize this loss by adjusting the network's parameters.
Mean Squared Error (MSE)
MSE = (1/n) ∑(yᵢ - ŷᵢ)²
Used for regression tasks. Penalizes large errors more heavily.
Binary Cross-Entropy
BCE = -∑(y log(ŷ) + (1-y)log(1-ŷ))
Used for binary classification. Measures probability distribution difference.
Categorical Cross-Entropy
CCE = -∑∑ yᵢⱼ log(ŷᵢⱼ)
Used for multi-class classification with one-hot encoded labels.
Mean Absolute Error (MAE)
MAE = (1/n) ∑|yᵢ - ŷᵢ|
Used for regression. More robust to outliers than MSE.
Key Takeaways
- →Neural networks are universal function approximators that learn from data by adjusting weights
- →Activation functions introduce non-linearity, enabling networks to learn complex patterns
- →Forward propagation computes predictions by passing inputs through layers of transformations
- →Loss functions measure prediction error and guide the learning process
- →Deep networks learn hierarchical representations, from low-level to high-level features