Kalman Filter: Simple but Powerful
A friendly, intuitive explanation of Kalman Filters. No heavy measure theory, just state estimates and noise.
When you write loss.backward() in PyTorch, magic happens.
Behind the scenes, PyTorch traverses a complex web of operations, calculates derivatives for millions of variables, and updates weights—all in a fraction of a millisecond.
For the longest time, this felt like black magic to me. I could train models, but I didn't really know what PyTorch was doing under the hood. Then I read Grokking Deep Learning by Andrew Trask and decided to build my own deep learning framework from scratch using nothing but NumPy.
It turns out that building a basic deep learning framework isn't as hard as it sounds. You only need to understand one core concept: Automatic Differentiation (Autograd).
Let's build a mini-PyTorch in about 100 lines of Python.
In PyTorch, everything is a Tensor. A Tensor is more than just a multi-dimensional array of numbers. It is a node in a computation graph.
A Tensor needs to know:
Let's write a basic Tensor class:
import numpy as np
class Tensor:
def __init__(self, data, creators=None, creation_op=None, autograd=True):
self.data = np.array(data)
self.creators = creators # Tensors that generated this tensor
self.creation_op = creation_op # The operation (e.g., "add", "neg")
self.grad = None # Gradient accumulator
self.autograd = autograd # Flag to track graph
self.children = {} # Track downstream child nodes
# Keep track of how many children have finished backpropagation
if creators is not None:
for c in creators:
if self.id not in c.children:
c.children[self.id] = 1
else:
c.children[self.id] += 1
@property
def id(self):
return id(self)
When we perform operations on Tensors, we need to connect them. For example, if we add two tensors x and y to get z, then z's creators are [x, y] and its creation operation is "add".
Let's implement the addition operation:
def __add__(self, other):
if self.autograd and other.autograd:
return Tensor(self.data + other.data,
creators=[self, other],
creation_op="add")
return Tensor(self.data + other.data)
Now, if we run z = x + y, we have automatically built a directed acyclic graph (DAG) where z points back to x and y.
To train a model, we need to calculate the gradient of our loss function with respect to every weight. This is where backpropagation comes in.
If we have a node z = x + y, how do gradients propagate backward?
According to calculus:
Let's implement a backward method that traverses the graph in reverse order, calculates gradients, and accumulates them:
def backward(self, grad=None):
if not self.autograd:
return
if grad is None:
if self.grad is None:
# Typically loss is a scalar, so we start with a gradient of 1
self.grad = Tensor(np.ones_like(self.data))
grad = self.grad
else:
if self.grad is None:
self.grad = grad
else:
self.grad.data += grad.data # Accumulate gradients from multiple paths
# Go backward only when all child gradients have been received
if self.creators is not None and self._all_children_grads_received():
if self.creation_op == "add":
# For addition, gradient passes straight through to both parents
self.creators[0].backward(self.grad)
self.creators[1].backward(self.grad)
Note: The _all_children_grads_received() helper is crucial. If a tensor is used in multiple operations (like a weight shared across layers), we must wait for all children to send their gradients before backpropagating further. Otherwise, we calculate incorrect gradients.
Let's verify that our framework can calculate basic derivatives. We will define:
$z = (x + y) + y$
We want to find the derivative of $z$ with respect to $x$ and $y$. Let's solve this analytically:
Let's see if our Python code gets it right:
x = Tensor([1.0], autograd=True)
y = Tensor([2.0], autograd=True)
# Forward pass
a = x + y
z = a + y
# Backward pass
z.backward()
print("x grad:", x.grad.data) # Expected: [1.0]
print("y grad:", y.grad.data) # Expected: [2.0]
It works! Our framework successfully calculated the gradients using the chain rule across the computation graph.
To build real neural networks, we need matrix multiplication (matmul) and activation functions (like Sigmoid or ReLU).
For matrix multiplication $Y = X \cdot W$:
Let's add the matmul operation to our Tensor class:
def matmul(self, other):
if self.autograd and other.autograd:
return Tensor(self.data @ other.data,
creators=[self, other],
creation_op="matmul")
return Tensor(self.data @ other.data)
And update our backward() function:
if self.creation_op == "matmul":
# Gradient of MatMul
act = self.creators[0] # Activation tensor
weights = self.creators[1] # Weight tensor
# Backprop to input activation
act.backward(Tensor(self.grad.data @ weights.data.T))
# Backprop to weight matrix
weights.backward(Tensor(act.data.T @ self.grad.data))
With just this simple Tensor class, you can build:
Linear layer that initializes a weight matrix and bias vector.Mean Squared Error or Cross Entropy.SGD class that updates weights: $W = W - \alpha \times W.\text{grad}$.By stripping away the complex C++ optimizations of PyTorch, we can see the mathematical machinery that drives modern deep learning. It's not magic—it's just a graph of operations, a list of derivatives, and the chain rule executed recursively.
If you want to understand deep learning deeply, stop importing PyTorch for a weekend. Grab NumPy, open a blank editor, and build it yourself.
I'd love to hear your thoughts or questions. Connect with me on LinkedIn or send me an email!