Backpropagation in Neural Networks: Calculus and the Chain Rule Explained
In a session at the Data Science Innovators Conference 2023, Ganapathi Pulipaka taught backpropagation using calculus, a small neural network and a Python demo. He also previewed singular value decomposition and explained how it relates to neural networks.
By The AI Enterprise Magazine Team3 min read
Key takeaways
- Math is the foundation. Calculus and linear algebra underpin neural networks across industries.
- Forward then backward. A forward pass computes outputs and error; backpropagation updates weights to reduce it.
- Chain rule. dx/dz equals dx/dy times dy/dz, letting error trace back to each weight.
- Frameworks save effort. TensorFlow and PyTorch offer built-in functions and GPU support.
- SVD optimizes, not replaces. It can help speed up a network but cannot create one.
In a session at the Data Science Innovators Conference 2023, Ganapathi Pulipaka, introduced as a chief data scientist and SAP technical lead, taught backpropagation in neural networks using calculus and linear algebra. The class combined hand-drawn network diagrams, the chain rule and a short Python demonstration, and ended with a preview of singular value decomposition, which Pulipaka calls a "billion dollar algorithm." The talk was aimed at people building a path into machine learning and deep learning.
Why do calculus and linear algebra matter for machine learning?
Pulipaka said everything from neural networks to singular value decomposition rests on applied mathematics. Whether a learner wants to become an ML engineer, a deep learning engineer or a data architect, he said, the fundamentals of calculus and linear algebra provide the foundation, and they apply in retail, life sciences, healthcare, utilities and energy alike.
He described how deep neural networks, which he dated to 2012 in his talk, allowed practitioners to build networks of any size. He said high performance computing labs use them to cut processing from years to minutes.
Why use TensorFlow or PyTorch instead of plain Python?
Pulipaka said plain Python requires a lot of manual coding of mathematical computations, while TensorFlow and PyTorch provide built-in functions and can run on GPUs. Using a GPU from plain Python, he said, needs manual wrapping operations. For the class he still chose plain Python so students could see the mechanics.
How does backpropagation work?
The example network had inputs A1 and A2, hidden neurons Y1 and Y2, and outputs Z1 and Z2, with a weight on every connection and bias nodes added. Pulipaka said the purpose of backpropagation is to optimize those weights.
First comes a forward pass. Each hidden neuron's net input is its weights multiplied by the inputs plus bias, and the logistic activation function turns that into an output. He named alternatives including hyperbolic tangent, ReLU, leaky ReLU and swish. The total error is half the squared difference between target and actual output, and training aims to minimize it.
What is the chain rule?
Pulipaka described the chain rule as differentiating a function of a function. "It is that simple guys, it's not super complicated," he said, writing dx/dz as dx/dy multiplied by dy/dz. In backpropagation, he explained, the same rule lets the error be traced back through the network to each weight through partial derivatives.
What did the code demonstration show?
Pulipaka built "a toy data set" of two clusters, initialized weights, ran forward propagation as vectorized equations and applied backpropagation. He looped through 10,000 episodes with a learning rate he set to one. A visualization before and after the weight updates showed the error shrinking.
He said his book gives step-by-step explanations in a cookbook style, and that his articles on Medium include code for each activation function.
Can singular value decomposition replace a neural network?
Pulipaka said SVD is a linear algebra technique used in recommendation systems at companies such as Amazon and Netflix and for text mining. Asked at conferences whether it can be used inside a neural network, he said it can help optimize or speed up an existing network but cannot create one, because the calculation is not how neural networks function.
He demonstrated SVD in TensorFlow on a small rank-2 tensor, rebuilding the original matrix from its three factors to confirm the result.
Where can you hear more from data and AI practitioners?
AI Connect continues to host free live webinars. The 2027 schedule includes 24 webinars in four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Details are at /webinars/, and registration for 2027 events is coming soon.
Frequently asked questions
What is backpropagation?
Pulipaka described it as the process of optimizing a network's weights after a forward pass by tracing the error backward.
What is the chain rule in neural networks?
It differentiates a function of a function, written dx/dz = dx/dy times dy/dz, and is used to compute how the error changes with each weight.
Which activation functions did he mention?
Logistic (sigmoid) in the demo, plus hyperbolic tangent, ReLU, leaky ReLU and swish.
Can SVD be used in a neural network?
He said it can help optimize or speed up an existing network but cannot build one.
This article is based on remarks by Ganapathi Pulipaka, Chief Data Scientist and SAP Technical Lead at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: AI Use Cases & Applications, People, Culture & Leadership
