The Billion Dollar Algorithm: SVD Behind Recommendation Systems
In a AI Connect session for aspiring data scientists, Ganapathi Pulipaka of DeepSingularity LLC explained why singular value decomposition is called a billion dollar algorithm. He traced neural network history and demonstrated SVD on a small matrix in TensorFlow.
By The AI Enterprise Magazine Team3 min read
Key takeaways
- Rules do not scale. Neural networks replaced hand-coded rules that could not cover every runtime situation.
- SVD powers recommendations. Pulipaka said large companies use it for book and content recommendations and text mining.
- Three matrices. SVD factors a data matrix into U, a diagonal matrix of singular values and V transposed.
- Verify by rebuilding. The demo multiplied the factors back together to confirm the original matrix.
- Start small. Beginners should practice on small matrices before large text datasets.
At a AI Connect conference session published in 2024 and aimed at aspiring data scientists, Ganapathi Pulipaka, Chief AI Scientist at DeepSingularity LLC, explained why he calls singular value decomposition (SVD) "a billion dollar algorithm." It also traced the history of neural networks and walked through a small working example in TensorFlow. Pulipaka said the session was meant for people getting into data science education and kept the math as simple as possible.
Why did machine learning move beyond hand-coded rules?
Pulipaka began with the limits of rule-based systems. A system that needed a million hand-coded rules to decide how a vehicle should navigate would struggle when something appeared at runtime that no rule covered, he said.
Neural networks, which mimic the structure of the human brain, addressed that problem by making a decision at each step of the network. He said early networks in the 1960s and 1970s were small and so limited in what they could process. From the early 2000s, deep networks with many layers and millions of neurons were applied across industries including drug discovery, retail, utilities and energy.
How does a neural network work?
Using a simple diagram, Pulipaka described an input layer, hidden layers and an output layer. Each neuron applies an activation function, and he used the logistic function in his example. He noted that the human brain contains about 100 billion neurons working together.
He tied the idea to reinforcement learning, which he said lets self-driving vehicles use lidar distance data to avoid obstacles and could let a spacecraft adapt to an unknown landing site at runtime. Training large networks, he added, uses forward and backward propagation and depends on scalable hardware such as GPUs.
What is the "billion dollar algorithm"?
Pulipaka said SVD earned the nickname because large companies use it in recommendation systems. As he described it, a retailer like Amazon can recommend a second book after a purchase by analyzing words in reviews and common interests, and Netflix can build recommendations from text mining of viewer feedback.
Mathematically, he said, SVD factors a data matrix X into three matrices: an orthogonal matrix U, a diagonal matrix of singular values and the transpose of an orthogonal matrix V. The same approach applies to natural language processing tasks such as computing word embeddings. He told students that the underlying linear algebra is something they may already be using at their level of education.
What does the hands-on example look like?
For the demonstration, Pulipaka created a small rank-2 tensor in TensorFlow with constant values and used its built-in linear algebra SVD function to compute the three matrices. He then multiplied the pieces back together to show that the reconstruction matched the original, which he described as proof that the algorithm worked.
He said the same function can handle much larger matrices, with words as rows and hashtags as columns, limited mainly by hardware. Python developers can use NumPy's equivalent routine, he said, though TensorFlow makes the computation simpler.
What should beginners practice first?
Pulipaka recommended starting with small exercises before moving to large text-mining or language datasets, and pointed attendees to exercises in his recent book. He also showed a short backpropagation walkthrough that uses the calculus chain rule, differentiating a function of a function, to reduce a network's error. Viewers could ask questions through the event's chat and follow up with him on LinkedIn.
Where can you keep learning from data and AI leaders?
AI Connect hosts free live webinars where practitioners explain data and AI topics. The 2027 schedule includes 24 webinars in four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Details are at /webinars/, and registration for 2027 events is coming soon.
Frequently asked questions
Why is SVD called the billion dollar algorithm?
Pulipaka said large companies use singular value decomposition in recommendation systems and text mining.
What does singular value decomposition do?
It factors a data matrix into an orthogonal matrix, a diagonal matrix of singular values and a transposed orthogonal matrix.
How can you compute SVD in code?
Pulipaka showed TensorFlow's built-in linear algebra SVD function and said NumPy offers an equivalent.
What math do beginners need?
He pointed to calculus and linear algebra as prerequisites for machine learning.
This article is based on remarks by Ganapathi Pulipaka, Chief AI Scientist, DeepSingularity LLC at AI Connect K-12 Data Science Conference 2024 on July 25, 2024. More coverage: More AI Connect Events.
Filed under: AI Use Cases & Applications, People, Culture & Leadership
