From Azeem Azhar's Exponential View, October 16:
A few days ago, Jensen Huang was a guest on the BG2 podcast. I’ve heard him speak many times before, but this conversation struck me as extraordinary. We get an almost unfiltered view of Nvidia, but even more so of the trajectory of AI’s development as a technology and an industry.
On my way to Dubai, I had a chance to listen to the whole thing. This is one of the most interesting conversations I’ve listened to in a while and here I’m going to share my favourite bits1 and takeaways… Enjoy!
Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner
The flywheel’s flywheel
Jensen spoke about how the stack of technologies that comprise AI is accelerating. Nvidia is focused on the rate of change of that acceleration… at its heart, the flywheel of machine learning helps them achieve 2-3x more performance every year.
Many people used to believe that designing a better chip with more FLOPs, more bits and bytes, was the key. You’d see their keynote slides full of charts and specs. And yes, horsepower does matter. But that’s old thinking.
In the past, software was static—just an application running on Windows. The way to improve it was to make faster chips. But machine learning isn’t human programming; it’s not just about the software. It’s about the entire data pipeline. The real key is the machine learning flywheel.
The most important part is enabling data scientists and researchers to be productive in this flywheel. It starts at the very beginning. People often don’t realise that it takes AI to curate data to teach an AI, and that process is incredibly complex. With smarter AI curating the data, we now even have synthetic data generation, adding more ways to prepare data for training. Before you even get to training, you have massive amounts of data processing involved. […]
Every step along the flywheel is challenging. In the past, we focused on making things like Excel or Doom faster. But now, you need to think about how to make the entire flywheel faster. […]
In the end, the exponential rise comes from accelerating the whole system.
Accelerating the entire system requires a holistic approach that considers Amdahl’s law. Amdahl’s law states that the overall system speedup is limited by the fraction of the system that cannot be parallelised or improved. To achieve significant acceleration, you need to optimise every component of the AI pipeline, from data preparation to inference, not just focusing on individual steps like training....
....MORE
We last visited Mr. Azhar at his LinkedIn page on October 22:
"SpaceX vs Boeing – A test of evolutionary fitness"
If interested here's Azeem at the Harvard Business Review last December.
And why Microsoft's chatbot turned into a genocidal racist back in 2016:
"This is an extension of the Boaty McBoatface saga, and runs all the way back to the Hank the Angry Drunken Dwarf write-in during Time magazine's Internet vote for Most Beautiful Person."