"NVIDIA Chief Scientist Bill Dally on How GPUs Ignited AI, and Where His Team’s Headed Next" (NVDA)

With the publication of the latest Top500 list (next post), the talk of the High Performance Computing world is the convergence of AI and HPC, both of which are facilitated by Graphics Processing Units.
So let's go to the source. From the NVIDIA blog:

Bill Dally has been working on neural networks since before they were cool.

Dally, chief scientist at NVIDIA, is an icon in the deep learning world. A prolific researcher with more than 150 patents, he previously chaired Stanford University’s computer science department.

Dally sat down with AI podcast host Noah Kravitz to share his reflections on artificial intelligence — a field he’s been working in for decades, which has had a renaissance thanks to GPU-driven deep learning. AI, he says, is “going to transform almost every aspect of human life.”

Roots of the Current AI Revolution
When Dally first started his neural networks research in the 1980s, “we had computers that were literally 100,000 times slower than what we have today,” he told Kravitz.

Today’s AI revolution is enabled by powerful GPUs. But it took a lot of work to get there, such as the 2006 launch of the CUDA programming language by NVIDIA’s Ian Buck.

“The GPUs had the computational resources, and CUDA unlocked it,” Dally said.
As GPU computing gained traction, Dally met with fellow deep learning luminary Andrew Ng for breakfast. Ng was working on a now well-known project that used unsupervised learning to detect images of cats from the web.

This work took 16,000 CPUs on Google Cloud. Dally suggested they collaborate to use GPUs for this work — and so began NVIDIA’s dive into deep learning.
Dally says there are two main focus areas for neural networks going forward: building more powerful algorithms that ramp up the efficiency of doing inference, and developing neural networks that train on much less data.

Technological advancements have an “evolutionary component and a revolutionary component,” he said. “In research, we try to focus on the revolutionary part.”

Strengthening Research Culture at NVIDIA
When Dally joined NVIDIA as chief scientist in 2009, the research team had less than a dozen scientists. Today, it’s 200 strong.

Dally’s goal is for NVIDIA researchers to do excellent work in areas that will have a major impact to the company in the future. He says publishing strong research in top-tier venues is essential because it provides peer review feedback that is key for quality control.

“It’s a humbling experience,” he said. “It makes you better.”
This week, NVIDIA researchers are presenting 14 accepted papers and posters, seven of them during oral sessions, at the annual Computer Vision and Pattern Recognition conference in Salt Lake City....MORE ( the podcast)

As an example of the convergence, also from the NVIDIA blog:

That Was Fast: Summit Already Speeding Research into Addiction, Superconductors

Just weeks after its debut, Summit, the world’s fastest supercomputer, is already blasting through scientific applications crucial to breakthroughs in everything from superconductors to understanding addiction.

Summit, based at the Oak Ridge National Laboratory, in Tennessee, already runs CoMet — which helps identify genetic patterns linked to diseases — 150x faster than its predecessor, Titan. It’s running another application, QMCPACK — which handles quantum Monte Carlo simulations for discovering new materials such as next-generation superconductors — 50x faster than Titan.

The ability to quickly accelerate widely-used scientific applications such as these comes thanks to our more than a decade of investment across what technologists call “the stack.” That is, everything from architecture improvements in our GPU parallel processors to system design, software, algorithms, and optimized applications. While innovating across the entire stack hard, it’s also essential, because, with the end of Moore’s law, there are no automatic performance gains.

Summit, powered by 27,648 NVIDIA GPUs, is the latest GPU-powered supercomputer built to accelerate scientific discovery of all kinds. Built for the U.S. Department of Energy, Summit is the world’s first supercomputer to achieve over a 100 petaflops, accelerating the work of the world’s best scientists in high-energy physics, materials discovery, healthcare and more.

But Summit delivers more than just speed. Instead of one GPU per node with Titan, Summit has six Tensor Core GPUs per node. That gives Summit the flexibility to do traditional simulations along with the GPU-driven deep learning techniques that have upended the computing world since Titan was completed six years ago.

How Volta Stacks the Deck
With Volta, we reinvented the GPU. Its revolutionary Tensor Core architecture enables multi-precision computing. So it can crank through deep learning at 125 teraflops at FP16 precision. Or when greater range or precision are needed, such as for scientific simulations, it can compute at FP64 and FP32....MORE