Well they would wouldn't they?*
The big get together at Stanford's Memorial Auditorium always sells out and this year we'll be visiting a few media outlets for their on-the-ground takes on what's being said.
The article below is based on Micron's Sunday tutorial, here's the rest of the schedule of events:
https://hotchips.org/
From WccfTech, August 23:
Micron Says AI’s Memory Wall is Worsening as Compute outruns HBM Bandwidth by 3x every Two Years, Advanced Packaging & Process To Tackle Rising Thermal Constraints
Micron explores what High Bandwidth Memory (HBM) is, why it is essential, how it compares to traditional DDR, and the key challenges that demand further innovation to sustain AI’s growing demands.
Micron Reveals HBM Failures Caused 17% of Meta's Llama 3 Training Interruptions as the Memory Wall Deepens
As AI LLMs continue to evolve, the need for advanced memory technologies has become vital for the continuous scaling and improvement of these models. But memory is now becoming a major bottleneck that is limiting the performance of AI systems.
HBM is currently the leading DRAM type used in leading AI accelerators. To attain maximum operations on a typical AI system, a large portion of the work is bound by memory as compute processors wait for more data to arrive. Once the work is complete, the system enters a compute-bound phase in which data is delivered to it fast enough to keep the processors busy. With HBM, the memory bandwidth ceiling is elevated, allowing memory-intensive AI workloads to achieve higher performance before hitting memory limits.
Micron's HBM Design Architecture Fellow, Raghu Sreeramaneni, shed light on this "Memory Well" during its Hot Chips 2026 presentation. The company states that while compute capabilities are advancing at a rate of 3x every two years, memory is only advancing at <2x every two years. Currently, there are three types of memory architectures: 2.5D attached memory, 2.5D advanced memory, and Processing-in-memory DRAM....
*****
.....MUCH MORE...The key takeaways of the Hot Chips 2026 talk include:
- Memory is central to scaling LLM performance (more compute, larger datasets, more parameters).
- Compute scales ~3× every two years; HBM bandwidth scales only ~2× every two years → the memory wall persists and may be worsening.
- In a typical GPU SIP with four 12-high HBM stacks, memory silicon accounts for ~90% of the total silicon (8× the GPU silicon).
- HBM is the most cross-functionally complex memory solution (packaging + process + design) in a form factor the size of a postage stamp.
- AI workloads are often memory-bound; higher HBM bandwidth shifts the roofline upward and unlocks more of the processor’s peak performance.
- Stack heights have grown from 4-high to 16-high (path to 20-high exists but faces major thermal/mechanical hurdles).
- Meta’s Llama 3 training paper attributed 17% of unintended interruptions to HBM.
*Mandy Rice-Davies was a former model and showgirl known for her role in the Profumo affair.
When informed by the prosecuting attorney that Lord
Astor disputed her version of events and denied having an affair she
responded:
"Well, he would, wouldn't he?"
Although Many died in 2014 we are keepers of her memory
Mandy Rice-Davies Alert: Christina Romer Says Maximum Tax Revenue at 84% Marginal Rate
She would, wouldn't she.*
*For British politicians of a certain age [often referred to as octo or nona-genarians -ed]
the scandal surrounding Secretary of State for War John Profumo's
affair with the alleged mistress of a Russian spy was highlighted by the
testimony of Miss Rice-Davies, a friend of the alleged mistress,
Christine Keeler.
From Wikipedia:
While giving evidence at the trial of Stephen Ward, charged with living off the immoral earnings of Keeler and Rice-Davies, the latter made a famous riposte. When the prosecuting counsel pointed out that Lord Astor denied an affair or having even met her, she replied, "Well, he would, wouldn't he?"We've tried to keep the phrase alive, using it about once per year:
2007
Gore Says Markets are Key in Battle to Combat Climate Change
"Well, he would, wouldn't he?"2007
Mandy Rice Davies*
ICE, Skating on Thin
TESTIMONY OF JEFFREY C. SPRECHER CHAIRMAN AND CHIEF EXECUTIVE OFFICER, INTERCONTINENTALEXCHANGE, INC.2008
Warning: Mandy Rice-Davies moment ahead.
"...we do not believe that a complete overhaul of the current regulatory structure is either warranted or advisable."
UN Can Regulate Emissions Trading Without Conflict of Interest
All together now: A Mandy Rice-Davies Moment!*2009
Major Problems at California's Public Pension Fund, CalPERS And: A Mandy Rice-Davies Moment!
One of these days I'll have to tell the story of how CalPERS got to this point. It is an ugly tale. For now we'll just post the slow motion train wreck.One of my favorite usages:
On a positive note: Mandy Rice-Davies* moment ahead!
Lord McIntosh of Haringey: My Lords, I am proud of many things that this Government have done. I pause to anticipate the interjection—"He would say that, wouldn't he?"...
Lords Hansard text for 6 Feb 2002
And many more over the years.
