CHIPS: Google's Tensor Processing Unit (Finally) A Viable Competitor For Nvidia (GOOG; NVDA)

First up, from Bloomberg, November 25:

Alphabet AI Chips, Gemini Model Position It to Rival Nvidia 

A rally in Alphabet Inc. shares is poised to shake up the ranking of the world’s most valuable companies, amid signs the search giant is making headway in efforts to rival Nvidia Corp.’s bestselling AI accelerator.

That’s prompting investors to reassess the technology landscape and the potential changes in stock market leadership, as rave reviews for Alphabet’s new Gemini artificial intelligence model and demand for AI chips pushes its shares higher.

Shares in the Mountain View, California-based firm rallied 2% to hit a $3.9 trillion market value. Its 37% rally since mid-October has added about $1 trillion in value over that time. Gains this year have left the Google parent about $300 billion away from Nvidia’s $4.2 trillion market cap.

Meta Platforms Inc. is in discussions to use Google chips — known as tensor processing units, or TPUs — in data centers in 2027, The Information reported, citing an unidentified person familiar with the talks. Meta also may rent chips from Google’s cloud division next year, the news outlet said.

“Nvidia is expected to have phenomenal growth and now it will be very natural to see a market that is much more wary of competition,” said Alexandra Morris, an investment director at Skagen AS. “The narrative that Nvidia is the only one that will provide chips to this buildup of data centers — that narrative has changed.”

An agreement with Meta would help establish TPUs as an alternative to Nvidia’s chips, the gold standard for big tech firms and startups from Meta to OpenAI that need computing power to develop and run AI models.

Read More: Nvidia Shares Drop on Report of Google Challenge in AI Chips

Nvidia slipped 5.1%, while shares in smaller-rival Advanced Micro Devices Inc., which also makes AI chips, fell 7.8%.

Nvidia’s valuations have pulled back, trading at 26 times forward earnings, well below its average over the past decade of 35 times. Alphabet’s has shot up to trade at 27 times forward earnings compared to an average of 20 times.

While a possible deal with Meta — one of the biggest spenders globally on data centers and AI development — draws attention to the potential for long-term challenges to Nvidia’s dominant market position, much depends on whether the tensor chips can demonstrate the power efficiency and computing muscle necessary to become a viable option in the long run.

This frenzy is causing Alphabet stock’s 14-day relative strength index to about 75, over the level of 70 that signals to some technical analysts that a stock is overbought...

....MORE 

While we've been tracking the GOOG's TPU progress for a decade, this morning Zerohedge found a dandy little write-up on the chips that we'll go with rather than use back-links.

From UncoverAlpha, November 24:

The chip made for the AI inference era – the Google TPU 

Hey everyone,

As I find the topic of Google TPUs extremely important, I am publishing a comprehensive deep dive, not just a technical overview, but also strategic and financial coverage of the Google TPU.

Topics covered:

• The history of the TPU and why it all even started?

• The difference between a TPU and a GPU?

• Performance numbers TPU vs GPU?

• Where are the problems for the wider adoption of TPUs

• Google’s TPU is the biggest competitive advantage of its cloud business for the next 10 years

• How many TPUs does Google produce today, and how big can that get?

• Gemini 3 and the aftermath of Gemini 3 on the whole chip industry

Let’s dive into it.

The history of the TPU and why it all even started?

The story of the Google Tensor Processing Unit (TPU) begins not with a breakthrough in chip manufacturing, but with a realization about math and logistics. Around 2013, Google’s leadership—specifically Jeff Dean, Jonathan Ross (the CEO of Groq), and the Google Brain team—ran a projection that alarmed them. They calculated that if every Android user utilized Google’s new voice search feature for just three minutes a day, the company would need to double its global data center capacity just to handle the compute load.

At the time, Google was relying on standard CPUs and GPUs for these tasks. While powerful, these general-purpose chips were inefficient for the specific heavy lifting required by Deep Learning: massive matrix multiplications. Scaling up with existing hardware would have been a financial and logistical nightmare.

This sparked a new project. Google decided to do something rare for a software company: build its own custom silicon. The goal was to create an ASIC (Application-Specific Integrated Circuit) designed for one job only: running TensorFlow neural networks.

Key Historical Milestones:

• 2013-2014: The project moved really fast as Google both hired a very capable team and, to be honest, had some luck in their first steps. The team went from design concept to deploying silicon in data centers in just 15 months—a very short cycle for hardware engineering.

• 2015: Before the world knew they existed, TPUs were already powering Google’s most popular products. They were silently accelerating Google Maps navigation, Google Photos, and Google Translate.

• 2016: Google officially unveiled the TPU at Google I/O 2016.

This urgency to solve the “data center doubling” problem is why the TPU exists. It wasn’t built to sell to gamers or render video; it was built to save Google from its own AI success. With that in mind, Google has been thinking about the »costly« AI inference problems for over a decade now. This is also one of the main reasons why the TPU is so good today compared to other ASIC projects.

The difference between a TPU and a GPU?

To understand the difference, it helps to look at what each chip was originally built to do. A GPU is a “general-purpose” parallel processor, while a TPU is a “domain-specific” architecture.

The GPUs were designed for graphics. They excel at parallel processing (doing many things at once), which is great for AI. However, because they are designed to handle everything from video game textures to scientific simulations, they carry “architectural baggage.” They spend significant energy and chip area on complex tasks like caching, branch prediction, and managing independent threads.

A TPU, on the other hand, strips away all that baggage. It has no hardware for rasterization or texture mapping. Instead, it uses a unique architecture called a Systolic Array.

The “Systolic Array” is the key differentiator. In a standard CPU or GPU, the chip moves data back and forth between the memory and the computing units for every calculation. This constant shuffling creates a bottleneck (the Von Neumann bottleneck).

In a TPU’s systolic array, data flows through the chip like blood through a heart (hence “systolic”).

1. It loads data (weights) once.

2. It passes inputs through a massive grid of multipliers.

3. The data is passed directly to the next unit in the array without writing back to memory.

What this means, in essence, is that a TPU, because of its systolic array, drastically reduces the number of memory reads and writes required from HBM. As a result, the TPU can spend its cycles computing rather than waiting for data.

Google’s new TPU design, also called Ironwood also addressed some of the key areas where a TPU was lacking:

• They enhanced the SparseCore for efficiently handling large embeddings (good for recommendation systems and LLMs)

• It increased HBM capacity and bandwidth (up to 192 GB per chip). For a better understanding, Nvidia’s Blackwell B200 has 192GB per chip, while Blackwell Ultra, also known as the B300, has 288 GB per chip.

• Improved the Inter-Chip Interconnect (ICI) for linking thousands of chips into massive clusters, also called TPU Pods (needed for AI training as well as some time test compute inference workloads). When it comes to ICI, it is important to note that it is very performant with a Peak Bandwidth of 1.2 TB/s vs Blackwell NVLink 5 at 1.8 TB/s. But Google’s ICI, together with its specialized compiler and software stack, still delivers superior performance on some specific AI tasks.

The key thing to understand is that because the TPU doesn’t need to decode complex instructions or constantly access memory, it can deliver significantly higher Operations Per Joule.

For scale-out, Google uses Optical Circuit Switch (OCS) and its 3D torus network, which compete with Nvidia’s InfiniBand and Spectrum-X Ethernet. The main difference is that OCS is extremely cost-effective and power-efficient as it eliminates electrical switches and O-E-O conversions, but because of this, it is not as flexible as the other two. So again, the Google stack is extremely specialized for the task at hand and doesn’t offer the flexibility that GPUs do.

Performance numbers TPU vs GPU?

As we defined the differences, let’s look at real numbers showing how the TPU performs compared to the GPU. Since Google isn’t revealing these numbers, it is really hard to get details on performance. I studied many articles and alternative data sources, including interviews with industry insiders, and here are some of the key takeaways.

The first important thing is that there is very limited information on Google’s newest TPUv7 (Ironwood), as Google introduced it in April 2025 and is just now starting to become available to external clients (internally, it is said that Google has already been using Ironwood since April, possibly even for Gemini 3.0.). And why is this important if we, for example, compare TPUv7 with an older but still widely used version of TPUv5p based on Semianalysis data:

• TPUv7 produces 4,614 TFLOPS(BF16) vs 459 TFLOPS for TPUv5p

• TPUv7 has 192GB of memory capacity vs TPUv5p 96GB

• TPUv7 memory Bandwidth is 7,370 GB/s vs 2,765 for v5p

We can see that the performance leaps between v5 and v7 are very significant. To put that in context, most of the comments that we will look at are more focused on TPUv6 or TPUv5 than v7.

Based on analyzing a ton of interviews with Former Google employees, customers, and competitors (people from AMD, NVDA & others), the summary of the results is as follows.....

....MUCH MORE 

Nvidia is getting whacked for $10.62 (-5.82%) to $171.93 while GOOG is up another1.99% (+$6.33) 

Related, November 24 - Trouble In Altman's World: "I’m Not Going Back To ChatGPT After Using Google’s Gemini 3: Salesforce CEO Marc Benioff"

Google's TPU Chips Will Deliver Up To 50% Better Perfomance Per Dollar On Some Inference Chores Vs. Nvidia (GOOG; NVDA)

Took ya long enough.*

From SemiAnalysis, September 7:

• TPU Inference Externalization Full Steam Ahead - InferenceX
• InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat 

For more than a decade, the industry has watched Google build an empire on its own silicon. Search, Ads, YouTube, and every generation of Gemini run on TPUs. Few accelerators have attracted as much architectural scrutiny or as much debate about what their performance and economics would look like outside the company that designed them. Anthropic being the biggest user of TPUs, surpassing Deepmind’s own use by 2029.

Google’s internal success was never the question. The question was how much of that advantage the rest of the industry could actually get. Could you take an open-weight model, serve it through a familiar inference engine, and beat NVIDIA on the economics that matter to your business?

Today, we are publishing the first third-party inference results for TPUv7 Ironwood on InferenceX Official Preview. In our apples-to-apples comparisons against B200/B300, Ironwood delivers up to 50% better performance per dollar. Its advantage extends across much of the Pareto curve, and we examine the economics from both sides: Google’s internal total cost of ownership and the external TCO an actual customer pays.

Ironwood (TPUv7) is the first generation in which Google is competing for others’ inference workloads with chips that can be purchased outright or rented through its own cloud. In November 2025, we already said that Anthropic loves TPUs and committed to over one million of them (around 400k+ in direct purchases and 600k+ rented through GCP), used mainly for training but also for inference. Our Accelerator Model has the latest figures for Anthropic’s TPU shipments and Google’s overall TPU shipments by quarter, plus estimates for TPUv8i, v8t, and various v9 / v10, and more

We are excited by how quickly the new TorchTPU stack is developing, the external stack for TPUs. Later in the article, we will discuss the upcoming work needed for TPU software externalization, including optimizing speculative decoding, disaggregated prefill, KV-cache offloading, multi-turn agentic workloads, and more. Even so, we at SemiAnalysis strongly believe that TPU externalization is heading in the right direction and moving full steam ahead. Furthermore, unlike AMD, which is still learning how to build a test-first software culture, Google has decades of software engineering experience and an extremely well established quality-driven culture, so we expect external TPU software to mature rapidly.

In this article, we will cover all the optimizations that went into TPU kernels and the serving stack for open-weight models, including DP attention optimization, MoE routing and kernel optimization, and reducing padding in GDN kernels. We will also take a deep dive into the TPU system and discuss the next steps the amazing TPU performance engineers are pursuing to make the stack widely available.

Google has spent more than a decade demonstrating what it can build with TPUs. Now we get to measure what the rest of the industry can do with them.

Shoutout to the Google (Chris Chan, Jahangir Hasan, Wangyuan Zhang, Anne Stern, Puneith Kaul, Ruizi Dong, Sangam Jindal, Qi Zhou, Madhan Jaganathan, Gang Ji, Jun Wan, Devanshu Jain, Jiaxin Cao, Srinath Mandalapu, Haowen Ning) and Inferact teams and RedHat Teams (Michael Goin) for this amazing TPU foundation and performance! Furthermore, shoutout to the RadixArk team that is also working on TorchTPU SGLang.

InferenceX Official Preview: TPUv7 Ironwood vs. Blackwell and Blackwell Ultra 

We are already seeing strong results from the upcoming native TorchTPU vLLM stack in apples-to-apples comparisons against Nvidia GPUs. Google is using Qwen3.5 397B in FP8 as the initial bring-up model. Later sections take a deep dive into why the new TorchTPU approach is a marked improvement over the previous TorchAX path for external TPU vLLM/SGLang serving.

Once that foundation is in place, Google plans to extend support to other open-weight models, including Kimi K3 and GLM5.3. Once a handful of models are well optimized, we believe adding optimized support for a wide range of popular open models near day 0 becomes far easier. Today, vLLM and SGLang concentrate their day-0 support on Nvidia, with passable day-0 coverage for AMD. We expect TorchTPU to be stable enough in the near future that vLLM and SGLang maintainers may add TPUs to that day-0 list. The stack is expected to leave private beta & be open sourced around October. 

In apples-to-apples comparisons of aggregated serving with FP8 and single-token prediction, we are seeing up to 50% better performance per dollar from TPU than from B200 and B300 running FP8 in aggregated serving. When serving models using FP4 on NVIDIA GPUs, there is quality loss verus FP8. TPUv7 does not have native FP4 computation thus on FP4, NVIDIA GPUs still maintains the lead. This will change with TPUv8i which has native FP4 support thus we strongly believe TPUv8i Boardfly will be competitive to Rubin NVL72.... 

....MUCH MORE
*April 2017 - Watch Out NVIDIA: "Google Details Tensor Chip Powers" (GOOG; NVDA)
We've said NVIDIA probably has a couple year head start but this bears watching, so to speak....

And many, many more, including November 2025 "CHIPS: Google's Tensor Processing Unit (Finally) A Viable Competitor For Nvidia (GOOG; NVDA)" 

Chips: "Google TPU v5e AI Chip Debuts after Controversial Origins"

From HPC Wire, August 30:

The dominance of Nvidia GPUs has companies scrambling to find non-GPU alternatives, and another mainstream option has emerged with Google’s TPU v5e AI chip.

The TPU v5e is also Google’s first AI chip being mainstreamed with a suite of software and tools for large-scale orchestration of AI workloads in virtual environments. The AI chip is now available in preview to Google Cloud customers.

The new AI chip succeeds the previous-generation TPUv4, which was used to train the newer PaLM and PaLM 2 large language models used in Google search, mapping, and online productivity applications.

Google has often compared its TPUs to Nvidia’s GPUs but was cautious with the TPU v5e announcement. Google stressed it was focused on offering a variety of AI chips to its customers, with Nvidia’s H100 GPUs in the A3 supercomputer and TPU v5e for inferencing and training.

The Cloud TPU v5e is also the first Google AI chip available outside the US. The TPUv4 was available only in North America. The TPU v5e computers will be installed in the Netherlands for the EMEA (Europe, Middle East, and Africa) markets and in Singapore for the Asia-Pacific markets.

The origins of TPU v5 were mired in controversy. Researchers at Google informally announced TPU-v5 in June 2021 and, in a paper, said AI was used to design the chip. Google claimed that AI agents helped floor-plan the chip in under six hours faster than human experts. There were internal debates on the claims made in the paper, and Google fired one researcher ahead of the paper’s appearance in Nature magazine.

Academic researchers also called out Google’s claims and criticized the company for not opening it up for public scrutiny. A researcher, Andrew B. Kahng, from the University of California, San Diego, later reverse-engineered Google’s chip-design techniques and found that human chip designers and automated tools were sometimes faster than Google’s AI-only technique....

....MUCH MORE

Ahead of Today's NVIDIA Earnings: A Look at One of the Competitors (GOOG; NVDA)

We'll have much more on Google next week but for today the next-gen Tensor Processing Unit.

From Extreme Tech:

Google Announces 8x Faster TPU 3.0 For AI, Machine Learning

For the past few years, Google has been building its own TPU (Tensor Processing Units) to handle various processing tasks related to artificial intelligence and machine learning. Google first announced the existence of TPUs in 2016, but said it had been using them internally for more than a year. The company’s second-generation TPUs have made the news recently for significantly improved performance, and the third-generation hardware will apparently continue that trend.

According to Google CEO Sundar Pichai, the new TPU 3.0 pods are 8x more powerful than the Google TPU 2.0 pods that we’ve previously covered, and Pichai declared that they’re power-hungry enough to need water cooling — something previous TPUs simply haven’t required. TechCrunch’s image shows a copper-plated cooler system, with water from the same pipe run through all four cooling plates.

So what do we know about TPU 3.0? Not much — but we can make a few educated guesses. According to Google’s own documentation, TPU 1.0 was built on a 28nm process node at TSMC, clocked at 700MHz, and consumed 40W of power. Each TPU PCB connected via PCIe 3.0 x16.

TPU 2.0 made some significant changes. Unlike TPU v1, which could only handle 8-bit integer operations, Google added support for single-precision floats in TPU v2 and added 8GB of HBM memory to each TPU to improve performance. A TPU cluster consists of 180 TFLOPS of total computational power, 64GB of HBM memory, and 2,400GB/s of memory bandwidth in total (the last thrown in purely of the purposes of making PC enthusiasts moan with envy).

Unlike TPU v1, which used 3.5-inch drive bays as its form factor, TPU v2 was welded together in groups of four ASIC chips. Google currently deploys TPUs in clusters of up to 64 boards, at 11.5 PFLOPS per cluster and 4TB of total HBM storage....MORE

As noted in last month's ""UPDATED—NVIDIA Wants to Be the Brains Behind the Surveillance State (NVDA)" there is so much power running through theTSU's and  GPU's that even the most energy-efficient require water-cooling.
Brave new world.

"Google launches Gemini, the AI model it hopes will take down GPT-4" (GOOG; MSFT)

From The Verge, December 6:

Google has been an ‘AI-first company’ for nearly a decade. Now, a year into the AI era brought on by ChatGPT, it’s finally making a big move. 

It’s the beginning of a new era of AI at Google, says CEO Sundar Pichai: the Gemini era. Gemini is Google’s latest large language model, which Pichai first teased at the I/O developer conference in June and is now launching to the public. To hear Pichai and Google DeepMind CEO Demis Hassabis describe it, it’s a huge leap forward in an AI model that will ultimately affect practically all of Google’s products. “One of the powerful things about this moment,” Pichai says, “is you can work on one underlying technology and make it better and it immediately flows across our products.” 

Gemini is more than a single AI model. There’s a lighter version called Gemini Nano that is meant to be run natively and offline on Android devices. There’s a beefier version called Gemini Pro that will soon power lots of Google AI services and is the backbone of Bard starting today. And there’s an even more capable model called Gemini Ultra that is the most powerful LLM Google has yet created and seems to be mostly designed for data centers and enterprise applications. 

Google is launching the model in a few ways right now: Bard is now powered by Gemini Pro, and Pixel 8 Pro users will get a few new features thanks to Gemini Nano. (Gemini Ultra is coming next year.) Developers and enterprise customers will be able to access Gemini Pro through Google Generative AI Studio or Vertex AI in Google Cloud starting on December 13th. Gemini is only available in English for now, with other languages evidently coming soon. But Pichai says the model will eventually be integrated into Google’s search engine, its ad products, the Chrome browser, and more, all over the world. It is the future of Google, and it’s here not a moment too soon.

At first, Gemini comes in three sizes, meant for three different purposes. 

Image: Google 

OpenAI launched ChatGPT a year and a week ago, and the company and product immediately became the biggest things in AI. Now, Google — the company that created much of the foundational technology behind the current AI boom, that has called itself an “AI-first” organization for nearly a decade, and that was clearly and embarrassingly caught off guard by how good ChatGPT was and how fast OpenAI’s tech has taken over the industry — is finally ready to fight back....

....MUCH MORE

 

Most of that "foundational technology" was developed under the direction of Professor Geoffrey Hinton, who we happened to visit on Sunday. 

We have a lot of posts on Google AI, here's one from July 2023 with the intro pointing up the reporting:

AI: Google DeepMind CEO Demis Hassabis On How The Goog Will Get Back In Front Of The Parade (GOOG)
The Verge has fielded a team to report on AI that puts some larger organizations to shame.... 

 

Here are some previous posts that pop up from a search of the blog:

2016  Google, Facebook, and Microsoft Are Remaking Themselves Around Artificial Intelligence (GOOG; FB; MSFT)

2016  Machine Learning: JP Morgan Compares Google's New Chip With NVIDIA's (GOOG; NVDA)

2017  What Google Says It Is Using Artificial Intelligence For: An Interview With the CEO (GOOG)

2017  "Google’s A.I. Has Made Some Pretty Huge Leaps This Week" (GOOG; NVDA)

2018  How Google Plans To Use AI To Reinvent The $3 Trillion US Healthcare Industry

2018   "Google Strategy Teardown: Google Is Turning Itself Into An AI Company As It Seeks To Win New Markets Like Cloud And Transportation" (GOOG)

2018  "How the AI cloud could produce the richest companies ever" (AMZN; MSFT; GOOG)

2023  "Google launched Bard chatbot despite ethics concerns, warnings it was a ‘pathological liar’" (GOOG; EVIL)  

2023  "Analyzing the deepening divide in learning capabilities between a few corporate giants and the rest of the world." (plus advantage flywheels)

2023  Chips: "Google TPU v5e AI Chip Debuts after Controversial Origins"

And dozens, maybe hundreds more. If interested use the 'search blog box', upper left. 

Watch Out NVIDIA: "Google Details Tensor Chip Powers" (GOOG; NVDA)

We've said NVIDIA probably has a couple year head start but this bears watching, so to speak.
From IEEE Spectrum:

In January’s special Top Tech 2017 issue, I wrote about various efforts to produce custom hardware tailored for performing deep-learning calculations. Prime among those is Google’s Tensor Processing Unit, or TPU, which Google has deployed in its data centers since early in 2015.

In that article, I speculated that the TPU was likely designed for performing what are called  “inference” calculations. That is, it’s designed to quickly and efficiently calculate whatever it is that the neural-network it’s running was created to do. But that neural network would also have to be “trained,” meaning that its many parameters would be tuned to carry out the desired task. Training a neural network normally takes a different set of computational skills: In particular, training often requires the use of higher-precision arithmetic than does inference.

Yesterday, Google released a fairly detailed description of the TPU and its performance relative to CPUs and GPUs. I was happy to see that the surmise I had made in January was correct: The TPU is built for doing inference, having hardware that operates on 8-bit integers rather than higher-precision floating-point numbers.

Yesterday afternoon, David Patterson, an emeritus professor of computer science at the University of California, Berkeley and one of the co-authors of the report, presented these findings at a regional seminar of the National Academy of Engineering, held at the Computer History Museum in Menlo Park, Calif. The abstract for his talk summed up the main point nicely. It reads in part: “The TPU is an order of magnitude faster than contemporary CPUs and GPUs and its relative performance per watt is even larger.”

Google’s blog post about the release of the report shows how much of a difference in relative performance there can be, particularly in regard to energy efficiency. For example, compared with a contemporary GPU, the TPU is said to offer 83 times the performance per watt.  That might be something of an exaggeration, because the report itself claims only that there’s a range of between 41 times and 83 times. And that’s for a quantity the authors call incremental performance. The range of improvement for total performance is less: from 14 to 16 times better for the TPU compared with that of a GPU.

The benchmark tests used to reach these conclusions are based on a half dozen of the actual kinds of neural-network programs that people are running at Google data centers. So it’s unlikely that anyone would critique these results on the basis of the tests not reflecting real-world circumstances. But it struck me that a different critique might well be in order.

The problem is this: These researchers are comparing their 8-bit TPU with higher-precision GPUs and CPUs, which are just not well suited to inference calculations. The GPU exemplar Google used in its report is Nvidia’s K80 board, which performs both single-precision (32-bit) and double-precision (64-bit) calculations. While they’re often important for training neural networks, such levels of precision aren’t typically needed for inference.

In my January story, I noted that Nvidia’s newer Pascal family of GPUs can perform “half-precision” (16-bit) operations and speculated that the company may soon produce units fully capable of 8-bit operations, in which case they might be much more efficient when carrying out inference calculations for neural-network programs....MORE

We'll be back with some of the other commentary but in the meantime here are a few (we have hundreds) NVDA posts that may be of interest.
Nvidia's Stock Is Not Yet Out Of The Woods (NVDA)

We focus on the stock, not the company. The company should be fine for at least the next couple years until the artificial intelligence biz catches up to NVIDIA and either takes a different approach or a really different approach and goes quantum computer....  

BMO Capital On Potential Competition For NVIDIA (NVDA)
Artificial Intelligence: What Could Derail NVIDIA? A Lab in Shenzhen; A Basement in Moscow; An Office in Bristol (NVDA)

"Google and Anthropic wave hands about mega TPU deal worth 'tens of billions'"

I had a similar reaction: "What's with the 'tens of billions'" when I saw the news yesterday,  probably because I had just read Carl Sagan swear he never said "billions and billions." 

From The Register, October 23: 

But AWS is still the AI upstart's primary partner 

Google and Anthropic have struck a deal that will see the AI upstart gain access to up to a million of the web giant’s tensor processing units (TPUs) and involve “tens of billions of dollars.”

The two companies announced the deal on Thursday, with Anthropic pitching it as “expanded capacity” that the company will use to meet surging customer demand and allow it to conduct “more thorough testing, alignment research, and responsible deployment at scale.”

Google’s take on the deal is that it will enable Anthropic to “train and serve the next generations of Claude models,” and involves “additional Google Cloud services, which will empower its research and development teams with leading AI-optimized infrastructure for years to come.”

The search and ads giant claims the deal “represents the largest expansion of Anthropic's TPU usage to date” and says the AI upstart “chose TPUs due to their price-performance and efficiency, and the company's existing experience in training and serving its models with TPUs.”

Anthropic’s announcement points out it’s betting on companies other than Google.

“Anthropic’s unique compute strategy focuses on a diversified approach that efficiently uses three chip platforms – Google’s TPUs, Amazon’s Trainium, and Nvidia’s GPUs,” the statement explains. “This multi-platform approach ensures we can continue advancing Claude's capabilities while maintaining strong partnerships across the industry.”

Anthropic also stated: “We remain committed to our partnership with Amazon, our primary training partner and cloud provider, and continue to work with the company on Project Rainier, a massive compute cluster with hundreds of thousands of AI chips across multiple U.S. data centers.”

AI infrastructure announcements nearly always feature enormous numbers and breathless enthusiasm about how the tech will improve everything … but little detail about how players will pay for, and profit from, the billions spent on giant chip farms....

....MUCH MORE, including four 'More Context' links. 

Recently - OpenAI turns to Google's AI chips to power its products, source says (GOOG; NVDA)

The Google product has been a long time acoming. Some previous posts:

June 2016 - Machine Learning: JP Morgan Compares Google's New Chip With NVIDIA's (GOOG; NVDA) 

April 2017 - Watch Out NVIDIA: "Google Details Tensor Chip Powers" (GOOG; NVDA)

May 2018 - Ahead of Today's NVIDIA Earnings: A Look at One of the Competitors (GOOG; NVDA)
We'll have much more on Google next week but for today the next-gen Tensor Processing Unit...

June 2023 - "Elon Musk Predicts Nvidia’s Monopoly in A.I. Chips Won’t Last" (NVDA; TSLA)

September 2023 -  Chips: "Google TPU v5e AI Chip Debuts after Controversial Origins"

July 2024 - Apple Trained Its Large Language Model On Broadcom-Designed TPUs In Google's Cloud, Meh (AVGO, GOOG, AAPL)

And many, many more, use the 'search blog' box, upper left, if interested. 

Google of course has its own AI model entrants, Gemini among others and so, of course, is not giving Mr. Altman their best chips. 

And Professor Sagan? 

I never said it. Honest. Oh, I said there are maybe 100 billion galaxies and 10 billion trillion stars. It’s hard to talk about the Cosmos without using big numbers. I said "billion" many times on the Cosmos television series, which was seen by a great many people. But I never said "billions and billions." For one thing, it’s too imprecise. How many billions DUH "billions and billions"? A few billion? Twenty billion? A hundred billion? "Billions and billions" is pretty vague. When we reconfigured and updated the series, I checked—and sure enough, I never said it.

But Johnny Carson—on whose Tonight Show I'd appeared almost thirty times over the years—said it.
He'd dress up in a corduroy jacket, a turtleneck sweater, and something like a mop for a wig. He had created a rough imitation of me, a kind of Doppelganger, that went around saying "billions and billions" on late-night television. It used to bother me a little to have some simulacrum of my persona wandering off on its own, saying things that friends and colleagues would report to me the next morning. (Despite the disguise, Carson—a serious amateur astronomer—would often make my imitation talk real science.)
Astonishingly, "billions and billions" stuck. People liked the sound of it. Even today, I'm stopped on the street or on an airplane or at a party and asked, a little shyly, if I wouldn't—-just for them—say "billions and billions."

"You know, I didn't actually say it," I tell them.
"It's okay," they reply. "Say it anyway."

I'm told that Sherlock Holmes never said, "Elementary, my dear Watson" (at least in the Arthur Conan Doyle books); Jimmy Cagney never said, "You dirty rat"; and Humphrey Bogart never said, "Play it again, Sam." But they might as well have, because these apocrypha have firmly insinuated themselves into popular culture.

I'm still quoted as uttering this simple-minded phrase in computer magazines ("As Carl Sagan would say, it takes billions and billions of bytes"), newspaper economics primers, discussions of players' salaries in professional sports, and the like.

For a while, out of childish pique, I wouldn't utter or write the phrase, even when asked to. But I've gotten over that. So, for the record, here goes: "Billions and billions."....

—Billions and Billions: Thoughts on Life and Death at the Brink of the Millennium, Carl Sagan, 1997 

"Google and Facebook Are Teaming Up on Artificial Intelligence Tech" (GOOG; FB)

From Fortune, October 3:

Google and Facebook are teaming up to make each company’s artificial intelligence technologies work better together.

The two companies said Tuesday that an unspecified number of engineers are collaborating to make Facebook’s open source machine learning PyTorch framework work with Google’s custom computer chips for machine learning, dubbed Tensor Processing Units, or TPU. The collaboration marks one of the rare instances of the technology rivals working together on joint tech projects.

“Today, we’re pleased to announce that engineers on Google’s TPU team are actively collaborating with core PyTorch developers to connect PyTorch to Cloud TPUs,” Google Cloud director of product management Rajen Sheth wrote in a blog post. “The long-term goal is to enable everyone to enjoy the simplicity and flexibility of PyTorch while benefiting from the performance, scalability, and cost-efficiency of Cloud TPUs.”

Facebook product manager for artificial intelligence Joseph Spisak said in a separate blog post that “Engineers on Google’s Cloud TPU team are in active collaboration with our PyTorch team to enable support for PyTorch 1.0 models on this custom hardware.”

Google first debuted its TPUs in 2016 during its annual developer conference, and pitched them as a more efficient way for companies and researchers to power their machine-learning software projects. The search giant sells access to its TPUs via its cloud computing business instead of selling the chips individually to customers like Nvidia, whose graphics processing units, or GPUs, are popular with researchers working on deep learning projects.

Artificial intelligence technologies like deep learning have grown in popularity over the years with tech giants like Google and Facebook that use the technologies to create software applications that can automatically do tasks like recognize images in photos....MORE

OpenAI turns to Google's AI chips to power its products, source says (GOOG; NVDA)

From Reuters via MSN, June 27:

OpenAI has recently begun renting Google's artificial intelligence chips to power ChatGPT and its other products, a source close to the matter told Reuters on Friday.

The ChatGPT maker is one of the largest purchasers of Nvidia's graphics processing units (GPUs), using the AI chips to train models and also for inference computing, a process in which an AI model uses its trained knowledge to make predictions or decisions based on new information. 

OpenAI planned to add Google Cloud service to meet its growing needs for computing capacity, Reuters had exclusively reported earlier this month, marking a surprising collaboration between two prominent competitors in the AI sector.

For Google, the deal comes as it is expanding external availability of its in-house tensor processing units (TPUs), which were historically reserved for internal use. That helped Google win customers including Big Tech player Apple as well as startups like Anthropic and Safe Superintelligence, two ChatGPT-maker competitors launched by former OpenAI leaders.

The move to rent Google's TPUs signals the first time OpenAI has used non-Nvidia chips meaningfully and shows the Sam Altman-led company's shift away from relying on backer Microsoft's data centers. It could potentially boost TPUs as a cheaper alternative to Nvidia's GPUs, according to the Information, which reported the development earlier....

....MORE 

The Google product has been a long time acoming. Some previous posts:

June 2016 - Machine Learning: JP Morgan Compares Google's New Chip With NVIDIA's (GOOG; NVDA) 

April 2017 - Watch Out NVIDIA: "Google Details Tensor Chip Powers" (GOOG; NVDA)

May 2018 - Ahead of Today's NVIDIA Earnings: A Look at One of the Competitors (GOOG; NVDA)
We'll have much more on Google next week but for today the next-gen Tensor Processing Unit...

June 2023 - "Elon Musk Predicts Nvidia’s Monopoly in A.I. Chips Won’t Last" (NVDA; TSLA)

September 2023 -  Chips: "Google TPU v5e AI Chip Debuts after Controversial Origins"

July 2024 - Apple Trained Its Large Language Model On Broadcom-Designed TPUs In Google's Cloud, Meh (AVGO, GOOG, AAPL)

And many, many more, use the 'search blog' box, upper left, if interested. 

Google of course has its own AI model entrants, Gemini among others and so, of course, is not giving Mr. Altman their best chips. 

A.I.—Google's Vertical Integration Is A Real Advantage (GOOG)

The GOOG is down a bit today, to $308.19 and like many other issues is trading just under the all-time-high from earlier this year.

From Barchart, November 25:

Google is Winning the AI Race by Stealing Exxon's Business Model. Here's How 

Alphabet (GOOG) (GOOGL) has been on a tear since the favorable court decision about monopolistic advantages. The chart below shows the percentage returns in Google stock compared to the other Magnificent 7 names – Apple (AAPL), Amazon (AMZN), Meta (META), Nvidia (NVDA), Microsoft (MSFT), and Tesla (TSLA) – over the past year. 

 

www.barchart.com

There’s more going on here than just a court victory – and I mean besides the obvious point that Google is the only Mag 7 stock that is truly beating the tape.

The answer? Google appears to have won the finish line to artificial intelligence (AI) monetization. Not only that, they’ve won by a wide margin, and stock market capital is flowing out of the other hyperscalers to the victor.

Gemini Helps Blaze a Path to AI Victory 
Google’s Gemini gained 8% user growth in the most recent quarter, with web traffic to the chatbot surging in September after the launch of new AI image generation tools. Rave reviews for Gemini 3, including from Salesforce (CRM) CEO Marc Benioff, have also pushed the stock higher this month.

Google's tensor processing unit (TPU) demand is exploding as Gemini gains wide acceptance. More adept than traditional GPUs at handling complex AI workloads, and with a longer life cycle, Google delivers 50% productivity improvement with TPU than what is offered using NVDA hardware. As a result, developers are switching to Gemini midstream at an increased cost to substitute.

Plus, Google is already generating income across all spectrums, while relative newcomers like OpenAI (parent company of Gemini rival ChatGPT) are still burning through billions of cash.

Is Google the 'Big Oil' of AI? 
In an industry where everyone is renting from each other, Google designs its chip; owns the data centers; runs the software; manages the pricing; manages the margins; and handles distribution.....

....MORE 

"NVIDIA’s Christmas Eve 'Hackquisition' Miracle" (NVDA; GOOG)

From Spyglass, December 25:

NVIDIA gets a creator of the TPU, access to inference tech & IP, and stops anyone else from getting such things. All for a mere $20B...  

Twas the night before Christmas and all through the house, not a creature was stirring, not even a... wait. What's that? Is that Jensen Huang scurrying around doing deals over the holidays?!

It sure is!

Well, presumably he had been working on the deal with the specialized AI chip startup Groq before the holidays started. But I also wouldn't rule out that this deal came together in a hurry given what we know about how he's operated in the past with some mega deals, such as the um, (up to) $100B investment in OpenAI. A deal which came together quickly, perhaps on a trip with a certain President, and perhaps right after the rumors of OpenAI potentially looking at a deal with Google to use their TPUs – we'll come back to that. It was also a deal which was announced three months ago but apparently still has not been finalized, by the way, so there's that.

Anyway, you can't help but wonder if the real key for the timing here may have been to bury it under the spirit of Christmas. Because boy is it a doozy.

CNBC first broke the news with the headline: "Nvidia buying AI chip startup Groq for about $20 billion in its largest acquisition on record". But as it turns out, that wasn't quite right. And actually, it was NVIDIA that wanted to make that clear. They weren't buying Groq, they were merely paying Groq that $20B for a "non-exclusive licensing agreement".

Oh, you've heard that terminology before? Of course you have. It's the magical incantation one must cite in order to summon the spirits of the "hackquisition".

With this deal, NVIDIA is following their Big Tech brethren down this now well-trodden path. Microsoft, Amazon, Google, Meta, and NVIDIA have done deals which aren't technically acquisitions, but effectively are. Because they allow the acquiring – sorry, non-acquiring – company to sort of pick and choose what they want from the acquired – sorry, non-acquired – company in exchange for considerations that can go to... well, sort of anyone they choose. Certainly the investors to get them to sign off on such deals, and often the key employees at those companies. Sometimes the other employees of the companies too, but that's mainly so they don't feel bad and/or raise a stink about the faux-deal. Because when they do that, all hell tends to break loose, at least from a PR-perspective.

Anyway, the first wave of "hackquisitions" were almost more like "hackquihires" because they were simply a way to get access to key talent at those companies and any licensing agreements were seemingly an excuse to make it all a little less blatant. But as such deals have grown in size, they've grown to look even more like actual acquisitions. For example, Meta's nearly-$15B deal with Scale.ai bought them 49% of that company alongside their key talent, namely CEO Alexandr Wang. Why? Presumably Meta wanted more for that amount of money and they thought Scale, a hot company at the time, might still give them some upside. Of course, they were also effectively gutting said company and it seems decidedly less hot now – especially since many of their customers were Meta competitors who no longer wanted to send their data to Scale – shocking, I know. Still, Meta also likely believed they could use Scale's service to help with their future model training – efforts which were rebooting alongside bringing Wang on board.

NVIDIA's Groq deal seemingly has similarities. Namely, the key was clearly to bring CEO Jonathan Ross on board. And every story also points to company president Sunny Madra being crucial as well. But beyond that, it sounds like NVIDIA may actually care about some of the IP rights here (which they're presumably getting a license to with their "non-exclusive licensing agreement"), to be able to leverage Groq's techniques in would-be future chips.

In that way, "non-exclusive" feels less important here – that's another framing to make it seem less like an acquisition, but is anyone else really getting access to this IP now? Regardless, NVIDIA probably feels confident that with Ross and Madra – not to mention their own in-house prowess – they'll be able to implement it and execute upon it far better than anyone else. And they're undoubtedly not wrong.

And that points to another layer to this as well. Ross is not just a co-founder of Groq, he's also the creator of the TPU – something which he cites in his own bio. You may recall the TPU was last a part of a major news cycle when Jensen Huang was "delighted" about Google's success with their AI chips. How do I know that NVIDIA is actually not so "delighted" about Google's success here and is in fact sweating the rise of the TPU? Well, this deal, for one!....

....MUCH MORE 

Recently:

• "Nvidia Licenses Groq's AI Chip Tech, Grabs Top Execs In Not-Quite Takeover" (NVDA)
• "Nvidia's Groq deal underscores how the AI chip giant uses its massive balance sheet to 'maintain dominance'" (NVDA)  

And on the TPU:

November 25 - CHIPS: Google's Tensor Processing Unit (Finally) A Viable Competitor For Nvidia (GOOG; NVDA)

....While we've been tracking the GOOG's TPU progress for a decade, this morning Zerohedge found a dandy little write-up on the chips that we'll go with rather than use back-links.... 

AI: When Deep Learning Meets the End Of Moore's Law (GOOG; NVDA; INTC; AMD)

Our old buddy Tiernan Ray, formerly Barron's Tech Trader Daily, going deeper than that format allowed.
From ZDNet:

Google says 'exponential' growth of AI is changing nature of compute
Google software engineer Cliff Young explains how the explosion in deep learning algorithms is coinciding with a breakdown in Moore's Law, the decades-old rule of thumb of progress in computer chips, forcing radical new computer designs.

The explosion of AI and machine learning is changing the very nature of computing, so says one of the biggest practitioners of AI, Google.

Google software engineer Cliff Young gave the opening keynote on Thursday morning at the Linley Group Fall Processor Conference, a popular computer-chip symposium put on by venerable semiconductor analysis firm The Linley Group, in Santa Clara, California.

Said Young, the use of AI has reached an "exponential phase" at the very same time that Moore's Law, the decades-old rule of thumb about semiconductor progress, has ground to a standstill.

"The times are slightly neurotic," he mused. "Digital CMOS is slowing down, we see that in Intel's woes in 10-nanometer [chip production], we see it in GlobalFoundries getting out of 7-nanometer, at the same time that there is this deep learning thing happening, there is economic demand." CMOS, or complementary metal-oxide semiconductor, is the most common material for computer chips.

Also: Google preps TPU 3.0 for AI, machine learning, model training

As conventional chips struggle to achieve greater performance and efficiency, demand from AI researchers is surging, noted Young. He rattled off some stats: The number of academic papers about machine learning listed on the arXiv pre-print server maintained by Cornell University concerning is doubling every 18 months. And the number of internal projects focused on AI at Google, he said, is also doubling every 18 months. Even more intense, the number of floating-point arithmetic operations needed to carry out machine learning neural networks is doubling every three and a half months.

All that growth in computing demand is adding up to a "Super Moore's Law," said Young, a phenomenon he called "a bit terrifying," and "a little dangerous," and "something to worry about."

"Why all this exponential growth?" in AI, he asked. "In part, because deep learning just works," he said. "For a long time, I spent my career ignoring machine learning," said Young. "It wasn't clear these things were going to take off."

But then breakthroughs in things such as image recognition started to come quickly, and it became clear deep learning is "incredibly effective," he said. "We have been an AI-first company for most of the last five years," he said, "we rebuilt most of our businesses on it," from search to ads and many more.

The demand from the Google Brain team that leads research on AI is for "gigantic machines" said Young. For example, neural networks are sometimes measured by the number of "weights" they employ, variables that are applied to the neural network to shape its manipulation of data.

Whereas conventional neural nets may have hundreds of thousand of such weights that must be computed, or even millions, Google's scientists are saying "please give us a tera-weight machine," computers capable of computing a trillion weights. That's because "each time you double the size of the [neural] network, we get an improvement in accuracy." Bigger and bigger is the rule in AI.

To respond, of course, Google has been developing its own line of machine learning chips, the "Tensor Processing Unit." The TPU, and parts like it, are needed because traditional CPUs and graphics chips (GPUs) can't keep up....

...MUCH MORE

"Why this critical Google business may be seriously underestimated on Wall Street" (GOOG)

From MarketWatch, November 5: 

Wall Street sees Google Cloud growing at 31% in 2026 — but one analyst is modeling a path to 50%

Alphabet Inc.’s cloud-computing business might be the smallest out of the major hyperscalers, but it’s on track to narrow the gap faster than expected, according to Morgan Stanley analyst Brian Nowak.

Among the “Big Three” cloud-computing providers, Google Cloud Platform trails Amazon.com Inc.’s AWS and Microsoft Corp.’s 

Azure in market share. Yet artificial intelligence could shake up the rankings as Google wins big deals, such as its recent cloud partnership that provides Anthropic with access to its custom tensor processing units.

While Nowak’s base case anticipates Google Cloud revenues growing around 44%, he wrote in a Wednesday note that the business could potentially achieve over 50% growth in 2026 if more large-scale cloud deals are signed. That’s significantly higher than the Wall Street consensus, which according to FactSet is only anticipating 31% cloud growth next year.

While the bull case may seem extreme, the Morgan Stanley analyst believes it’s a very achievable goal based on his Google Cloud revenue forecast. He breaks the segment’s revenues down into cloud backlog, referring to the unrecognized revenue from long-term contracts, and on-demand, or the pay-as-you-go business.

Nowak estimates that Google Cloud will add $106 billion in net backlog and grow its on-demand business at a 25% rate for 2025. Google Cloud would only need to add $50 billion in net backlog and achieve 15% growth in its on-demand business to grow at 50% in 2026. Or, if the on-demand business continues to grow at 25%, just a $20 billion increase in net backlog could result in over 50% cloud growth....

....MUCH MORE 

And at VentureBeat, November 6:

Google debuts AI chips with 4X performance boost, secures Anthropic megadeal worth billions 

Google Cloud is introducing what it calls its most powerful artificial intelligence infrastructure to date, unveiling a seventh-generation Tensor Processing Unit and expanded Arm-based computing options designed to meet surging demand for AI model deployment — what the company characterizes as a fundamental industry shift from training models to serving them to billions of users.

The announcement, made Thursday, centers on Ironwood, Google's latest custom AI accelerator chip, which will become generally available in the coming weeks. In a striking validation of the technology, Anthropic, the AI safety company behind the Claude family of models, disclosed plans to access up to one million of these TPU chips — a commitment worth tens of billions of dollars and among the largest known AI infrastructure deals to date.

The move underscores an intensifying competition among cloud providers to control the infrastructure layer powering artificial intelligence, even as questions mount about whether the industry can sustain its current pace of capital expenditure. Google's approach — building custom silicon rather than relying solely on Nvidia's dominant GPU chips — amounts to a long-term bet that vertical integration from chip design through software will deliver superior economics and performance....

Machine Learning: JP Morgan Compares Google's New Chip With NVIDIA's (GOOG; NVDA)

From Barron's Tech Trader Daily:

Google’s Tensor Intriguing, But Nvidia is the Incumbent, Says JP Morgan

J.P. Morgan semiconductor analyst Joseph Moore today reflects on Alphabet’s (GOOGL) announcement last week, at its annual developer conference, Google I/O, that it has been developing in secret a chip of its own to speed up machine learning, called the “Tensor Processing Unit,” or “TPU,” which the company says is already “powering many applications at Google.”

Google has had input, I would note, from luminaries of the chip world, such as Dave Patterson of U.C. Berkeley, who took a sabbatical at the company a year ago, though he’s not mentioned in the Google materials.

Barron’s magazine’s print edition predicted a Google chip in a cover story last October, “Watch Out Intel, Here Comes Facebook.”

Writes Moore, Nvidia (NVDA), a company that of late has seen tremendous results in selling into the machine learning market made up of Google and Facebook (FB) and others, is the “incumbent” and the one to beat:

Google “Tensor” machine learning chip raises eyebrows – Arguably a positive for NVIDIA despite competitive concerns, as Google investment validates the importance of specialized chips for AI applications. Google announced this week that it has been building applications around a custom chip called “Tensor” to support machine learning applications. There was minimal architectural detail provided, but there were quotes from Google in the press to the effect that a custom solution will be “more efficient than graphics chips” in those applications.... 

...MORE

"The Trillion-Dollar Race to Fragment the Nvidia Monopoly" (NVDA)

Am overview but an overview with insight. 

From EE Times, December 9: 

Nvidia’s ongoing fight to maintain technological and market dominance in AI inference. 

For the past ten years, Nvidia has dominated the market for advanced computer chips used in machine learning and artificial intelligence.

Driven by its proprietary CUDA software and rapid innovation, Nvidia became synonymous with AI processors, briefly reaching a $5 trillion market value this year. Between February and October 2025, Nvidia reported $147.8 billion in sales from chips, network connections, and related hardware supporting AI growth.

Even though Nvidia’s newest and most powerful processors, the Grace Blackwell series, are selling out quickly, its dominance is starting to weaken. Instead of facing just one competitor, Nvidia is now challenged on many fronts as the industry shifts toward more specialized hardware.

Nvidia’s strong sales and high profit margins are driven by scarcity, primarily due to manufacturing limits. The main bottleneck for high-end chips is the limited capacity for advanced Chip-on-Wafer-on-Substrate (CoWoS) packaging at TSMC.

Nvidia consumes most of this limited production, but competition for chip supply is intensifying. TSMC, the sole manufacturer, plans to expand capacity to 100,000 wafers per month by 2026. As supply constraints ease, companies like Google and AMD are expected to benefit.

Nvidia now faces significant risk as the industry shifts from experimenting with large foundation models to prioritizing large-scale, cost-effective inference.

Major cloud providers are moving away from reliance on Nvidia’s CUDA ecosystem. They are investing in their own specialized chips for high-volume inference, where operating costs now exceed training costs.

Nvidia’s largest customers become competitors

The “Big Four” North American hyperscalers—Google, Amazon Web Services, Microsoft, and Meta—account for the largest capital expenditures in this sector. Their collective shift to custom silicon is a strategic move to ensure competitiveness.

Google, part of Alphabet, initiated the move toward custom AI chips with its Tensor Processing Units (TPUs). The latest version, TPU v7 ‘Ironwood,’ is optimized for inference. Ironwood features a large shared memory, enabling up to 9,216 chips to connect in a single ‘superpod,’ which addresses memory constraints in large Mixture-of-Experts (MoE) models.

Meta Platforms may lease and potentially purchase Google’s TPU chips for its data centers starting in 2027. This would mark a shift for Google, positioning it as a merchant chip supplier. Some estimates suggest Google could capture up to 10% of Nvidia’s annual revenue, amounting to billions of dollars.

Amazon focuses on training and cost efficiency

Amazon Web Services (AWS) is pursuing improved price-performance to attract businesses seeking alternatives to Nvidia’s expensive chips. AWS claims its Trainium chips can reduce training costs by up to 50% compared to GPUs, targeting mid-range AI workloads....

....MUCH MORE 

Apple Trained Its Large Language Model On Broadcom-Designed TPUs In Google's Cloud, Meh (AVGO, GOOG, AAPL)

From The Street, July 30:

NVIDIA's Stock Almost Crashes Below $100 After Hours - Then It Surges: Here's Why

It was another miserable day for NVIDIA (Nasdaq: NVDA) as the stock fell 7.04% today. After closing at $103.73, the sell-off continued in after-hours trading, with shares of NVIDIA trading hands for little more than $100 per share. 

And then something changed rapidly. NVIDIA shares jumped and are now trading for $109 per share in after-hours trading, a 5% jump from where they closed trading today. Let’s look at why NVIDIA shares lost so much today and then explore why they’re suddenly rebounding so strongly in after-hours trading. 

Key Points in This Article 

• NVIDIA shares dropped more than 7% today. The losses were primarily attributable to continuing sector rotation out of technology stocks and also fears around the company’s competitive position after it was revealed Apple trained their new AI models on Google’s TPUs. 
• In after-hours trading NVIDIA is rebounding due to positive earnings from rival Advanced Micro Devices (NYSE: AMD) and also positive cloud spending comments from Microsoft (Nasdaq: MSFT)....

...Why NVIDIA Shares Sank 7% Today 

NVIDIA shares sank 7% today as technology stocks in general struggled. The Nasdaq-100 fell 1.38%. Each day the Nasdaq falls, the important metric to watch is its relative performance to the Russell 2000. If the Russell 2000 is strongly outperforming, that demonstrates continuing ‘sector rotation’ of investors moving out of strategies like large technology stocks and into other areas like small caps or value stocks. 

The Russell 2000 gained .35% today, so it outperformed the Nasdaq-100 by 1.73 percentage points, a massive divide between the two major indexes. 

However, there was company-specific news that also played into NVIDIA’s decline today. Apple released a large research paper on the creation of their large language models used in the upcoming release of Apple Intelligence. Here’s the relevant section that impacts NVIDIA (emphasis mine):

“The AFM models are pre-trained on v4 and v5p Cloud TPU clusters with the AXLearn framework [Apple, 2023], a JAX [Bradbury et al., 2018] based deep learning library designed for the public cloud. Training is conducted using a combination of tensor, fully-sharded-data-parallel, and sequence parallelism, allowing training to scale to a large number of model parameters and sequence lengths at high utilization. This system allows us to train the AFM models efficiently and scalably, including AFM-on-device, AFM-server, and larger models. AFM-server was trained on 8192 TPUv4 chips provisioned as 8 × 1024 chip slices, where slices are connected together by the data-center network (DCN) [Chowdhery et al., 2022]. Only data-parallelism crosses the slice boundary, other types of state sharding are within-slice only as the within-slice interconnect bandwidth is orders of magnitude higher than the DCN. The sustained model-flop-utilization (MFU) for this training run was approximately 52%. AFM-on-device was trained on one slice of 2048 TPUv5p chips.”

....MUCH MORE

Pre-market NVDA is up $6.68 (+6.44%) at $110.41.

When I want to scare myself I look at the two gaps—February and May—on the chart as possible downside targets:

 

 

TradingView

On the other hand the Tensor Processing Units still haven't achieved their promise of 2017:

Watch Out NVIDIA: "Google Details Tensor Chip Powers" (GOOG; NVDA)
We've said NVIDIA probably has a couple year head start but this bears watching, so to speak....

Or 2018
Ahead of Today's NVIDIA Earnings: A Look at One of the Competitors (GOOG; NVDA)
We'll have much more on Google next week but for today the next-gen Tensor Processing Unit....

Or 2023:
Chips: "Google TPU v5e AI Chip Debuts after Controversial Origins"