What is a TPU? Google's new TPUs, explained
What is a TPU? Google just unveiled the TPU 8tx and TPU 8i Max, so here's what TPUs are, how they stack up against GPUs and what makes Google's different.
On this page 5 sections ▾
TL;DR
- You can't buy Google's data center TPUs. Google rents them out through Google Cloud, and Bloomberg reports both new systems should be available in 2027.
- This is Google's play against Nvidia: it says nearly 100 companies now use TPUs, and Bloomberg names Apple, Meta and Oracle among them.
- TPUs aren't a drop-in replacement for GPUs. They win on huge, matrix-heavy AI jobs and lose on flexibility, and Google still rents out Nvidia GPUs alongside them.
Google just added two new TPU systems to its lineup, the TPU 8tx and the TPU 8i Max, and the coverage is mostly about one thing: whether Google can take a real bite out of Nvidia. If you've seen the name and wondered what a TPU actually is, the short answer is a chip Google designed for AI math and nothing else, and it's a big part of why Google doesn't have to buy all of its AI hardware from Nvidia like most of the industry does.
Quick facts
The TPU started with a scare: Jeff Dean worked out that if hundreds of millions of people talked to Google for 3 minutes a day using speech recognition, Google would need to double the number of computers in its data centers.
Google figured it might build fewer than 10,000 of its first TPU. It ended up building over 100,000.
Google's first TPU pod, back in the second generation, linked 256 chips with a custom high-speed interconnect. That idea grew into the giant superpods Google builds today.
Google says it now uses its own AI, AlphaChip and Gemini, to lay out chip designs in hours instead of weeks.
#What Google announced
On November 9, Google added two systems to its eighth generation of TPUs, which it first unveiled in April. The TPU 8tx is aimed at governments and heavily regulated industries, and uses hardware-level security to protect data even while it's being processed. The TPU 8i Max puts two existing TPU 8i chips into one package, doubling the computing power and memory bandwidth for work like many AI agents chewing on the same large codebase.
Here's what the two April chips do:
- TPU 8tBuilt for training huge models, linking 9,600 chips into one superpod with 2 PB of shared high-bandwidth memory.
- TPU 8iBuilt for answering requests fast, with 288 GB of high-bandwidth memory and 384 MB of on-chip memory per chip.
For your information
Google says this is the first time it has launched four TPU systems in a single year, but it hasn't announced pricing or published full specs for the two new systems.
#What is a TPU?
TPU is short for Tensor Processing Unit, and it's what the industry calls an ASIC: a chip built for a single purpose. That purpose is multiplying huge grids of numbers (matrices) together, which is most of what training and running an AI model comes down to. At the heart of every TPU is a matrix unit that Google describes as a 128 by 128 systolic array.
ELI5: explain it simply▾
THE WORDS
- ASIC
- A chip built to do one specific job, instead of anything you throw at it.
- Matrix
- A grid of numbers. AI models are mostly huge grids being multiplied together.
- Systolic array
- A grid of small calculators that each do a step and hand the result to the next one.
THE IDEA
Think of a car factory. A CPU is a skilled mechanic who can build any part, one at a time. A TPU is an assembly line that only builds one kind of car, but every station passes the car straight to the next, so it pumps them out far faster than the mechanic ever could.
A systolic array is a grid of tiny calculators wired only to their neighbors. Each number gets read once and then passed along to be reused in many calculations, with every in-between result handed straight to the next calculator instead of being written to memory, which saves a lot of power and time. The name comes from the heart: data pulses through the chip in waves, the way a heart pumps blood.
Google's first TPU only ran finished models (inference). For training, Google still bought off-the-shelf GPUs at the time.
#TPU vs GPU: what's the difference?
A CPU is the jack-of-all-trades brain of a computer. GPUs started out as specialized helpers that work alongside the CPU on graphics, video and simulations, and they turned out to handle AI math well too. A TPU goes a step further and is built for nothing but the matrix and vector math that AI models run on.
| ● GPU | ● TPU | |
|---|---|---|
| Originally built for | Graphics, video, simulations | AI math only |
| Who makes it | Nvidia, AMD and others | Google, with chip design partners |
| How you get one | Buy it or rent it in the cloud | Rent it on Google Cloud |
| Google suggests it for | Models with lots of custom operations | Huge matrix-heavy models that train for weeks |
| Google's docs steer away from it for | ~Small, simple models (a CPU will do) | Lots of branching, high-precision math, data shapes that change |
Where it breaks
Chip for chip, a TPU isn't the fastest thing out there. Tom's Hardware estimates Nvidia's and AMD's newest chips have roughly 3 times the raw compute of a single TPU 8t, which is why Google's pitch is all about what happens when thousands of TPUs work together.
#What makes Google's TPU special
- Thousands of chips act as oneA TPU 8t superpod links 9,600 chips, while Nvidia's current GPUs top out at 576 in a single NVLink setup, and Google's network can tie more than a million TPUs together across sites.
- Its own network designsThe 8i swaps the old grid-style layout for one Google calls Boardfly, which cuts the worst-case trip between chips in a 1,024-chip setup from 16 hops to 7.
- Split for the jobThis is the first generation with separate chip designs for training (8t) and for answering requests (8i), instead of one chip doing both.
ELI5: explain it simply▾
THE WORDS
- Superpod
- A huge group of TPUs wired together so they can work on one job.
- Hop
- One step data takes when it passes from one chip to the next.
- Training
- Teaching a model by feeding it data. Answering requests afterward is called inference.
THE IDEA
One fast chef can only cook so much. Google's bet is a kitchen with thousands of decent chefs who pass ingredients to each other in as few steps as possible, so together they out-cook a few superstars working alone.
Both April chips use Google's fourth generation of liquid cooling, which handles power levels that air cooling can't, and Google has been running its own models, including Gemini, on TPUs for years.
Want to see what these chips actually run? The AI Pro ProCourse walks through how AI models work, starting from Linux basics and building up to ML foundations and fine-tuning, all in a simulated terminal in the browser, and Level 1 is free.
#How Google got here
- TPU v1 goes into service inside Google's data centers.
- Google reveals the TPU publicly at its I/O conference.
- Cloud TPUs launch, letting Google Cloud customers rent them for the first time.
- Google announces Trillium, its sixth-generation TPU, at I/O.
Is a TPU better than a GPU?
Can I buy a Google TPU?
What is a TPU used for?
Who uses Google TPUs?
What this means for you
If you run AI workloads on Google Cloud, TPUs are now worth pricing against GPUs, especially for big training runs or long-context serving, though the two new systems aren't out yet. If you're learning AI, the brand of chip matters less than the ideas underneath it: training versus inference, why AI is mostly matrix math, and why memory and networking matter alongside raw chip speed.
Sources