What is a TPU? Google's new TPUs, explained
What is a TPU? Google just unveiled the TPU 8tx and TPU 8i Max, so here's what TPUs are, how they stack up against GPUs and what makes Google's different.
On this page 5 sections ▾
TL;DR
- You can't buy one off the shelf. Google rents TPUs through Google Cloud, and Bloomberg (via Benzinga) reports both new systems should be available in 2027.
- This is Google's play against Nvidia: Google says TPUs now have nearly 100 customers, and Bloomberg (via Benzinga) names Apple, Meta and Oracle among them.
- TPUs aren't a drop-in replacement for GPUs. They win on huge, matrix-heavy AI jobs and lose on flexibility, and Google still rents out Nvidia GPUs alongside them.
Google just added two new TPU systems to its lineup, the TPU 8tx and the TPU 8i Max, and the coverage is mostly about one thing: whether Google can take a real bite out of Nvidia. If you've seen the name and wondered what a TPU actually is, the short answer is a chip Google designed for AI math and nothing else, and it's a big part of why Google doesn't have to buy all of its AI hardware from Nvidia like most of the industry does.
Quick facts
The TPU started with a scare: Jeff Dean worked out that if 100 million people talked to their phones for 3 minutes a day using Google's speech recognition, Google would need to double the number of computers it had.
Google figured it might build fewer than 10,000 of its first TPU. It ended up building over 100,000.
Google's first TPU pod, back in the second generation, linked 256 chips with a custom high-speed interconnect. That idea grew into the giant superpods Google builds today.
Google says it now uses its own AI, AlphaChip and Gemini, to lay out chip designs in hours instead of weeks.
#What Google announced
On November 9, Google added two systems to its eighth generation of TPUs, which it first unveiled in April. The TPU 8tx is aimed at governments and heavily regulated industries, and uses hardware-level security to protect data even while it's being processed. The TPU 8i Max puts two existing TPU 8i chips into one package, doubling the computing power and memory bandwidth for work like many AI agents chewing on the same large codebase.
The two new systems build on the pair of chips Google announced in April:
| ● TPU 8t | ● TPU 8i | |
|---|---|---|
| Built for | Training huge models | Answering requests fast (inference) |
| High-bandwidth memory | 2 PB shared across a 9,600-chip superpod | 288 GB per chip |
| On-chip memory (SRAM) | Not published | 384 MB per chip, 3 times the last generation |
| Cooling | 4th-generation liquid cooling | 4th-generation liquid cooling |
For raw speed, ServeTheHome's coverage of Google's Hot Chips talk puts a full TPU 8t superpod at 121 EFLOPS of FP4 compute, with about twice the performance per watt of Ironwood, the TPU generation before it. Google hasn't published a per-chip figure, and FP4 numbers can't be compared directly with numbers quoted in other formats.
ELI5: explain it simply▾
THE WORDS
- FLOPS
- Math operations per second. An EFLOP is a billion billion of them.
- FP4
- A tiny 4-bit number format. Less precise, but AI can use it to go much faster.
- Performance per watt
- How much work a chip does for each unit of electricity it uses.
THE IDEA
Chip makers quote speed in whichever number format makes them look fastest, a bit like quoting a car's top speed downhill with a tailwind. That's why a FLOPS number only means something next to the format it was measured in.
For your information
Google says this is the first time it has launched four TPU systems in a single year, but it hasn't announced pricing or published full specs for the two new systems.
#What is a TPU?
TPU is short for Tensor Processing Unit, and it's what the industry calls an ASIC: a chip built for a single purpose. That purpose is multiplying huge grids of numbers (matrices) together, which is most of what training and running an AI model comes down to. At the heart of every TPU is a matrix unit that Google describes as a 128 by 128 systolic array.
ELI5: explain it simply▾
THE WORDS
- ASIC
- A chip built to do one specific job, instead of anything you throw at it.
- Matrix
- A grid of numbers. AI models are mostly huge grids being multiplied together.
- Systolic array
- A grid of small calculators that each do a step and hand the result to the next one.
THE IDEA
Think of a car factory. A CPU is a skilled mechanic who can build any part, one at a time. A TPU is an assembly line that only builds one kind of car, but every station passes the car straight to the next, so it pumps them out far faster than the mechanic ever could.
A systolic array is a grid of tiny calculators wired only to their neighbors. Each number gets read once and then passed along to be reused in many calculations, with every in-between result handed straight to the next calculator instead of being written to memory, which saves a lot of power and time. The name comes from the heart: data pulses through the chip in waves, the way a heart pumps blood.
Google's first TPU only ran finished models (inference). For training, Google still bought off-the-shelf GPUs at the time.
#TPU vs GPU: what's the difference?
A CPU is the jack-of-all-trades brain of a computer. GPUs started out as specialized helpers that work alongside the CPU on graphics, video and simulations, and they turned out to handle AI math well too. A TPU goes a step further and is built for nothing but the matrix and vector math that AI models run on.
| ● GPU | ● TPU | |
|---|---|---|
| Originally built for | Graphics, video, simulations | AI math only |
| Who makes it | Nvidia, AMD and others | Google, with chip design partners |
| How you get one | Buy it or rent it in the cloud | Mostly rented on Google Cloud |
| Google suggests it for | Models with lots of custom operations | Huge matrix-heavy models that train for weeks |
| Google's docs steer away from it for | ~Small, simple models (a CPU will do) | Lots of branching, high-precision math, data shapes that change |
Where it breaks
Chip for chip, a TPU isn't the fastest thing out there. Nvidia's newest Rubin GPU is much more powerful than either TPU 8 chip on its own, which is why Google's pitch is all about what happens when thousands of TPUs work together.
The bigger reason teams stick with GPUs is software. Most AI development relies on PyTorch, which teams tune for CUDA, Nvidia's software platform, while TPUs have historically worked best with JAX, Google's own library. Moving off GPUs can be expensive and slow, and engineers with TPU experience are much harder to find.
That's the wall Google is trying to knock down. Alongside the new systems, Google says PyTorch code and Hugging Face models now run natively on TPUs, and that popular open-source serving tools like vLLM and SGLang work on TPUs out of the box.
#What makes Google's TPU special
- Thousands of chips act as oneA TPU 8t superpod works as one machine, far bigger than the groups Nvidia wires together over NVLink, and Google can stitch more than a million TPUs into a single training cluster.
- Its own network designsThe 8i swaps the old grid-style layout for one Google calls Boardfly, which cuts the worst-case trip between chips in a 1,024-chip setup from 16 hops to 7.
- Split for the jobThis is the first generation with separate chip designs for training (8t) and for answering requests (8i), instead of one chip doing both.
ELI5: explain it simply▾
THE WORDS
- Superpod
- A huge group of TPUs wired together so they can work on one job.
- Hop
- One step data takes when it passes from one chip to the next.
- Training
- Teaching a model by feeding it data. Answering requests afterward is called inference.
THE IDEA
One fast chef can only cook so much. Google's bet is a kitchen with thousands of decent chefs who pass ingredients to each other in as few steps as possible, so together they out-cook a few superstars working alone.
Google has also been running its own models, including Gemini, on TPUs for years.
Want to see what these chips actually run? The AI Pro ProCourse walks through how AI models work, starting from Linux basics and building up to ML foundations and fine-tuning, all in a simulated terminal in the browser, and Level 1 is free.
#How Google got here
- TPU v1 goes into service inside Google's data centers.
- Google reveals the TPU publicly at its I/O conference.
- Cloud TPUs launch, letting Google Cloud customers rent them for the first time.
- Google announces Trillium, its sixth-generation TPU, at I/O.
Is a TPU better than a GPU?
Can I buy a Google TPU?
What is a TPU used for?
Who uses Google TPUs?
What this means for you
If you run AI workloads on Google Cloud, TPUs are now worth pricing against GPUs, especially for big training runs or long-context serving, though the two new systems aren't out yet. If you're learning AI, the brand of chip matters less than the ideas underneath it: training versus inference, why AI is mostly matrix math, and why memory and networking matter alongside raw chip speed.
Sources