Tensor processing units (TPUs) are custom application-specific integrated circuits (ASICs) developed by Google to accelerate machine learning workloads, primarily neural network training and inference.
Why were TPUs created?
In 2013, Google engineers calculated that if users ran voice search through a neural network for just three minutes a day, the company would need to double its global data center capacity. Standard CPUs couldn’t handle the volume without prohibitive cost, and GPUs weren’t optimized for the specific math neural networks require at scale.
That constraint drove Google to design purpose-built silicon. The first TPU deployed internally in 2015, before any public announcement. By the time Google revealed them at Google I/O 2016, TPUs had already been running production workloads inside Google’s AI data centers for over a year.
How do TPUs work?
Neural networks perform billions of matrix multiplications during training and inference. TPUs are built around a matrix multiply unit (MXU) that contains a 128×128 systolic array, 16,384 multiplier-accumulator units that pass data through in a wave-like pattern. Rather than fetching each value from memory independently, data flows continuously through the array, eliminating the memory bottleneck that slows general-purpose processors on matrix math.
The architecture also includes:
- High-bandwidth memory (HBM): Stacked memory that sits close to the processing units to reduce data transfer latency
- Reduced-precision arithmetic: TPUs use bfloat16 (a 16-bit format that preserves the range of 32-bit floats) and INT8 for faster computation with minimal accuracy loss
- XLA compiler: Google’s Accelerated Linear Algebra compiler translates ML framework code into optimized instructions for the TPU’s specific hardware layout
What makes TPUs different from GPUs?
Both accelerate AI workloads, but they reflect different design philosophies. GPUs were built for graphics and later adapted for parallel computation. TPUs were designed from day one for tensor operations, which makes them faster for that workload but inflexible outside of it.
| TPU | GPU | |
|---|---|---|
| Design origin | Matrix math for ML | Graphics rendering |
| Precision support | bfloat16, INT8 | FP32, FP16, INT8, FP64 |
| Framework support | TensorFlow, JAX, PyTorch (via plugin) | All major frameworks |
| Custom operations | Limited | Broad support |
| Energy efficiency | High for ML workloads | Moderate |
| Availability | Google Cloud only | AWS, Azure, GCP, on-premise |
The GPU’s broader ecosystem fits more workflows. The TPU’s narrower design is precisely what gives it the performance advantage when training large language models.
When should you use TPUs for AI workloads?
Google’s own guidance draws clear lines between workloads that benefit from TPUs and those that don’t.
TPUs perform well for:
- Models built primarily on matrix multiplications (transformers, CNNs)
- Training large language models with billions or trillions of parameters
- Long training runs measured in days or weeks
- Large batch sizes, ideally multiples of 128, to align with the MXU grid
- High-volume production inference at Google Cloud scale
GPUs or CPUs are better suited for:
- Rapid prototyping where framework flexibility matters
- Models with custom operations TPUs can’t natively support
- Workloads requiring FP64 precision
- Code with heavy branching or sparse computation
- Teams outside Google Cloud who need hardware portability
If you’re early in development or need to iterate quickly, GPUs are the practical starting point. TPUs become advantageous once a model’s architecture is stable and you’re optimizing throughput at scale.
What are the different generations of TPUs?
Google has released six major TPU generations since 2015, each expanding both raw performance and the types of workloads supported:
- TPU v1 (2015): 92 TOPS, inference-only, deployed internally before any public announcement
- TPU v2 (2017): 45 TFLOPS, added training capability with bfloat16 support, first version offered via Google Cloud
- TPU v3 (2018): 90 TFLOPS, introduced liquid cooling to handle increased thermal output
- TPU v4 (2021): 275 TFLOPS, optical interconnects for pod-scale networking, available in 4,096-chip configurations
- TPU v5 (2023): Roughly 2x faster than v4; also introduced a v5e variant optimized for cost-efficient inference
- Trillium / TPU v6 (2024): Over 4x the compute of v5e per chip, with 4x higher memory bandwidth
Google also ships an Edge TPU, a low-power chip for on-device inference. It appears in Coral hardware and targets IoT scenarios where sending data to a remote server is not practical. For background on distributed on-device learning, see what is federated learning.
How can developers access and use TPUs?
Access runs almost entirely through Google Cloud:
- Cloud TPU VMs: Direct VM instances with attached TPU accelerators
- Google Kubernetes Engine (GKE): Orchestrated TPU workloads for containerized ML pipelines; relevant if you’re already managing MLOps workflows
- Vertex AI: Managed training and serving with TPU support, useful for teams that want infrastructure abstracted away
- Google Colab and Kaggle: Free TPU access capped at session time limits, practical for experimentation
TPUs work with TensorFlow, PyTorch (via the torch_xla plugin), and JAX. JAX tends to produce the best performance because it compiles directly through XLA with minimal overhead. Code written for JAX generally requires fewer architecture-specific changes to run efficiently on TPU hardware.
One practical constraint: tensor dimensions that are multiples of 128 align with the MXU grid and avoid padding overhead. Misaligned shapes can cut effective utilization significantly, which matters when you’re paying for TPU pod time.
What real-world applications use TPUs?
Google runs a large share of its own products on TPU infrastructure:
- Search and RankBrain: Query understanding at billions of requests per day
- Google Photos: Image recognition and organization across massive photo libraries
- AlphaGo, AlphaZero, and AlphaFold: Game-playing research agents and protein structure prediction
- Gemini: Google’s flagship large language model family, trained on TPU pods
- Street View: Text extraction from the full global Street View database, completed in under five days
Outside Google, researchers access TPUs through Google Cloud for training runs that would be cost-prohibitive on other hardware. The TPU Research Cloud program has historically offered free access to academic groups.
The protein folding work in AlphaFold illustrates how TPU-scale compute opens research that was previously out of reach. Projects at that scale also depend heavily on AI infrastructure investment and synthetic data pipelines that make large-scale training operationally viable.
Conclusion
TPUs trade flexibility for performance on a specific class of workloads. For teams training large transformer models or running inference at Google Cloud scale, they offer better throughput and energy efficiency than general-purpose GPUs. For everyone else, GPUs remain the more practical choice. The tradeoff is clear: TPUs win on raw ML performance; GPUs win on adaptability.
As model sizes grow, demand for purpose-built accelerators will increase alongside it. That pressure is already driving investment in alternatives, including neuromorphic chips, analog compute, and quantum approaches to AI power consumption. TPUs are one answer to the hardware problem that AI scale creates. They won’t be the last.