In plain words
The biggest AI models are far too large to train on one chip, or even one computer. So companies link thousands of GPUs or other AI chips with very fast cables and switches, in one data center or a few close together. That linked group is a training cluster.
The chips split the work. Each one handles a slice of the model or a slice of the training data, and after every small step they all swap results so the whole model learns together. If the cables are slow, the chips spend their time waiting, so the network matters as much as the chips.
Clusters keep growing. Meta trained its Llama 3 model in 2024 on up to 16,000 GPUs. By early 2026, Epoch AI estimated that the largest known AI data center had the computing power of about 1.1 million Nvidia H100 chips.
Everyday example
One giant rangoli
Picture 1,000 people making one huge rangoli for Diwali. Each person colours one patch. Every few minutes everyone pauses and checks with the people around them, so the lines meet and the pattern stays one picture. If those checks are slow, people stand idle. A training cluster works the same way: each chip does its patch, then all of them sync before the next step.
Where the example breaks: A rangoli carries on if one person walks away. In a training cluster one failed chip can stall the whole job, so runs save their progress often and restart from the last save.
How it actually works
A training cluster is designed around one problem: keeping every chip busy while they constantly share results with each other.
Splitting the work
Engineers combine several ways of splitting. Data parallelism gives each group of chips different examples. Tensor and pipeline parallelism cut the model itself into pieces spread across chips. Meta's Llama 3 used four kinds of splitting at once.
Two layers of network
Inside a server or rack, chips talk over very fast direct links such as Nvidia's NVLink, which can join 72 GPUs so they act as one big accelerator. Between racks they use networks such as InfiniBand or fast Ethernet.
Things break, constantly
With tens of thousands of parts, failures are routine. Over 54 days of Llama 3 training, Meta counted 419 unexpected interruptions, about 78% traced to hardware. Clusters save checkpoints often, so a failure costs minutes of work, not weeks.
Power swings
When thousands of chips start and finish each step together, the building's power draw can swing by tens of megawatts in moments, which the power systems must absorb.
Growing fast
Epoch AI found that the performance of leading AI supercomputers doubled about every nine months between 2019 and 2025, while their power needs roughly doubled every year. This growth is driven by scaling laws: bigger training runs have kept producing better models.
- Llama 3 (2024)
- Trained on up to 16,000 Nvidia H100 GPUs, 8 per server, linked at 400 gigabits per second
- Largest known site, early 2026
- About 1.1 million H100-equivalent GPUs of compute (Epoch AI estimate, February 2026)
- Power of the largest
- Approaching a gigawatt, enough for roughly a million US homes (Epoch AI)
- Who owns them
- Companies held about 80% of AI supercomputer computing power in 2025, up from about 40% in 2019 (Epoch AI)
The same clusters are often used for inference too, answering users between or alongside training runs.
Questions people ask
How many GPUs does it take to train a large AI model?
For the largest models, tens of thousands, running for weeks or months. Meta's Llama 3, published in 2024, used up to 16,000 H100 GPUs. Many labs do not disclose their numbers, and sites built since then are far larger. Smaller models can be trained on a handful of GPUs.
What is a GPU cluster?
A group of computers, each holding several GPUs, linked by fast networks so they can work on one job together. A training cluster is a GPU cluster built and tuned for training AI models, where the chips must swap results many times a second.
Why do AI companies keep building bigger clusters?
Because, so far, training with more compute has reliably produced more capable models, a pattern described by scaling laws. A bigger cluster lets a lab train a larger model, or train faster. Whether that pattern will keep holding is debated.
Is a training cluster the same as a supercomputer?
Close. Both link many processors with fast networks to work as one machine. Traditional supercomputers serve science, such as weather and physics, and need very precise maths. AI training clusters are built around AI chips and lower-precision maths, and are now mostly owned by companies.