FLOPs in AI: what they are, explained simply

In plain words

Computers do arithmetic with decimal numbers, like 3.14 times 2.7. Each single step, one multiplication or one addition, is a floating-point operation, or FLOP.

AI work is made almost entirely of these steps, so FLOPs are the natural way to measure it. The total FLOPs used to train a model tells you how big the job was. A chip's FLOPs per second tells you how fast it can work.

The numbers are enormous, so people write them as powers of ten. 10²⁵ means a 1 followed by 25 zeros. GPT-4, released in 2023, was the first known model trained with more than 10²⁵ FLOPs, and the European Union now uses that same line to decide which AI models face extra rules.

Everyday example

Counting rotis at a langar

A community kitchen wants to know how big its job is. It counts rotis: 50,000 today. That is the size of the work. Separately, it knows one team rolls 2,000 an hour. That is the speed. Divide the first by the second and you get how long it takes. AI people do the same with FLOPs: total FLOPs is the job, FLOPs per second is the speed.

Where the example breaks: Rotis are all the same. FLOPs are not quite: chips work faster with small, less precise numbers, so a FLOP at low precision is cheaper than one at high precision, and chip makers often quote the fastest kind.

How it actually works

The spelling causes real confusion, so it is worth getting straight.

FLOP, FLOPs and FLOP/s

FLOP is one operation. FLOPs (small s) usually means a count of them, the size of a job. FLOPS or FLOP/s means per second, a speed. "Trained with 10²⁵ FLOPs" is a size; "a 4 petaFLOP/s chip" is a speed.

The prefixes

Tera is 10¹² (a trillion), peta 10¹⁵, exa 10¹⁸. A modern AI chip delivers a few petaFLOP/s at low precision. A pod of thousands of chips reaches exaFLOP/s.

Estimating training compute

For a transformer language model, a standard estimate from the 2020 scaling laws paper is about 6 FLOPs per parameter per training token: the backward pass, where the model learns, costs about twice the forward pass. Multiply parameters by tokens by 6 for a rough total.

Peak versus real

Chips rarely run at their quoted peak, because they wait for data and for each other. Google reported its PaLM model (2022) used about 46% of peak during training, and earlier large models about 20 to 33%. Real runs take longer than the simple division suggests.

Precision matters

A chip may do far more 8-bit or 4-bit operations per second than 32-bit ones. Always check which precision a FLOP/s figure refers to before comparing chips. See quantization.

First model above 10²⁵ FLOPs

GPT-4 (March 2023), by Epoch AI estimates. Over 30 models from 12 developers had crossed it by June 2025 (Epoch AI)

EU AI Act

General-purpose models trained with more than 10²⁵ FLOPs are presumed to have high-impact capabilities (Article 51)

One modern chip

Google's Ironwood TPU: about 4.6 petaFLOP/s at 8-bit precision, at peak

Growth

Training FLOPs for frontier models have grown about 5x a year since 2020 (Epoch AI)

Questions people ask

What does FLOPs mean in AI?

Floating-point operations: single arithmetic steps such as one multiplication of two decimal numbers. In AI, the total FLOPs used to train a model measures how much computing went into it, and FLOPs per second measures how fast a chip is.

What is the difference between FLOPs and FLOPS?

Usually FLOPs (small s) is a count, the total work of a job, and FLOPS or FLOP/s (capital S or a slash) is a rate, operations per second. Writers are not always consistent, so check whether a number is a total or a speed.

How many FLOPs does it take to train a large language model?

For today's leading large language models, somewhere above 10²⁵ and, for the largest, likely 10²⁶ or more. Companies rarely publish exact figures, so most public numbers are estimates by researchers such as Epoch AI.

What is a petaflop?

A thousand million million operations, 10¹⁵. A chip rated at one petaFLOP/s can do that many floating-point operations every second at peak. An exaflop is a thousand times more, 10¹⁸.