Products, laws and numbers here change fast. Checked October 2026.
In plain words
Training a model is a big one-time bill. But every time someone asks a chatbot a question, the model has to run again on chips in a data center. That is inference, and its running cost is paid on every single answer.
AI companies measure that work in tokens, small chunks of text about three quarters of a word long. Developers who use a model through an API are usually charged per million tokens, with one price for the words they send in and a higher price for the words the model writes back.
The cost per token has been falling fast as chips, software and smaller models improve. But people now ask for longer answers, attach whole documents and use reasoning models that think at length before replying, so the total bill can still grow.
Everyday example
Paying an auto by the meter
An auto meter charges a little to get in and then a set rate for every kilometre. AI pricing works like a meter too. The words you send are the short hop to the stand; the words the model writes back are the longer journey, charged at a higher rate. A reasoning model takes the scenic route, thinking through extra tokens you never see, and those are on the meter too.
Where the example breaks: An auto fare depends only on distance. An AI bill also depends on which model you pick: the same question can cost many times more on a large model than on a small one.
How it actually works
Inference cost is mostly chip time: how many seconds of a GPU or other accelerator each answer uses, plus the electricity, cooling and buildings around it.
Input and output are priced differently
Reading your prompt happens in one fast parallel pass. Writing the reply happens one token at a time, which keeps the chip busy longer. So output tokens cost more: one major provider charges five times as much for output as for input, as of October 2026.
Hidden thinking is billed
Reasoning models work through internal steps before answering. OpenAI's documentation says these reasoning tokens are not shown to users but are billed as output tokens. A short visible answer can hide thousands of paid tokens.
Sharing chips lowers the price
Each new token means reading the model's weights from memory. Serving many users together, called batching, reads the weights once for many requests, so each person's share of the chip time falls.
Reuse and patience are cheaper
Providers discount prompt text that is stored and reused, called prompt caching, and jobs that can wait hours instead of seconds. Anthropic, for example, offers 50% off for such batch jobs.
The price keeps falling
Stanford's AI Index 2025 found the cost of GPT-3.5-level performance fell more than 280 times between November 2022 and October 2024. Epoch AI found the price of a fixed level of benchmark performance falling 9 to 900 times a year, depending on the task.
- Usual unit
- Price per million tokens, split into input and output
- Size of a token
- About three quarters of an English word; many other languages use more tokens for the same text
- GPT-3.5-level cost
- From about $20 per million tokens in November 2022 to $0.07 in October 2024 (Stanford AI Index 2025)
- Price trend
- Falling 9 to 900 times a year for a fixed level of performance (Epoch AI estimate, March 2025)
- Hardware
- AI hardware costs fell about 30% in a year, and new chips were about 40% more energy efficient (AI Index 2025)
Falling prices per token and rising use are both true at once, which is why AI companies' total chip bills keep climbing.
Questions people ask
How much does one ChatGPT question cost to answer?
OpenAI does not publish a cost per question, so outside figures are estimates. What is clear is what drives it: longer prompts, longer answers, bigger models and reasoning all use more tokens and more chip time. A short chat costs far less than summarising a long document.
Why are output tokens more expensive than input tokens?
The model reads your whole prompt in one parallel pass, which uses the chip efficiently. It writes its answer one token at a time, and each step needs another read through the model's weights. Writing ties up the chip for longer, so providers charge more for it.
Why is AI getting cheaper?
Several things at once: faster and more efficient chips, better serving software, techniques like quantization and distillation that make models smaller, and competition between providers. Stanford's AI Index found the cost of GPT-3.5-level performance fell more than 280 times in about two years.
Why do free AI apps have message limits?
Because every answer uses chip time that the company pays for, whether or not you do. Limits stop a few heavy users from running up costs. Paid plans buy more of that chip time, and the most demanding features, such as reasoning or video, are usually limited most.
What does cost per million tokens mean?
It is how AI providers price their API. A million tokens is roughly 750,000 English words. A price is listed for input tokens, the text you send, and a higher one for output tokens, the text the model writes. Your bill is the tokens you used times those rates.