In plain words
A GPU has thousands of small cores, but on its own it was built to draw graphics. To use it for anything else, a programmer needs a way to say: here is a big job, split it up and run it on all your cores at once.
CUDA is that way in. It is a set of software tools from Nvidia, first introduced in 2006, that lets ordinary programs send work to Nvidia GPUs. The programmer writes one small piece of code, and CUDA runs copies of it on thousands of cores in parallel, each on its own slice of the numbers.
Over the years Nvidia built ready-made CUDA libraries for the maths AI uses most. Popular AI toolkits call those libraries underneath, so most people who train a model never write CUDA themselves. It is also a big reason AI work tends to run on Nvidia chips.
Everyday example
5,000 laddoos for a wedding
A caterer gets an order for 5,000 laddoos. The head cook does not roll them herself. She writes one simple recipe card, makes 40 copies and hands one to each helper with a tray of dough. Everyone follows the same card at the same time. CUDA is the system of recipe cards and trays: one instruction, copied to thousands of GPU cores, each working on its own share of the numbers.
Where the example breaks: Helpers can improvise when something goes wrong. GPU cores cannot: they follow the same instructions in lockstep, and the programmer must plan exactly how the work and the data are split, or the cores sit idle.
How it actually works
CUDA is both a programming model, a way of describing parallel work, and a platform: a compiler, drivers and libraries for Nvidia GPUs.
Host and device
In CUDA's terms the CPU and its memory are the host, and the GPU and its memory are the device. A program starts on the CPU, copies data into the GPU's memory, asks the GPU to do the work, then copies the results back.
Kernels and threads
The code that runs on the GPU is called a kernel. Launching a kernel starts thousands of threads, each running the same code on a different piece of the data, for example one cell of a big grid of numbers.
Blocks and grids
Threads are grouped into blocks, and blocks into a grid. Each block runs on one section of the chip, so the same program can spread across a small GPU or a huge one without being rewritten.
Libraries do most of the work
Few AI researchers write kernels by hand. Libraries such as cuDNN, first described in 2014, hold tuned versions of the sums neural networks need, and frameworks such as PyTorch call them for you.
Why switching is slow
Years of code, tools and know-how assume CUDA. AMD offers HIP, which can convert much CUDA code for AMD GPUs, and OpenAI's Triton (2021) lets people write fast GPU code in a Python-like language. Both help, but moving a big project still takes work.
- Made by
- Nvidia. It runs only on Nvidia GPUs
- Introduced
- November 2006, with the GeForce 8800 graphics card; first public toolkit in 2007
- Name
- Originally short for Compute Unified Device Architecture; Nvidia now rarely spells it out
- Current version
- CUDA Toolkit 13.4, as of October 2026
Other AI chips, such as Google's TPU, use their own software stacks instead of CUDA.
Questions people ask
What does CUDA stand for?
When Nvidia introduced it in 2006, CUDA was short for Compute Unified Device Architecture. Nvidia has since stopped expanding the name, and today CUDA simply means its platform for running general-purpose maths on Nvidia GPUs.
Is CUDA a programming language?
Not exactly. CUDA is a platform: a compiler, drivers and libraries. Programmers usually write CUDA code in C++ with a few extra keywords, and Nvidia also offers ways to use it from Python. Most AI developers use it indirectly, through frameworks such as PyTorch.
Can CUDA run on AMD or Intel GPUs?
No. CUDA is built for Nvidia GPUs. AMD's HIP tools, part of its ROCm software, can convert much CUDA code to run on AMD chips, but AMD itself says HIP is not a drop-in replacement, and porting a project usually needs some hand work and tuning.
Do I need CUDA to use ChatGPT?
No. Apps like ChatGPT run in the company's data centers, and you only see the answer. You meet CUDA only if you run or train models on your own Nvidia GPU, where AI tools often ask you to install a matching CUDA version.
Why is CUDA called Nvidia's moat?
A moat is a business advantage rivals find hard to cross. So much AI software is written and tuned for CUDA that moving to another chip means re-testing and often rewriting code. That keeps many buyers on Nvidia GPUs even when rivals offer strong hardware.
Further reading
- CUDA Programming Guide: programming modelNvidia, 2026
- Nvidia introduces CUDA architecture for computingComputer Graphics World, 2006
- cuDNN: Efficient Primitives for Deep Learning (Chetlur et al.)arXiv, 2014
- What is HIP?AMD ROCm documentation, 2026
- Introducing Triton: open-source GPU programming for neural networksOpenAI, 2021