In plain words
Most AI you use lives far away. When you ask a chatbot something, your words travel to a data center, a big model works out the answer, and it is sent back.
On-device AI skips the trip. A smaller model is stored on the phone or laptop itself and does the work right there, often on a special AI chip called an NPU. Features like summarising a voice recording, suggesting a reply or cleaning up a photo can run this way.
That brings three benefits. Your data can stay on your device, which helps privacy. It works without a network, on a flight or in a lift. And it can feel faster, with no round trip. The catch: a phone has far less memory and power than a data center, so on-device models are smaller and less capable. Many apps mix both, handling easy jobs locally and sending hard ones to the cloud.
Everyday example
A calculator in your pocket
You could phone a friend who is good at maths every time you need a sum, or you could use the calculator on your phone. The friend can handle harder problems, but needs a call, takes longer, and hears your numbers. The calculator works anywhere, instantly and privately, but only for what it was built to do. On-device AI is the calculator; cloud AI is the friend.
Where the example breaks: A calculator never gets things wrong. An on-device AI model is a small language model and can still make mistakes, sometimes more than its big cloud cousin.
How it actually works
Running a model on a phone means squeezing it into a few gigabytes of memory and a few watts of power.
Smaller models
On-device models usually have a few billion parameters, compared with hundreds of billions or more for large cloud models. Apple described its 2025 on-device model as about 3 billion parameters. See small language models.
Compression
Weights are stored with fewer bits each, a technique called quantization. Apple says it compressed its on-device model to 2 bits per weight. Smaller numbers mean less memory and less energy per answer.
Dedicated chips
Phones and newer laptops include an NPU, a block built for AI maths that does it using far less battery than the main processor. The operating system routes the model's work to it.
Shared system models
Rather than every app shipping its own model, the phone's maker often provides one that apps can call. Google's Gemini Nano runs inside an Android service called AICore; Apple offers its on-device model to developers too.
Hybrid by design
When a task is too big for the device, apps can fall back to a larger model in the cloud. Google's own developer guide describes this mix, routing requests to cloud Gemini when Nano is unavailable or reaches its limits.
- Early mainstream example
- December 2023: Google's Gemini Nano began summarising recordings on the Pixel 8 Pro, working without a network
- Apple Intelligence
- Uses a roughly 3 billion parameter model on the device, plus larger models on Apple's own servers (Apple, July 2025)
- Laptops
- Microsoft's Copilot+ PCs, launched May 2024, require an NPU of 40 trillion operations per second or more
- Also called
- Edge AI or local AI
Questions people ask
What does on-device AI mean?
It means the AI model runs on your own device, such as a phone, laptop or car, instead of on a company's servers. Your request is processed locally, so it can work offline and your data does not have to be sent over the internet.
Is on-device AI more private?
Usually, because the data you give it can stay on your device. But check the details: some features quietly hand harder requests to the cloud, and an app can still choose to send data elsewhere. Look for whether a feature says it runs fully on device.
Can you run ChatGPT offline on a phone?
Not the main ChatGPT service, which runs in data centers. You can run smaller open-weight models offline with dedicated apps, or use built-in features such as Gemini Nano or Apple's on-device model, but they are less capable than the big cloud models.
What is the difference between on-device AI and cloud AI?
Cloud AI runs large models on powerful chips in data centers and sends results over the internet. On-device AI runs smaller models on your own hardware. Cloud is more capable; on-device is more private, works offline and avoids network delay.
Does on-device AI drain the battery?
It uses some power. Phones send this work to an NPU, which does AI maths using far less energy than the main processor; chipmakers claim many times less. Long or heavy tasks, such as generating lots of text, can still warm the phone and use noticeable battery.