AI inference is when a trained machine learning model makes a prediction on new data it has never seen before. It is the moment an AI stops learning and starts doing work, producing an actual output from an input.
In standard deployments, a large language model’s parameters are frozen after training. The core model does not persistently learn from user inputs in real time. It does not consult a database of memorized facts. Instead, it performs mathematical operations to predict the next sequential token based on statistical probabilities baked into its weights during training. For models deployed at scale, these continuous inference operations account for the majority of the model’s total lifetime computing power and energy consumption.
AI training vs inference vs fine-tuning: What is the difference?
If training is cramming for an exam, inference is sitting the exam itself.
AI training is the computationally intensive learning phase. A model processes a massive dataset, iteratively adjusting its internal parameters until it can recognize patterns. Each training run generates predictions on training examples, calculates the error using a loss function, then backpropagates those errors to update the model’s weights. This can take weeks and cost millions in compute.
AI fine-tuning adapts a pre-trained model to a narrower task using a smaller dataset. Think of it as a specialist course after general education. It costs far less than training from scratch.
AI inference is the execution phase. The trained model takes in live, real-world data and generates a response in a single forward pass. No weight updates, no learning. Just pattern matching against everything it already knows.
A single inference is cheap compared to training, but serving millions of them in real time is where the real infrastructure challenge lives. Here is the part most people get wrong: inference is not the lightweight afterthought. For an LLM operating at scale, inference compute dwarfs training compute over the model’s lifetime. Training happens once. Inference happens billions of times.
How does AI inference work?
Every inference call follows three steps:
First, the input gets prepared. Raw user text gets broken into tokens, the atomic units the model was trained on. A question like “What is AI inference?” might become four or five tokens. The model is picky about input format. Wrong dimensions, wrong tokenizer, and nothing works.
Second, the model executes a forward pass. Each layer of the neural network applies its learned transformations, passing the result to the next layer. A modern LLM might run through dozens of layers, each one chewing on the output of the last. There is no backward pass here, no error correction. The weights stay frozen.
Third, the model generates output tokens one at a time. For an autoregressive model, each predicted token gets fed back as input for the next prediction, iterating until the model produces an end-of-sequence token or hits a length limit. This is why longer responses cost more and take longer.
Token generation is where latency bites hardest. A GPU can compute a single forward pass in milliseconds, but generating a 500-token response means running 500 sequential forward passes. You cannot parallelize this. Each token depends on the previous one.
Types of AI inference
The way you run inference depends entirely on how fast the answer needs to arrive and how much data you are pushing through.
Real-time inference
Real-time inference processes individual requests immediately, returning results in milliseconds. This is what powers chatbots, live translation, fraud detection systems, and product recommendation engines.
The engineering tradeoff is straightforward: speed costs money. You keep GPUs warm and idle, waiting for requests, because any cold-start delay would break the user experience. For a customer-facing chatbot, 200ms feels snappy. 2,000ms feels broken.
Batch inference
Batch inference collects inputs over time and processes them in bulk, typically on a schedule. An insurance company running nightly fraud checks on the previous day’s claims is doing batch inference. A streaming service refreshing all user recommendations every six hours is doing batch inference.
Batch wins on cost. You can pack inputs densely onto GPU memory, maximizing throughput per dollar. The obvious tradeoff: results arrive minutes or hours after the input was submitted. Fine for overnight reports. A problem for anything that needs an answer now.
Streaming inference
Streaming inference sits between real-time and batch. The model receives a continuous flow of data, not discrete prompts, and updates its outputs incrementally as new data arrives. This is common in industrial monitoring, sensor networks, and real-time video analysis where the “input” never really stops.
Unlike a chatbot that processes one prompt at a time, streaming inference systems maintain persistent state and update predictions as conditions change. A factory floor sensor network checking for equipment anomalies every few seconds is streaming inference. There is no clear beginning or end to the request.
What is micro-batching?
Micro-batching is a clever middle ground between online and batch inference. Instead of waiting to fill a full batch, the system groups a handful of requests, typically four to eight, and processes them together every few milliseconds.
The goal is to capture most of the throughput gains of batching while keeping latency low enough that individual users do not notice the delay. Major LLM platforms use micro-batching under the hood. When thousands of users prompt ChatGPT simultaneously, the inference server is almost certainly grouping those requests into micro-batches behind the scenes.
AI inference types at a glance
| Type | Latency | Best for | Key tradeoff |
|---|---|---|---|
| Real-time | Milliseconds | Chatbots, fraud detection, live translation | Higher cost per request |
| Batch | Minutes to hours | Nightly reports, periodic analytics, recommendation refreshes | Cheapest per prediction; no real-time capability |
| Streaming | Near-real-time | Sensor monitoring, video analytics, industrial IoT | Complex state management |
| Micro-batching | Low milliseconds | High-volume LLM serving, API endpoints | Slightly more complex orchestration |
Where does AI inference run?
The physical location of the hardware matters enormously for latency, privacy, and cost. Here are the four environments where inference workloads live:
- Cloud inference is the default for most organizations. Remote servers in data centers handle the compute, and you pay per request or per GPU-hour. The cloud offers near-infinite scalability. If your traffic spikes 10x overnight, a cloud provider absorbs that. Your on-prem rack cannot.
- Edge inference moves compute closer to where data originates. A security camera running object detection on-device, rather than streaming video to the cloud for analysis, is doing edge inference. No network round trip means sub-millisecond responses, and sensitive data never leaves the device.
- On-device inference takes this further. Smartphones running speech-to-text locally, camera apps applying real-time filters, face ID unlocking your phone. The model lives on the device itself, limited by whatever compute the phone can provide. These models tend to be smaller, quantized, and optimized for their specific task.
- On-premise deployment means running inference on hardware you own, in a facility you control. Financial institutions and healthcare organizations often choose this route because regulations require it. You get full data sovereignty but shoulder the capital cost of GPUs that may sit idle during off-peak hours.
Hardware that powers AI inference
GPUs dominate inference workloads because they are built for the kind of math neural networks need: massive matrix multiplications running in parallel across thousands of cores. NVIDIA’s CUDA platform, which the company describes as “the software layer that enables applications to harness the power of GPUs,” let developers write code that runs directly on those cores, and the ecosystem has been compounding ever since.
But GPUs are not the only player.
TPUs (Tensor Processing Units) are Google’s custom-developed application-specific integrated circuits (ASICs), designed specifically for neural network operations. They trade the general-purpose flexibility of a GPU for raw speed on matrix math. If your model runs in TensorFlow, PyTorch, or JAX, TPUs can offer better performance per watt than comparable GPUs, though you give up the broad software ecosystem.
NPUs (Neural Processing Units) appear in smartphones and edge devices. Tuned for low-power inference on compact models, an NPU can run facial recognition continuously without draining your battery. Something a GPU would struggle with in a thermal-constrained phone chassis.
FPGAs (Field-Programmable Gate Arrays) are configurable circuits that can be reprogrammed for specific workloads. They offer less raw throughput than top-tier GPUs but allow precise customization. A trading firm might use FPGAs for inference because they can tune the hardware pathway to squeeze out every microsecond of latency.
ASICs are custom chips built for one task and one task only. TPUs are a type of ASIC. Maximum efficiency at the target workload, zero flexibility if your model architecture changes.
How to speed up AI inference
Distributed inference
When a single GPU cannot hold your entire model in memory, you distribute the work across multiple devices. Three main strategies exist.
Data parallelism copies the full model onto each GPU, then splits the input batch across them. Each GPU processes a shard independently. Simple, but every GPU must be big enough to fit the entire model. For a 70-billion-parameter LLM, that is a problem.
Tensor parallelism splits individual layers across GPUs. Each device holds a portion of the weight tensor and processes a corresponding slice of the input. Memory requirements drop dramatically per GPU, but the devices need to communicate constantly to stitch outputs back together.
Pipeline parallelism assigns different layers to different GPUs. GPU 1 handles layers 1 through 8, GPU 2 handles 9 through 16, and so on. Data flows sequentially through the pipeline. The trick is keeping all GPUs busy simultaneously using mini-batches, so GPU 1 starts on the next batch while GPU 2 is still finishing the current one.
What vLLM does
vLLM is an open-source inference engine that speeds up LLM serving by solving a specific memory problem. Traditional inference frameworks reserve GPU memory for the maximum possible sequence length for every request, even if most requests are short. That pre-allocated but unused memory is called fragmentation.
vLLM uses a technique called PagedAttention that allocates memory dynamically in smaller blocks, similar to how operating systems manage virtual memory with pages. The approach lets far more requests share the same GPU. According to research from Anyscale, continuous batching in vLLM enables 23x throughput in LLM inference while reducing p50 latency.
Mixture of Experts architecture
Mixture of Experts (MoE) is a model design that speeds up inference by activating only a fraction of the model’s total parameters for any given input. An MoE model might have a trillion parameters total but only 10% of them fire on any single forward pass. The model learns to route different types of inputs to different expert sub-networks.
This means MoE models can be dramatically larger than dense models while maintaining similar inference latency. A dense 70B model uses all 70B parameters on every token. An MoE model with eight experts and 70B total parameters might use only 12B per token, giving you roughly the speed of a 12B model with the capability of a much larger one.
Solving the cold start problem
When an inference server receives a request for a model that is not currently loaded in GPU memory, it must load the model weights from disk first. For a 70B parameter model, that can take minutes. Every one of those minutes is a request sitting in a queue, or worse, timing out.
Techniques to mitigate cold starts include model pre-loading on standby instances, GPU memory swapping that pages inactive model sections to host memory, and model streaming tools that load weights concurrently across multiple I/O threads. Some orchestration platforms now pre-warm GPU instances based on predicted demand patterns, so a model is already memory-resident before the first request arrives.
Real-world AI inference use cases
AI inference shows up in more places than most people realize.
Fraud detection systems run inference on every transaction as it happens. A credit card swipe in London at 3pm followed by another in Tokyo at 3:05pm triggers a real-time inference that flags the second transaction as suspicious. The model did not learn about geography explicitly. It learned from billions of past transactions that this pattern correlates with fraud.
Healthcare diagnostics use inference to compare patient scans against patterns learned from millions of prior cases. A radiologist reviewing a chest X-ray might see an AI-generated heatmap highlighting a region the model scored as anomalous. The radiologist decides. The inference suggests.
Autonomous vehicles run continuous inference across multiple sensor streams simultaneously. Cameras, lidar, and radar each feed into separate models that detect objects, predict trajectories, and plan routes, all in the time it takes a car to travel a few inches at highway speed. This is edge inference at its most demanding.
Recommendation engines and chatbots represent the most visible inference workloads. Every time Netflix suggests a show or ChatGPT generates a paragraph, an inference call happened. These systems may feel lightweight from the outside, but the infrastructure required to serve millions of simultaneous inference requests is the same class of problem that drives GPU shortages and multi-billion-dollar data center builds.
Frequently asked questions
What is the difference between AI inference and AI training?
Training teaches a model to recognize patterns by iteratively updating its weights on a labeled dataset. Inference uses the trained model to make predictions on new data without modifying the model. Training is expensive and infrequent. Inference is cheap per call but dominates total compute cost over a model’s lifetime. This is a core concept covered in machine learning fundamentals.
Does AI inference require GPUs?
Not strictly. CPUs can run inference for small models. But any model large enough to be useful in production performs orders of magnitude better on GPUs or specialized accelerators. The matrix multiplications at the heart of neural networks map naturally to GPU parallelism.
What is the difference between cloud inference and edge inference?
Cloud inference runs on remote servers accessed over the internet. Edge inference runs on devices physically near the data source. Cloud wins on scalability and raw compute. Edge wins on latency and privacy. Most production systems use a mix: edge for latency-sensitive tasks, cloud for heavy lifting.
How much does AI inference cost?
Costs vary wildly depending on model size, request volume, and hardware. Running inference on a small classification model might cost fractions of a cent per thousand predictions. Running GPT-4 class inference at scale can cost millions per month. The dominant cost driver is GPU-hours. For a detailed breakdown, see this LLM token pricing guide that maps out per-token costs across models.
What is test-time compute?
Test-time compute is the practice of programming a model to generate intermediate reasoning tokens before delivering its final output. It has been empirically shown to improve performance on logical benchmarks without altering the model’s underlying weights. This is why models like OpenAI o1 and DeepSeek-R1 produce chain-of-thought reasoning before answering.
What comes after understanding inference
AI inference is where models earn their keep. Training is the investment. Inference is the return.
If you want to go deeper on the hardware side, the guide to what TPUs are and how they compare to GPUs breaks down the accelerator landscape. If you are thinking about operationalizing inference at scale, MLOps covers the full deployment lifecycle, from model packaging to monitoring. And if you are concerned about protecting these inference pipelines, AI security best practices addresses the risks that come with running models in production.