How LLMs Do Inference

Introduction

You type a question into ChatGPT or Claude, and almost instantly, words start appearing on your screen. It feels like magic, or perhaps like a human typing on the other end. But what's actually happening deeply inside the silicon chips of the server?

The process is called inference. It's a complex dance of mathematics, memory management, and massive parallel computation. Roughly speaking, the model is playing a very sophisticated game of "guess the next word," over and over again, billions of times a day.

In this interactive explainer, we'll peel back the layers. We'll trace the journey of a prompt as it gets sliced into numbers, processed by giant matrices, and converted back into human language.

Step 1: Tokenization

Computers don't understand words; they understand numbers. Before an LLM can do anything effectively, your text input must be broken down into smaller chunks called tokens.

Modern LLMs use Byte Pair Encoding (BPE), which starts with individual characters and repeatedly merges the most common pairs. This creates a vocabulary of subwords that balances efficiency with flexibility.

Step 2: Embeddings

Each token ID is converted into a dense vector of numbers called an embedding. These vectors live in a high-dimensional space where similar words cluster together.

The embedding captures semantic meaning: "cat" and "dog" will be closer to each other than "cat" and "computer". This is where the model's understanding of language begins.

The Missing Piece: Position

Here's a problem: if we just fed these embeddings into the attention mechanism, "The cat ate the fish" and "The fish ate the cat" would look identical! The model wouldn't know which word came first.

To fix this, we add a Positional Encoding vector to each word's embedding. By using sine and cosine waves of different frequencies, we give every position in the sequence a unique mathematical "fingerprint" that the model can learn to recognize.

Step 3: The Transformer Block

The heart of an LLM is the Transformer, a neural network architecture invented in 2017. It consists of stacked layers, each containing attention mechanisms and feed-forward networks.

A modern LLM like GPT-4 or Llama-3 might have 80+ of these layers, each refining the model's understanding of the input and building toward the final prediction.

Inside the Transformer

Once the input is tokenized and embedded, it enters the Transformer layers. This is where the "intelligence" lives. A modern LLM might have dozens (or even roughly 100) of these identical blocks stacked on top of each other.

The key mechanism here is Self-Attention. It allows each token to "look at" other tokens in the sequence to gather context. For example, in the sentence "The animal didn't cross the street because it was too tired," attention helps the model understand that "it" refers to "the animal" and not "the street."

Multi-Head Attention

Rather than having a single attention mechanism, transformers use multiple attention heads that each learn to focus on different types of relationships.

One head might specialize in grammatical dependencies (subjects and verbs), another in semantic similarity (synonyms and related concepts), and another in positional patterns (nearby words). The model combines all these perspectives for a richer understanding.

The Two Phases: Prefill & Decode

Generation isn't just one smooth process; it actually happens in two distinct phases that behave very differently under the hood.

• Prefill (The "Reading" Phase): The model processes your entire prompt at once. This is highly parallelizable and usually very fast. It's like reading a whole page of a book in a glance to understand the context.
• Decode (The "Writing" Phase): The model generates the response one token at a time. Each new token depends on all the previous ones (including the prompt and what it just generated). This is sequential and harder to speed up.

Training vs. Inference

It's crucial to distinguish between training (teaching the model) and inference (using the model).

Step 4: Sampling

After passing through all the layers, the model produces a set of scores (logits) for every possible token in its vocabulary. These scores represent the likelihood of each token being the next one.

But the model doesn't just always pick the most likely word (that would be boring and repetitive). Instead, strategies like Temperature, Top-P, Top-K, and Min-P introduce controlled randomness to make the text feel more creative and human-like.

Decoding Strategies: Beam Search

Sampling adds randomness, but what if we want the best possible sentence? Simply picking the most likely word at each step (Greedy Decoding) can lead to dead ends.

Beam Search explores multiple parallel futures simultaneously. It keeps the "top K" most promising incomplete sentences at every step, allowing it to find better overall sequences that might start with a lower-probability word.

Optimization: The KV Cache

Generating one token at a time sounds like it could be incredibly slow if we had to re-read the entire history for every single new word. Fortunately, engineers use a clever trick called the KV (Key-Value) Cache.

Instead of recalculating the attention keys and values for all previous tokens at every step, the model saves them in GPU memory. This means for each new step, it only needs to compute the values for the newest token. The trade-off? It uses a huge amount of memory!

Throughput: Continuous Batching

Serving one user at a time is inefficient because GPU cores are massive parallel machines. To fix this, we process multiple requests in a batch.

But what if one request finishes early? In a naive batch, the GPU waits for the slowest request. Continuous Batching (or cellular batching) solves this by immediately ejecting finished sequences and inserting new ones into the free slots, keeping the GPU 100% busy.

Real-World Inference: Quantization

In production, engineers struggle with the Memory Wall. The weights of a model like Llama-3-70B take up ~140GB of VRAM. Moving that data from memory to the compute units is often the bottleneck.

To solve this, we use Quantization: reducing the precision of the model's numbers from high-definition (16-bit) to lower fidelity (8-bit or 4-bit). It's like lowering the bitrate on a video file—you save massive amounts of space with surprisingly little loss in quality.