Anatomy of an LLM request: prefill, decode, latency, and cost

TL;DR

An LLM request has two phases that dominate latency and cost: prefill (processing the input) and decode (generating the output one token at a time). Prefill is compute-bound and scales with input length. Decode is memory-bandwidth-bound and scales with output length.

Total latency follows:

Total latency ≈ TTFT + (output_tokens × TPOT)

Output tokens cost 3-5x more than input tokens on commercial APIs. The biggest cost wins come from request design (shorter outputs, prefix caching, smaller RAG chunks), not infrastructure.

For interactive chat, target TPOT under 100ms (10 tokens/sec). Self-hosted LLM serving is cheaper than API at sustained utilisation above approximately 30%. On most LLM workloads the token bill ends up larger than the Kubernetes bill underneath it, which is what makes serving efficiency a cost lever rather than an engineering nicety.

What is a token in an LLM?

A token is a chunk of text from a model's fixed vocabulary, typically 50,000 to 150,000 unique pieces. Models do not process characters or words directly. The tokenizer splits input text into tokens before the model sees anything.

Token-to-text conversion rates for English

• 1 token ≈ 4 characters
• 1 token ≈ 0.75 words
• 100 tokens ≈ a short paragraph
• 1,000 tokens ≈ one page of prose

Non-English languages, code, and JSON tokenize less efficiently. Chinese text typically uses 2-3x more tokens than equivalent English. JSON with many field names uses more tokens than plain prose.

Why tokenization matters for production

Every operational metric (context limits, API pricing, latency, KV cache size) is measured in tokens, not characters. Estimating in characters will mislead capacity planning.

Tokenizers also vary by model family. The same text might be 1,200 tokens for Llama and 1,400 for GPT-4. Switching models in production requires re-doing the token budget math.

What is the context window in an LLM?

The context window is the total number of tokens a model can attend to in one request. It is a shared budget across all components of a prompt.

A 32,000 token context window must accommodate:

• System prompt (instructions, persona)
• Tool / function definitions
• Few-shot examples
• Retrieved RAG chunks
• Conversation history
• The current user message
• Reserved space for the response

Why large context windows are not always useful

Three constraints limit useful context length:

Models attend reliably to the start and end of long contexts but get fuzzy in the middle. This is the "lost in the middle" effect, documented by Liu and colleagues in 2023 in Lost in the Middle: How Language Models Use Long Contexts.

• Cost scales linearly or worse. Doubling input tokens at least doubles prefill cost.
• Latency scales linearly. 32k of input takes approximately 8x longer to prefill than 4k.

Most production systems use 4k to 32k of context even on models that support 200k+ tokens.

What are the phases of an LLM request?

An LLM request passes through five phases. Two of them dominate latency and cost.

What is prefill in LLM inference?

Prefill is the phase where the model processes all input tokens together in one parallel matrix multiplication. It is compute-bound: the GPU is doing serious arithmetic at full utilisation.

Properties of prefill

• Scales linearly with input length. 10,000 input tokens take approximately 10x longer than 1,000.
• Compute-bound. GPU TFLOPS (compute throughput) is the bottleneck.
• Determines TTFT. The user sees no response until prefill completes.
• Hardware matters. H100 is significantly faster than A100 on prefill due to higher TFLOPS.

Prefill latency for common configurations

Prefill time can be hidden by streaming, but only the first token; users still wait for prefill to complete before seeing any response.

What is decode in LLM inference?

Decode is the phase where the model generates output tokens one at a time. Each token requires reading the entire model weights and KV cache from VRAM, performing a small computation, and producing one new token.

Properties of decode

• Scales linearly with output length. 500 output tokens take approximately 5x longer than 100.
• Memory-bandwidth-bound. The GPU is mostly waiting on VRAM reads, not computing.
• Benefits from batching. Multiple users decoding in parallel share the model weight reads.
• Determines TPOT. The gap between consecutive streamed tokens.

Decode latency targets

Human reading speed averages approximately 250 words per minute, or roughly 5 tokens per second. TPOT faster than this is perceived as instant.

Why are prefill and decode bottlenecked by different hardware?

Prefill processes thousands of tokens in parallel: the GPU's compute units are saturated. The bottleneck is TFLOPS (floating-point operations per second).

Decode processes one token at a time per user. The GPU's compute units are mostly idle; the bottleneck is how fast model weights can be read from VRAM. The bottleneck is memory bandwidth (GB/sec).

This means:

• H100 vs A100 on prefill: H100 is approximately 3x faster (more TFLOPS).
• H100 vs A100 on decode: difference is smaller (memory bandwidth gap is smaller).

The right GPU for a workload depends on whether the traffic is "long prompts, short outputs" (prefill-dominated) or "short prompts, long outputs" (decode-dominated).

How is total LLM request latency calculated?

Total latency is the sum of prefill time (TTFT) and the cumulative decode time across all output tokens:

Total latency ≈ TTFT + (output_tokens × TPOT)

Worked example

An 8,000 token input, 500 token output, on a Qwen3.5-9B model running on an L4:

• TTFT (8k prefill): ~1.5 seconds
• TPOT: ~25 ms/token
• Output time: 500 × 0.025 = 12.5 seconds
• Total latency: ~14 seconds

In this example, decode dominates total latency. For workloads with short outputs (classification, extraction), prefill dominates. Optimisation strategies differ for each case.

What are TTFT and TPOT in LLM serving?

TTFT (Time To First Token): The latency from request sent to first response token received. Dominated by prefill (input length).

TPOT (Time Per Output Token): The gap between successive output tokens during streaming. Also called ITL (Inter-Token Latency). Dominated by decode (memory bandwidth and batch contention).

Why track P50, P95, and P99 separately

Averages hide tail latency. A system with average latency of 2 seconds may have P99 latency of 30 seconds, meaning 1% of users experience a 30 second wait. This is why SRE practice tracks percentile distributions rather than means for user-facing services.

Production LLM systems should track P50, P95, and P99 for TTFT, total latency, and tokens per response.

What is the throughput vs latency trade-off in LLM serving?

LLM serving has one fundamental trade-off: per-user latency versus aggregate throughput. The variable is batch size.

When to optimise for low latency

• Consumer chat applications
• Interactive coding assistants
• Anything where a single user is waiting

When to optimise for throughput

• Batch document processing
• Async pipelines
• Bulk content generation
• Embedding generation

Both cannot be maximised simultaneously. The right choice depends on the workload.

What types of caching are used in LLM serving?

Four distinct mechanisms are commonly called "caching" in LLM systems. They are not equivalent.

Which caching strategy provides the largest gains

Prefix caching produces the largest gains when prompts share a stable prefix (system prompt, tool definitions, RAG context). It can reduce TTFT by 5-20x with zero code changes. Enable with --enable-prefix-caching in vLLM.

Anthropic prices cached input reads at 0.1x the base input rate, so a cached prefix costs 90% less than sending it again. The mechanism is the same as vLLM's prefix caching, exposed through the API.

Semantic caching saves entire LLM calls but risks serving stale responses to similar but distinct queries. Best for FAQ-style workloads with bounded query distributions.

How is LLM cost calculated for API usage?

Commercial LLM APIs charge separately for input and output tokens. Output is typically 3-5x more expensive than input.

Why output costs more than input

Input tokens are processed in one parallel prefill batch (compute-efficient). Output tokens are generated sequentially in decode (cannot parallelise within one user's response).

Worked example: RAG chatbot cost

Using GPT-4-class pricing ($3 per million input tokens, $15 per million output tokens):

Cost per turn:

• Input: 10,200 × $3/M = $0.0306
• Output: 500 × $15/M = $0.0075
• Total: ~$0.038 per turn

At 100,000 turns per day: $3,800 per day, or approximately $115,000 per month.

In this example, input dominates cost because of large RAG context. Halving RAG chunks saves approximately $1,500 per month from one configuration change.

How is LLM cost calculated for self-hosted serving?

Self-hosted serving uses GPU-hours instead of token pricing:

cost_per_token = GPU_$/hr ÷ aggregate_throughput_tokens_per_hour

Worked example

An L4 spot instance at $0.30/hr, sustaining 500 tokens/sec aggregate throughput:

• Throughput: 500 tokens/sec × 3,600 sec/hr = 1.8M tokens/hr
• Cost per token: $0.30 / 1,800,000 = $0.00000017 (or $0.17 per million tokens)

This is approximately 17x cheaper than GPT-4o input pricing and 90x cheaper than GPT-4o output pricing, provided utilisation is high.

When self-hosting wins on cost

Self-hosted LLM serving is cheaper than commercial APIs when sustained utilisation exceeds approximately 30%. Below this threshold, idle GPU time exceeds API token cost.

Above 60% sustained utilisation, self-hosting is typically 10-50x cheaper per token than commercial APIs. When LMSYS moved Vicuna serving onto vLLM it cut its GPU count by half while serving 2-3x more requests per second, which is what high-utilisation self-hosting economics look like in practice.

What latency targets should LLM applications meet?

Streaming should be enabled for any user-facing interactive use case. It does not change total latency but reduces perceived latency dramatically.

How can LLM request cost and latency be reduced?

In order of impact:

• Reduce output length. Output costs 3-5x input. Prompting for concise responses can cut total cost by 30-50%.
• Enable prefix caching. Free win for any workload with stable prefixes. 5-20x faster TTFT.
• Reduce RAG context. Fewer or smaller chunks reduce input cost per request.
• Use a smaller model where quality permits. A 9B model for synthesis may match a 70B at 10x lower cost.
• Quantize. FP8 or AWQ-4bit reduces memory 2-4x with minor quality loss.
• Cache at the application layer. Semantic or exact-match caching skips LLM calls for repeated queries.

Key terms

Token: Chunk of text from the model's vocabulary. Approximately 4 characters or 0.75 words in English.

Context window: Total tokens a model can attend to in one request, shared across system prompt, history, RAG, and response.

Prefill: Phase where the model processes all input tokens in parallel. Compute-bound. Sets TTFT.

Decode: Phase where the model generates output tokens one at a time. Memory-bandwidth-bound. Sets TPOT.

TTFT (Time To First Token): Latency from request sent to first response token received.

TPOT (Time Per Output Token): Gap between consecutive output tokens during streaming.

ITL (Inter-Token Latency): Synonym for TPOT.

Streaming: Sending output tokens to the client as generated, rather than waiting for full completion.

Throughput: Aggregate tokens per second across all in-flight requests.

Batch size: Number of requests processed concurrently. Trades off per-user latency for aggregate throughput.

Prefix caching: Reusing KV cache blocks across requests with shared starting prefixes.

Prompt caching: API-exposed prefix caching (Anthropic, OpenAI).

Semantic caching: Application-layer caching based on prompt similarity, not exact match.

Frequently asked questions

About NudgeBee

NudgeBee builds AI-powered SRE automation for cloud-native production systems. We use LLMs at scale and have made most of the mistakes documented in this series. Learn more at nudgebee.com.

Series navigation

• Part 1: How to serve LLMs in production: GPU memory, KV cache, and sizing
• Part 2: Anatomy of an LLM request
• Part 3: How to build production applications on LLMs: RAG, agents, and tool calling
• Part 4: LLM serving cheat sheet
• Part 5: Common mistakes when serving LLMs in production