LLM latency optimisation for startups comes down to a short list of levers, in rough order of payoff: generate fewer output tokens, make fewer sequential calls, run independent steps in parallel, stream the answer, cache repeated prompt prefixes and pick a faster model or service tier for the steps that are user-facing. OpenAI's own latency guide puts the biggest lever bluntly: cutting 50% of output tokens may cut about 50% of latency, while cutting 50% of the prompt may improve it by only 1 to 5%. Measure first, then work down the list, and judge success by time to first token and p95 latency on real user journeys rather than averages.

Where the waiting time actually goes

A request to a hosted model passes through several stages: network round trip, any queueing at the provider, processing of the input (often called prefill), token-by-token generation, and whatever your application does before and after. Each extra model call in a chain repeats most of those costs.

The single most important fact about that pipeline is that generation is sequential. Every output token depends on the one before it, so long answers are slow answers. OpenAI's latency optimisation guide states that "generating tokens is almost always the highest latency step when using an LLM", and offers the heuristic above: halve the output and you may roughly halve the latency. Input tokens are processed in parallel and matter far less, which is why trimming a system prompt rarely transforms response times on its own, though it does cut cost and can help caching.

The same guide groups every technique into seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelise, make your users wait less, and do not default to an LLM. That last one deserves more attention than it gets. A regular expression, a database lookup or a cached template answers in milliseconds.

Measure the right numbers first

Anthropic's guide to reducing latency defines the two terms most teams need. Baseline latency is the time the model takes to process the prompt and generate a response. Time to first token (TTFT) is the time from sending the prompt until the first token arrives, which it calls "particularly relevant when you're using streaming".

For a startup, a workable measurement set per user journey looks like this:

  • TTFT, which governs how responsive a streaming interface feels.
  • Total time to a usable result, including every chained call, tool call and validation step.
  • Output tokens per request, because this drives generation time directly.
  • p95 and p99 latency, since averages hide the slow tail that users remember.
  • Retry and 429 counts, because a retried request is a slow request.

Tie these to journeys rather than endpoints. A drafting assistant can take several seconds if text starts flowing at once; an autocomplete feature cannot. Jakob Nielsen's long-standing response time limits still give a useful frame: about 0.1 second feels instantaneous, about 1 second keeps the user's flow of thought uninterrupted, and about 10 seconds is the limit for keeping attention on the task.

Cut output tokens and sequential calls

Since output dominates, start there. Ask for the format you need and nothing more: short field names in structured output, no restated question, no closing summary. Set a hard ceiling with max_tokens, which Anthropic's guide recommends as a way to prevent "overly long outputs". OpenAI's worked example shortens JSON keys and moves explanations into code comments purely to generate fewer tokens.

Next, count the round trips. OpenAI's guide notes that each request incurs round-trip latency and suggests combining sequential steps into one prompt that returns named fields in a JSON object. Its customer-service walkthrough merges query rewriting and a retrieval check into a single call. The opposite move also appears: splitting one large prompt in two so that a smaller, faster model can handle part of it. Which wins depends on your token counts, so test both.

Where steps do not depend on each other, run them at once. Retrieval, a moderation check and a classification call can usually be fired in parallel rather than one after another. The guide also describes speculative execution in application code: guess the result of an early step, start the later step anyway, and keep the work if the guess was right.

For edits to existing text or code, OpenAI's Predicted Outputs feature lets you pass the expected output so that unchanged sections are produced faster; the documentation lists support for GPT-4o and the GPT-4.1 family (checked October 2026).

Streaming and perceived speed

Streaming does not shorten total generation, but it changes the experience. OpenAI's guide calls streaming "the single most effective approach" to making users wait less, "as it cuts the waiting time to a second or less." Both major APIs stream over server-sent events, a standard browser technology; Anthropic's streaming documentation shows "stream": true delivering text, tool use and thinking deltas incrementally.

Structured output needs more care. If the client needs valid JSON before it can render anything, raw tokens are useless to it. Options include streaming a short natural-language status while the structured call runs, splitting the response so a readable part streams first, or chunking output so processed pieces can be displayed as they complete, which the OpenAI guide also suggests. Always give users a visible stop control on long generations.

Prompt caching and application caching

Provider-side prompt caching reuses work on an unchanged prompt prefix. OpenAI's prompt caching documentation explains that a later request with the same prefix lets the model "reuse the saved state instead of processing those tokens again." As of October 2026, it lists a minimum cacheable prefix of 1,024 tokens for its newer models, and cache reads on most models cost 0.1 times the normal input rate. Anthropic's prompt caching uses a default five-minute cache lifetime, refreshed free each time the cache is hit, with a one-hour option at extra cost; cache reads are priced at 0.1 times base input for most models (checked October 2026).

The architectural consequence is simple. Put stable content first (system instructions, tool definitions, reference documents) and variable content last, so the prefix stays identical across requests. A timestamp or user name at the top of a system prompt quietly defeats the cache.

Application caching is the layer you own. Exact-match caches suit deterministic tasks such as classification or extraction on identical inputs. Caches for retrieval results and embeddings save both a model call and a database query. Semantic caches, which return a stored answer for a similar but not identical question, are the riskiest: test false matches on real queries and keep them away from personalised, priced or fast-changing answers. Set expiry by how quickly the underlying data changes.

Model choice, service tiers and self-hosting

A smaller model generates faster, so route simple steps (classification, extraction, short rewrites) to one and keep the larger model for steps that need it. Anthropic's latency guide also points to reasoning effort settings on its current models, where lower effort is the faster option. Build a small human-reviewed test set and choose the fastest model that passes it.

Several providers now sell speed directly, and the details change often (checked October 2026). OpenAI's Fast mode advertises "up to 2.5× faster speeds and more consistent latency with pay-as-you-go pricing" for selected models. AWS offers latency-optimised inference on Amazon Bedrock, still marked as a preview, enabled by setting the latency parameter to "optimized" for a limited set of models and regions. Work that is not user-facing can leave the hot path entirely: OpenAI's Batch API offers a 50% discount, separate higher rate limits and a 24-hour completion window.

If you self-host open models, serving software matters as much as hardware. The vLLM paper on PagedAttention (Kwon and colleagues, September 2023) reported throughput gains of 2 to 4 times at the same latency compared with systems such as FasterTransformer and Orca. Speculative decoding, by Yaniv Leviathan and colleagues and presented as an oral paper at ICML 2023, uses a small draft model to propose tokens that the large model verifies, and showed a 2 to 3 times speed-up on T5-XXL with identical outputs. Neither is free to operate, so most startups should exhaust the application-level levers first.

Rate limits, spikes and fallbacks

At peak load, queueing and throttling can dwarf model speed. Provider limits are measured in units such as requests per minute and tokens per minute, and OpenAI's rate limit guide explains that exceeding a temporary limit returns a 429 error, sometimes with a Retry-After header. Its advice: honour that header when present, otherwise use exponential backoff with jitter, cap both attempts and total retry time, and avoid nested retry loops between your code and the SDK.

On the product side, decide in advance how each journey degrades under pressure: a faster model, a shorter answer, a queued job with a notification, or an honest "try again shortly". A fallback model is useful for genuine failures, but if it fires often it becomes your real latency profile, so track how frequently it triggers. George Bregman writes about running AI operations in young companies on georgebregman.com, and latency budgets are a good example of an operational number worth reviewing alongside cost and quality.

FAQ

How do I make LLM responses faster?

Shorten outputs first, then remove or parallelise sequential calls, stream the response, structure prompts so the provider can cache the prefix, and route simple steps to a smaller model. Measure TTFT and p95 per journey before and after each change.

Does a shorter prompt reduce latency?

Only a little. OpenAI's guide estimates that cutting 50% of the prompt may improve latency by just 1 to 5%. It still lowers cost and can make caching easier, but output length and the number of calls usually matter far more.

Is self-hosting faster than an API?

Sometimes, with the right serving stack such as vLLM, but it brings capacity planning and operational work. For most early products, hosted APIs plus caching, streaming and faster tiers deliver the bigger gains for the effort.