Choosing an LLM for your product means writing down what the feature must do, testing two or three candidate models on real examples against your own scoring rules, and picking the cheapest one that clears the bar on accuracy, latency, output format and data handling. Public leaderboards help you draw up the shortlist; they do not make the decision. Keep the model behind a thin layer in your code, because the lineups and prices you choose from today will change within months.

Write the job description before the shortlist

Start with one sentence describing the work: "draft a reply to a support ticket using our help centre", "pull ten fields out of a supplier invoice", "classify an incoming lead". Then set success criteria you can measure. Anthropic's guide to defining success criteria and building evaluations says good criteria are specific, measurable, achievable and relevant, and gives the example of a sentiment classifier that must reach an F1 score of at least 0.85 on a held-out test set. A target like that turns "pick the best model" into a test you can pass or fail.

The same guide lists criteria worth considering beyond raw accuracy: consistency between similar inputs, relevance and coherence, tone and style, privacy preservation, context utilisation, latency and price. Most products care about four or five of these. Rank them, because the trade-offs between them drive the final choice.

Know what the current lineups look like

Every major provider now sells a tiered family rather than a single model, and the tiers differ far more in price than in context length (all figures checked October 2026).

  • OpenAI. Its model catalogue recommends GPT-6 Astra for complex reasoning and coding at $10 per million input tokens and $50 per million output tokens, GPT-6.1 Sol at $2 and $10 to balance intelligence and cost, and GPT-6 Luna at $0.10 and $0.50 for high-volume work. All three list a context window of 1.05 million tokens.
  • Anthropic. The Claude models overview lists Claude Fable 5.1 ($10 and $50), Claude Opus 5.5 ($4 and $20), Claude Sonnet 5.5 ($2 and $10) and Claude Haiku 5.5 (from $0.10 and $0.50), each with a 1 million token context window and 128,000 maximum output tokens. Anthropic suggests starting with Opus 5.5 for most workloads.
  • Google. The Gemini API model list describes Gemini 4 Argon as its flagship reasoning model with a 1 million token context window, alongside several Gemini 3 series Flash and Flash-Lite models for cheaper work.

The spread between the top and bottom tiers is roughly a hundredfold. That gap, not the difference between vendors at the same tier, is where most of your cost decision sits.

Why leaderboards only shortlist

Public rankings measure something real, just not your product. The Chatbot Arena paper by Wei-Lin Chiang and colleagues describes ranking models through crowdsourced pairwise comparisons, and reported more than 240,000 votes at the time of writing. That tells you which model a broad set of users preferred on their own prompts. It says little about how a model handles your invoices, your tone of voice or your edge cases.

OpenAI's evaluation best practices make the same distinction. It separates industry benchmarks such as MMLU from "specific tests you implement to measure your LLM application's performance", and lists "overly generic metrics" and eval sets that do not reproduce production traffic as anti-patterns. Use the leaderboard to choose which two or three models to test, then stop reading it.

Long inputs deserve their own test. The paper Lost in the Middle by Nelson Liu and colleagues found that performance "is often highest when relevant information occurs at the beginning or end of the input context" and drops when the answer sits in the middle, even for models built for long contexts. A headline context window of a million tokens tells you what fits, not what the model will reliably use.

Build a small evaluation harness first

An evaluation harness is a set of real inputs, an expected answer or scoring rule for each, and a script that runs every candidate and records the results. Pull inputs from logs, support tickets or customer documents (with the right permissions), and deliberately include awkward cases. Score the first round by hand so you learn what good output looks like, then automate the checks you trust.

You do not need to write the plumbing from scratch. The UK AI Security Institute and Meridian Labs maintain Inspect, an open-source evaluation framework that works across providers and ships implementations of more than 200 benchmarks (checked October 2026). If you planned to rely on OpenAI's hosted Evals platform, note that its documentation says the platform becomes read-only for existing users on 31 October 2026 and shuts down on 30 November 2026. Keep your test cases in files you own so the harness survives any vendor's product decisions.

The harness pays for itself twice: once when you choose, and again every time a new model is released or an old one retires and you need to know whether switching is safe.

Weigh cost per task, latency and output format

Price per million tokens is only an input to the real number, which is cost per completed task. Two details make the difference. First, output and reasoning tokens usually cost several times more than input tokens; Google's Gemini API pricing states that the output price for Gemini 4 Argon includes thinking tokens. Second, prices move. The same page lists Argon's input price as $2.00 per million tokens through 30 November 2026, rising to $4.00 from 1 December 2026 (checked October 2026). Recalculate before you sign anything.

Discounts change the maths for work that can wait. Anthropic's Message Batches API cuts costs by 50%, accepts up to 100,000 requests or 256 MB per batch, and says most batches finish in under an hour. Overnight enrichment, document backfills and evaluation runs rarely need real-time answers.

Latency should be measured on your own prompt lengths, both time to first token and total completion time. Anthropic's model selection guide adds a lever many teams miss: an effort setting that trades intelligence for speed and cost within one model, which it says "is often a better lever than switching models".

If your code parses JSON, test format reliability directly or remove the risk. OpenAI's Structured Outputs feature "ensures the model will always generate responses that adhere to your supplied JSON Schema", and Anthropic's structured outputs documentation describes constrained decoding that guarantees schema-compliant responses. Check which of your shortlisted models support these features before you score them.

Check data handling and model lifecycle

Read the data terms for the exact tier you will use. OpenAI's data controls page says data sent to its API has not been used to train its models since 1 March 2023 unless you opt in, and that abuse monitoring logs are kept for up to 30 days by default. Google's pricing page marks free-tier Gemini API content as used to improve its products and paid-tier content as not used (checked October 2026). For a product handling customer data, that difference alone may settle which tier you are allowed to use.

Then look at how long your chosen model will exist. OpenAI's deprecations page promises at least six months' notice before retiring generally available models, at least three months for specialised variants and possibly as little as two weeks for preview models, and it advises against preview models for business-critical workloads. Anthropic's model deprecations page commits to at least 60 days' notice for publicly released models. Its overview lists a "not sooner than" retirement date for each current model, which is worth copying into your planning documents.

Route between tiers and design for switching

Few products need the top model for every call. A common pattern sends classification, routing and extraction to a small model and reserves the large one for hard generation or reasoning. Anthropic's selection guide frames this as a choice between starting efficiency-first with a small model and upgrading only where tests show a gap, or starting capability-first and moving work down once the prompts are tuned. Research supports the idea: the RouteLLM paper reported routers that cut costs "by over 2 times in certain cases" without lowering response quality on its benchmarks. Treat that as a reason to test routing, not as a forecast for your traffic.

Design for switching from the first commit. Keep prompts in version control, send every model call through one internal interface, log the model identifier on each response and make the evaluation harness runnable against any provider. Avoid product logic that depends on one model's formatting habits or refusal style. When a new model appears or a retirement notice lands, you should be able to test the replacement in an afternoon and roll it out gradually.

For founders who want a second opinion on these decisions, George Bregman writes about AI strategy and offers mentoring.

Frequently asked questions

How do you choose an LLM for a product?

Define the task and measurable success criteria, shortlist two or three models using provider documentation and public rankings, then run them on real examples in an evaluation harness. Pick the cheapest model that meets your thresholds for accuracy, latency, format and data handling, and keep the integration swappable.

OpenAI or Anthropic for a startup?

At comparable tiers the published prices and context windows are close (checked October 2026), so the honest answer comes from your own tests. Compare results on your data, structured output support, data terms and deprecation policies rather than reputation.

Should I use the most powerful model available?

Usually not for every call. Test whether a smaller tier matches the flagship on each task; where it does, the saving can be large. Reserve the top tier for the tasks where your evaluations show it earns its price.