Running AI operations in a startup means treating every model-powered feature as a live service: you log each call, measure output quality against your own tests, cap spend, plan for vendor model changes and keep a tested fallback for when things go wrong. Most of the work is operational discipline rather than model research. A small team can cover it with a written feature spec, a handful of metrics, an evaluation set that runs on every change and a short incident routine, all mapped loosely to a published framework such as the NIST AI Risk Management Framework.

What AI operations actually covers

AI operations sits between product and engineering. The model itself is usually rented from a provider, so the startup's job is everything around it: the prompt, the data passed in, the checks on what comes out, the cost of each call and the response when quality drops.

The problem is older than large language models. In 2015 a group of Google engineers led by D. Sculley published Hidden Technical Debt in Machine Learning Systems, which argued that quick wins from machine learning carry "massive ongoing maintenance costs" and named risks such as entanglement, hidden feedback loops and "changes in the external world". Swap "model you trained" for "model you call over an API" and most of that list still applies.

A useful scaffold for a small company is the NIST AI Risk Management Framework, released on 26 January 2023 and intended for voluntary use. Its core, set out in the AI RMF 1.0 document, has four functions: GOVERN, MAP, MEASURE and MANAGE. NIST added a Generative AI Profile (NIST-AI-600-1) on 26 July 2024, and its page notes that AI RMF 1.0 is being revised (checked October 2026). You do not need to adopt it formally. Using the four headings as a checklist keeps the operations work from collapsing into firefighting.

Define each AI feature before you pick a model

Start with the job the user is trying to get done, not with a provider. For each feature, write down the input, what an acceptable output looks like, what happens downstream when the output is wrong and the most you are willing to pay per completed task. That document becomes the reference for every later decision about prompts, models and alerts.

It also forces an honest question: does this path need a model at all? OpenAI's own latency optimisation guide lists seven principles, and the last one is "Don't default to an LLM". Where a template, a lookup table or a rules engine produces the right answer every time, it is cheaper to run and far easier to support.

Map the failure modes while the feature is still on paper. A wrong summary in an internal tool is an irritation; a wrong figure in a customer invoice is an incident. The severity of a bad output decides how much review the feature needs, which is the MAP function in NIST's language.

Instrument every model call from launch day

You cannot operate what you cannot see. Before a feature goes live, log for every request the prompt template version, the model identifier returned by the API, input and output token counts, latency, cost and a user-facing outcome such as accepted, edited, retried or abandoned. The outcome field is what links technical behaviour to value.

Google's SRE book, in a chapter written by Rob Ewaschuk, describes the four golden signals of monitoring as latency, traffic, errors and saturation, adding: "If you can only measure four metrics of your user-facing system, focus on these four." For an AI feature, add a quality signal (your acceptance or edit rate) and a spend signal (cost per completed task), and you have a compact dashboard that catches most regressions.

To avoid inventing your own field names, look at the OpenTelemetry GenAI semantic conventions, which define spans, metrics and events for generative AI clients, including metrics such as gen_ai.client.operation.duration and gen_ai.server.time_to_first_token. The metrics specification is still marked "Development" (checked October 2026), so expect names to shift, but adopting it early makes switching observability vendors less painful.

Keep alerting simple at first. Static thresholds on error rate, latency and hourly spend, sent to whoever is on call, beat a sophisticated anomaly detector nobody trusts. If you handle personal data in prompts or logs, read the ICO's guidance on AI and data protection before deciding what to retain and for how long.

Build an evaluation loop that runs on every change

Model output varies even with identical input, so ordinary unit tests are not enough. OpenAI's evaluation best practices recommend "eval-driven development", logging everything so you can mine production traffic for test cases, and using human feedback to calibrate automated scoring. The same page warns against "overly generic metrics" and against eval sets that do not reproduce real traffic.

In practice that means a curated set of real inputs, each paired with an expected answer or a scoring rule, run automatically whenever someone edits a prompt, changes retrieval or switches model. Keep it small enough for a person to review the failures in one sitting and add a case every time a user reports a bad result. A weekly read of a random sample of live outputs catches problems no fixed test set anticipates.

Tooling is now free. The UK AI Security Institute and Meridian Labs maintain Inspect, an open-source evaluation framework whose listing includes implementations of more than 200 benchmarks (checked October 2026). One warning for teams on hosted tools: OpenAI's documentation says its Evals platform becomes read-only for existing users on 31 October 2026 and is scheduled to shut down on 30 November 2026, so keep eval datasets in a format you own.

Treat vendor model changes as a standing risk

The model behind an API can change under you, and the effects are measurable. A 2023 study by Lingjiao Chen, Matei Zaharia and James Zou, How is ChatGPT's behavior changing over time?, found GPT-4's accuracy at identifying prime numbers fell from 84% in its March 2023 version to 51% in its June 2023 version, and concluded that "the same" LLM service can change substantially in a short time, "highlighting the need for continuous monitoring".

Two habits reduce the exposure. First, pin dated model snapshots in production rather than floating aliases, and log the model identifier on every call so a behaviour change can be traced. Second, track retirement schedules. Anthropic's model deprecations page commits to at least 60 days' notice before retiring publicly released models; OpenAI's deprecations page promises at least six months for generally available models and as little as two weeks for preview models (both checked October 2026). Put those dates in the same calendar as contract renewals.

Roll out any model or prompt change gradually. Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation", and the idea transfers directly: send a small share of traffic to the new version, compare quality and cost against the control and only then widen it.

Decide where humans stay in the loop

Not every output should reach a customer unchecked. A workable split has three levels: outputs that ship automatically because errors are cheap and easy to reverse, outputs drafted by the model and approved by a person, and work done by a person with the model only suggesting. Start conservative and loosen a feature only when its acceptance and error data justify it.

When a reviewer takes over, hand them the model's draft, the source material it used and any validation failures, rather than a blank form. That makes review faster and helps reviewers spot outputs that are fluent but wrong.

Regulation can make some of this mandatory. The European Commission's AI Act implementation timeline shows the Article 50 transparency rules applying from 2 August 2026, and, after the Digital Omnibus amendments, rules for Annex III high-risk systems applying from 2 December 2027 (checked October 2026). If you sell into the EU, read the official text for your use case; this article is not legal advice.

Manage cost and abuse as operating limits

Cost per completed task is the number that matters, because a request that is rejected and retried three times costs four calls. Set a budget for each feature and alert when the rolling average breaches it.

Providers offer three main levers. Prompt caching reuses an identical prompt prefix: in Anthropic's prompt caching documentation, cache reads are priced at 0.1 times the base input price on most models, with a default five-minute cache lifetime (checked October 2026). Batch processing suits anything that does not need an instant answer; OpenAI's Batch API offers a 50% discount with a 24-hour completion window. Routing sends easy requests to cheaper models: the FrugalGPT paper reported matching GPT-4's performance with up to 98% cost reduction on its benchmarks, and the RouteLLM paper reported cost cuts of over two times in certain cases without lowering response quality. Treat those as research results, then measure on your own traffic.

Cost is also a security issue. The OWASP Top 10 for LLM Applications 2025 lists prompt injection first and includes "Unbounded Consumption", whose entry describes denial of wallet attacks and recommends rate limiting and per-user quotas. Put both in place before launch.

Write and rehearse the incident routine

AI failures often arrive without an error code: answers get worse, a prompt injection leaks a system prompt or spend jumps overnight. Write the routine down before you need it, in four steps.

  • Detect. Alerts on quality, latency, errors and spend, plus a clear channel for user reports.
  • Contain. A feature flag that switches traffic to a fallback, such as a previous model snapshot, a cached or template answer, or a human queue.
  • Diagnose. Pull the logged prompts, model identifiers and outputs, and reproduce the failure against your evaluation set.
  • Recover. Fix, rerun the evaluations, canary the change and record what you learnt as a new test case.

Rehearse the kill switch in staging so you know it works and how long it takes. If you are building this function for the first time and want an outside view, George Bregman writes about AI strategy and offers mentoring for founders.

Frequently asked questions

What does an AI operations lead do day to day?

They watch quality, latency and spend dashboards, review samples of live outputs, keep the evaluation set current, test model and prompt changes before rollout, track provider deprecation notices and run incidents. In a small startup this is often a part-time role held by an engineer or product lead.

Which metrics should a startup track first?

Latency, traffic, errors and saturation from classic service monitoring, plus an acceptance or edit rate for quality and cost per completed task. Add token usage and time to first token once traffic grows.

Does a startup need a formal AI governance framework?

Not formally at the start. The NIST AI RMF is voluntary, but its four functions work as a lightweight checklist, and the EU AI Act may impose specific duties depending on what you build and where you sell.