The distance between an AI demo and production.

A prototype answers one question: can this idea work? It is usually built by the person who will judge it, on inputs they chose, for one user, with no concurrency, on whichever model version was current that week. Every one of those conditions changes in production.

Real users ask questions the author never imagined, upload documents in the wrong format, and phrase the same request ten different ways. The provider updates the model. The retrieval index falls behind its sources. Latency that felt fine in a notebook becomes an eleven-second 95th percentile under load. None of this shows up in a demo, because a demo is a sample of one from a distribution nobody has measured yet.

So the production question is different: when this system fails, how will we know, who will notice, and what happens next? Those answers are architecture. They decide what you log, what you test, what the model is allowed to do, and what the user sees when something goes wrong. They can be retrofitted after launch, but usually only by rebuilding the parts that made the demo look easy.

The most valuable artifact in an AI project is not the prompt. It is a set of representative inputs with an agreed definition of a good answer and an unacceptable one. Without it, every change to the prompt, the model, the chunking strategy, or the retrieval ranking is judged by whoever happens to try it, on whatever they happen to type.

Start small and real. Fifty to two hundred cases drawn from actual requests (support tickets, search logs, the documents the system will read) are worth more than a thousand synthetic ones. Have a domain expert label each case: which source documents a correct answer must be grounded in, which facts it must contain, and which requests the system should decline as out of scope. Tag every case with a category, so that a regression in billing questions is visible even when the overall pass rate barely moves.

Grade with deterministic checks wherever you can. Whether the output parses against the expected schema, whether it cites the required documents, whether it refused when it should have: these checks are cheap, exact, and stable. Keep model-graded checks, where another model acts as the judge, for genuinely subjective criteria such as tone or completeness, and calibrate them against a sample of human labels before trusting their scores.

from dataclasses import dataclass, field

@dataclass(frozen=True)
class Case:
    id: str
    category: str                    # e.g. "billing", "out_of_scope"
    question: str
    must_cite: frozenset[str] = field(default_factory=frozenset)
    must_refuse: bool = False

def grade(case: Case, result) -> dict[str, bool]:
    """Deterministic checks first: cheap, exact, and they never drift."""
    cited = {c.doc_id for c in result.citations}
    return {
        "schema_valid": result.parsed is not None,
        "refusal_correct": result.refused == case.must_refuse,
        "grounded": case.must_refuse or case.must_cite <= cited,
    }

def regressions(baseline: dict[str, float], current: dict[str, float], tolerance: float = 0.02):
    """Compare pass rates per category; a drop in any one category fails the build."""
    return {cat: (baseline[cat], rate) for cat, rate in current.items()
            if cat in baseline and rate < baseline[cat] - tolerance}

Run the suite in CI on every change to prompts, retrieval configuration, or model version, and keep the history per category. The per-category view matters: a new model can raise the average while quietly breaking the one category your largest customer depends on. When production surfaces a failure, add it to the set. Over time, the evaluation set becomes the working specification of how the product behaves.

Retrieval-augmented systems read text the team did not write: web pages, uploaded files, emails, tickets, tool responses. Any of that text can contain instructions, and a model cannot reliably tell the instructions you gave it from instructions embedded in a document it was asked to summarize. This is prompt injection. It is first on the OWASP Top 10 for LLM Applications because there is no complete defense at the model layer.

The practical response is to limit what a successful injection can achieve. Design as if any model output could have been written by an attacker, then decide what that output is allowed to cause.

• Least privilege for tools. Give each workflow only the tools it needs, called with the requesting user's permissions, not a service account that can see everything.
• Separate reading from acting. The step that reads untrusted content should not be the same step that can send email, move money, or change records.
• Validate output as data. Parse model output against a schema and check values against business rules before anything downstream uses them. Never pass it unescaped into a shell command, a SQL string, or an HTML template.
• Require approval in proportion to impact. Classify every action by how reversible it is and whom it affects.

The last row matters most. The safest tool is the one the model was never given.

A conventional web request has a handful of places where it can go wrong. An AI request has more, and they compound: query rewriting, retrieval, reranking, one or more model calls, tool calls, output validation, and approval. When a user reports a bad answer, you need to reconstruct exactly what the system saw and did, without guessing.

Trace each request end to end, with a span per stage. OpenTelemetry's semantic conventions for generative AI define attribute names for this, such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.response.finish_reasons. They are still marked as in development, but adopting them now is cheaper than inventing your own names and migrating later.

For each span, record what you will need during an incident: the retrieved document IDs and scores, the model version the provider actually served, token counts, latency, finish reason, which validation rules passed or failed, and whether a person approved the action. Be deliberate about content. Prompts and outputs often contain personal data, so decide what is stored, for how long, and who can read it before the first trace is written.

Then roll the same data up into the numbers that run the service: cost per successful task rather than per call, tail latency per stage, validation failure rate, refusal rate, and the share of requests escalated to a person. A sudden rise in refusals or validation failures after a provider update is often the first sign that model behavior has changed.

Model providers have outages, rate limits, and latency spikes. Retrieval indexes fall behind their sources. A production system needs a defined answer to each of these before launch, not an improvised one during the incident.

• Pin model versions and treat an upgrade as a release: run the evaluation suite against the new version, compare per category, and roll it out behind a flag.
• Set a timeout per stage and decide what the user sees when it fires. For many products, returning the retrieved sources without a generated summary is far better than a spinner followed by an error.
• Degrade, then escalate. If validation fails twice, stop retrying the model and route the request to a person or a deterministic path, with the context attached.
• Budget the cost. Cap tokens per request and per tenant, and alert on unit cost, so a prompt change or a looping agent cannot multiply spend unnoticed.

None of these are model settings. Production readiness is a property of the whole system. The model is one component, and rarely the one that decides whether users trust it.

1. A labeled evaluation set drawn from real inputs runs in CI, with pass rates per category.
2. Retrieval quality is measured separately from answer quality.
3. Every tool runs with the requesting user's permissions, and consequential actions require approval.
4. Model output is validated against a schema and business rules before anything acts on it.
5. Requests are traced per stage, with model version, tokens, latency, and validation outcomes.
6. Model versions are pinned, and upgrades are evaluated and rolled out like any other release.
7. Timeouts, fallbacks, and the route to a person are defined and tested.
8. Unit cost and tail latency have budgets and alerts.

A system missing some of these may still work. The team just will not know when it stops. If you are planning an AI feature and want a second opinion on the architecture, that is the work of our AI development practice.