12 Things to Check Before You Ship an AI Feature to Production

Most AI feature failures are not model failures. The model does exactly what you asked. The problem is everything around it — the infrastructure you assumed was there, the edge cases you did not test, the costs you did not budget for. This checklist exists because every team re-learns the same 12 lessons.

Why Does Every AI Feature Fail the Same Way?

Because the model is the easy part. LLM APIs are well-documented, fast to prototype, and immediately impressive in demos. What is hard is the surrounding engineering: cost control, safety constraints, observability, fallbacks, and the human systems that catch what the model gets wrong. Teams ship the model and skip the scaffolding. Then they get paged at 2am.

Is This Checklist for Agentic AI or Simpler LLM Features?

Both. The stakes are higher for agentic systems — an agent with tool access and no guardrails can do real damage — but every item here applies to any production LLM feature, from a basic summarization endpoint to a fully autonomous workflow. The complexity of your feature determines how critical each check becomes, not whether it applies.

The 12-Item Checklist

1. Pin your model version before any production deployment. Do not use floating identifiers like "gpt-4" or "claude-latest" in production. Model providers update these silently, and a silent model swap can change output format, tone, refusal behavior, or structured output compliance overnight. Pin to a specific version and create a scheduled task to evaluate upgrades deliberately, not by accident.

2. Set a hard cost budget and alert threshold for every AI call. Measure your average cost per request, multiply by expected peak traffic, then add 3x. Set a budget alert at 50% of that ceiling and a hard cutoff before you hit your monthly limit. A runaway loop, a misconfigured batch job, or a sudden traffic spike can burn through your AI spend in hours. Cost control is not optional — it is a deployment requirement.

3. Validate and schema-check every model output before using it. Never trust raw LLM output as structured data. If your feature depends on JSON, a list, a number, or any specific format, validate against a schema before processing. Models hallucinate formats, drop fields under load, and occasionally return prose where you expected an object. Treat model output like untrusted user input — parse defensively.

4. Test your prompt against injection attacks relevant to your input surface. If any part of your prompt includes user-controlled input, you have a prompt injection surface. Test it. Common vectors: role overrides, context stuffing that shifts model behavior, and jailbreaks that extract system prompt content. Use a dedicated injection test suite before shipping, not after your first incident report.

5. Audit every code path for PII leakage into model inputs. Map what data flows into your LLM calls. Names, email addresses, session identifiers, health data, payment details — any of these in your prompt means you are sending them to a third-party API. Check your provider's data retention policy. Check whether this conflicts with your privacy policy, GDPR obligations, or HIPAA requirements. If you cannot answer this question, you are not ready to ship.

6. Build and run an eval harness before the first production deploy. An eval harness is a set of known inputs with expected outputs that you can run against any model version or prompt change. It does not need to be large — 20 to 50 cases covering happy paths, edge cases, and known failure modes is enough to start. Without an eval harness, every prompt change is a blind deploy. This is the minimum viable testing infrastructure for any AI feature.

7. Implement a model fallback for every critical AI path. Define what happens when your primary model is rate-limited, returns an error, or times out. For latency-sensitive features, a lighter, faster model is often a better fallback than a retry loop. For accuracy-sensitive features, a cached response or graceful degradation is better than a 503. The fallback is not an afterthought — it is the feature contract for your reliability SLA.

8. Set a latency budget and test under realistic concurrency. Know your p50, p95, and p99 latency before shipping. LLM calls are slower than typical API calls, and streaming does not eliminate tail latency — it just redistributes it. Load test your feature at 2x expected peak traffic. If the latency is unacceptable at peak, you have an architectural problem that cannot be solved at launch by tuning prompts.

9. Add structured audit logging for every LLM interaction. Log the model version, input token count, output token count, latency, the prompt template name, and the outcome (success, error, fallback triggered). Do not log raw user inputs or outputs unless you have explicit consent and a data retention policy. You need this data to debug issues, optimize costs, detect abuse, and comply with audit requirements in regulated industries.

10. Define and test your human escalation path for low-confidence or sensitive outputs. Not every output should be acted on automatically. Define confidence thresholds, output categories, and content types that require human review before action. For customer-facing features, implement an explicit fallback to a human agent. For internal tools, define who gets paged when the model is uncertain. The escalation path is part of the feature spec — design it before you ship, not after the first production incident.

11. Implement rate limiting at the AI feature layer, separate from your general API limits. Your general rate limiter protects your infrastructure. Your AI-specific rate limiter protects your LLM cost budget and downstream model quotas. These are different things. A single user running a loop against a summarization endpoint should not be able to exhaust your entire monthly token budget. Implement per-user, per-session, and per-endpoint limits with sensible defaults.

12. Document and test your rollback procedure before go-live. Know exactly how to disable the AI feature, revert to the previous behavior, and restore service in under 10 minutes. This means a feature flag, a config-driven model selection, or a kill switch — not a code deploy. AI features fail in unexpected ways. The rollback procedure is your last line of defense. If you cannot describe it in one sentence, you do not have one.

How Do You Prioritize This Checklist for a Small Team?

Do items 1, 2, 3, and 5 before you ship anything to real users — these cover cost catastrophe, output corruption, and regulatory exposure. Add items 6, 9, and 12 in your first sprint after launch — eval harness, audit logging, and rollback turn your feature from a prototype into an owned system. The remaining items are a function of your traffic, sensitivity of data, and uptime requirements. None of them are optional forever — they are just sequenced by blast radius.