Cloud infrastructure for AI agents in production.
AI agents fail in production in ways traditional systems don't — model timeouts, cost spikes, prompt regressions. Cloud architecture for AI products is monitoring, cost control, and rollback strategy. We build the infrastructure that catches these before they hit users.
The cloud infrastructure discipline has a new cost center that did not exist three years ago: GPU compute. Whether you are self-hosting inference servers (vLLM on your own Kubernetes cluster), running batch embedding jobs (millions of documents per week through an embedding model), or running fine-tuning experiments, GPU infrastructure is now part of the engineering equation for any serious AI application.
GPU availability on public clouds remains constrained relative to demand. Teams building AI infrastructure in 2026 design for multi-cloud from the start — not for theoretical resilience, but because the practical reality is that AWS might have better A10G availability this quarter and GCP next quarter. Locking into a single GPU provider is a capacity risk.
The standard pattern for self-hosted inference in 2026: a Kubernetes cluster with dedicated GPU node pools, vLLM or TGI running as Deployments with pod anti-affinity to spread across GPU nodes, horizontal pod autoscaling based on GPU utilization or queue depth, and a load balancer in front routing to healthy pods. This stack handles most production inference workloads and is well-documented.
The parts that require judgment: GPU node pool sizing (too small means queuing, too large means idle GPU costs), KV cache configuration in vLLM (the right setting depends on your model and typical sequence lengths), and batch size tuning for embedding jobs (larger batches are more GPU-efficient but add latency). These are not parameters you find in the official documentation — they come from running the stack under real load.
The number one unexpected cost in AI applications is inference — more specifically, inference that could be avoided or made cheaper with better system design. Semantic caching (returning stored responses for semantically similar queries) is the highest-leverage optimization. Model routing (using a cheaper, smaller model for simple queries and a more capable model for complex ones) is the second. Getting these right requires instrumentation: you need to know the cost per feature, per user, and per query type before you can optimize.
Carbon-aware scheduling is an emerging practice for batch workloads. Embedding generation jobs and fine-tuning runs can typically tolerate an hour of schedule flexibility. Running them when the grid carbon intensity is low (night time in coal-heavy regions, when renewable output is high) costs nothing extra and reduces the environmental footprint of AI workloads. GCP and Azure both expose carbon intensity data via API.
AI workloads have a different infrastructure profile than traditional web applications — GPU availability is constrained, inference costs can dominate the monthly bill, and multi-cloud isn't optional when you're hedging against single-provider GPU shortages. We've designed and operated these systems, so we know where the cost surprises hide and what prevents them. Our scope covers everything from initial cloud architecture through CI/CD pipelines, observability stacks, and ongoing cost rightsizing.
On the inference layer, we work with vLLM and Text Generation Inference for self-hosted model serving, configure KV cache settings and tensor parallelism, and implement semantic similarity caches at the application layer to cut repeated inference costs. For Kubernetes, we handle GPU node pools, CUDA-aware scheduling, and spot instance interruption handling for batch embedding jobs. Everything — every resource, every policy, every pipeline — is defined in Terraform or Pulumi, not clicked together in a cloud console.
We map each component to its compute profile: CPU-bound APIs, GPU-bound inference endpoints, memory-intensive vector operations, high-IOPS database workloads. This mapping drives instance type selection and prevents the expensive over-provisioning that happens when teams treat everything as a generic compute problem.
We design the cloud topology — VPC structure, subnet layout, service mesh decision, multi-region requirements. The inference serving layer gets its own design pass separate from the application layer, because the two have different scaling triggers, different uptime requirements, and different failure modes.
Every resource is implemented as Terraform modules or Pulumi programs — reviewed, version-controlled, and reproducible. No manual console changes, no configuration drift, no tribal knowledge required to rebuild an environment from scratch.
We build pipelines that run tests, build and push container images, deploy to staging, and promote to production on approval. GitHub Actions or CircleCI for orchestration; ArgoCD or Flux for Kubernetes deployments via GitOps, where every production change traces back to a reviewed git commit.
We deploy Prometheus, Grafana, Loki, and Tempo (or Jaeger) for metrics, logs, and traces. Alerting is configured on metrics that predict user impact — inference latency p95, queue depth, error rates — not just CPU and memory.
We review utilization against provisioned capacity, rightsize instance types, convert consistent workloads to reserved instances, and move batch jobs to spot with checkpointing. Inference caching for repeated queries is evaluated early — it's frequently the highest-leverage cost reduction available.
Tools and infrastructure we use for this capability.
Inference costs can 10x overnight from a bad prompt or an upstream model price change. We instrument cost per request, per user, per workflow — with alerts before the bill arrives, not after.
Prompts and model versions are deploy artifacts. We treat them as such — with staging, canarying, and one-command rollback when a new prompt regresses on production traffic.
Traces that follow a request from the web layer through the agent state machine through the model call through the downstream API. When something is slow or wrong, we can find it.
Three levers: right-sizing (most inference workloads run fine on A10G or L4 instances rather than A100s — significant cost difference, small latency difference for typical batch sizes), spot instances for batch embedding and training jobs with checkpointing so interruptions don't lose progress, and inference caching at the application layer. A semantic similarity cache — checking if a close-enough question was recently answered before making a new model call — can cut inference volume by 30-50% for many applications.
Kubernetes is the right call when you have multiple services with different scaling profiles, need GPU scheduling and node affinity for inference workloads, or require rolling deployments with health checking across a fleet. For a handful of services with predictable load, managed container services like ECS, Cloud Run, or Fly.io carry less operational overhead. Starting on Kubernetes before the complexity justifies it is a common, expensive mistake.
GitOps means git pull requests are the mechanism for all production changes — including Kubernetes deployments. ArgoCD and Flux watch your repos and apply changes when they merge, so your production state always traces back to a reviewed, auditable commit. For teams with solid CI/CD discipline already, GitOps is the natural next step; for teams building their deployment process from scratch, it's worth investing in at the start rather than retrofitting.
Kubernetes Secrets are base64, not encrypted at rest by default — not adequate for production. We use AWS Secrets Manager or GCP Secret Manager with the external-secrets-operator (secrets live in your cloud provider, injected into pods at runtime), or HashiCorp Vault for multi-cloud environments where you need fine-grained access control. The hard rule: secrets never live in git, even encrypted, without a proper secrets management solution behind them.
Edge inference runs AI models at edge locations — Cloudflare Workers AI, Fastly, Vercel Edge — rather than centralized compute, cutting latency for geographically distributed users. It makes sense when the round-trip to a central region adds 100-200ms that users will notice, and when the model fits within edge runtime constraints (typically small distilled models, not GPT-4o scale). Cloudflare Workers AI is the most accessible entry point for evaluating whether edge inference is worth the operational complexity for your use case.
Anonymized engagements with real outcomes — no client names per NDA.
94%
Classification Accuracy
340ms
Avg Processing Time
78%
Manual Review Reduction
“The context understanding is the part that changed the team's view of AI moderation. It is not just pattern matching — it is understanding that the same phrase can be a policy violation in one context and completely acceptable in another.”
— Trust and Safety Lead, User Content Platform
19%
Energy Cost Reduction
99.1%
Sensor Data Uptime
4.2s
Alert Latency
“We were making energy management decisions from monthly utility bills. Having real-time sensor data and anomaly detection changed what was even possible — we caught equipment inefficiency that had been running for years without anyone knowing.”
— Head of Facilities Operations, Manufacturing Conglomerate
34%
Forecast Accuracy Improvement
61%
Stockout Reduction
23%
Overstock Reduction
“The stockout reduction was measurable within the first planning cycle. Category managers were sceptical initially, but the forecast accuracy on the products that had historically been hardest to predict won them over.”
— Head of Supply Chain Planning, FMCG Distributor
Cloud Infrastructure & DevOps sits beneath the services we sell and the agents we ship. If you are scoping outcomes rather than tools, start with one of these.
Tell us what you are building. We will scope it, price it honestly, and give you a clear plan.
Start a ConversationFree 30-minute scoping call. No obligation.
Fordel Studios