A predictive GPU auto-scaling engine and a domain-trained SLM, ~40% lower compute cost
The client needed a domain-aware language model that kept sensitive data inside their own infrastructure and out of commercial AI APIs, a compliance requirement for their regulated workload. We trained a small language model (SLM) on their proprietary data, deployed it on AWS EC2 GPU instances, and built a predictive auto-scaling engine that forecasts transactions-per-minute from historical patterns and scales the GPU fleet ahead of demand. The fleet runs lean by default, and continuous observability gives the team end-to-end visibility into utilisation, cost, and forecasted load.
Client anonymised at their request. Industry, scale and every figure are real.
~40%
Reduction in monthly compute cost
100%
Inference inside client infrastructure
24/7
Observability across the GPU fleet
The constraint
Sensitive data could not leave the account, and a fleet sized for peak would idle most of the day.
The workload was regulated. Every inference touched sensitive records, so sending prompts to a commercial AI API was ruled out before the first architecture conversation. Whatever model answered had to run inside the client's own AWS account, on infrastructure their security team already governed.
That ruled out the easy option but created a cost problem. A general-purpose model served privately at their throughput would have been expensive to run, and a GPU fleet sized for the daily peak would sit idle for most of the twenty-four hours. The team needed a model small enough to serve cheaply and a fleet that was only as large as the next hour demanded.
What we built
A small model trained on their data, and a fleet that scales before the spike arrives.
We trained a small language model on the client's proprietary data, evaluated it on their own tasks rather than public benchmarks, and served it on EC2 GPU instances inside their account. A domain-trained SLM beat a general model on the work that mattered and cost a fraction to run per request.
Around it we built a predictive auto-scaling engine. It forecasts transactions per minute from historical patterns, including the weekly and intraday cycles in the client's traffic, and scales the GPU fleet ahead of demand rather than reacting once latency has already climbed. The fleet runs lean by default and grows only when the forecast says it must.
- SLM training and evaluation pipeline on the client's labelled data
- GPU serving on EC2 with health checks and warm capacity
- Time-series forecasting of load, retrained on a schedule
- Scaling policies driven by the forecast, with reactive guardrails
- CloudWatch dashboards for utilisation, cost and projected load
- Everything provisioned with Terraform in the client's account
What changed
About forty percent off the monthly bill, with every request answered inside the boundary.
Monthly compute cost fell by roughly forty percent against the previous always-on fleet, with no change to the compliance posture: one hundred percent of inference stays inside the client's infrastructure. The operations team now sees utilisation, spend and forecasted load in one place, and the scaling decisions are explainable because the forecast that drove them is logged.
What the team owns
What was running, and in whose hands, when we stepped back.
- Model weights, the training pipeline and the evaluation harness
- The forecasting model, its retraining job and the scaling policies
- Terraform for the whole fleet, reviewed by the client's platform team
- CloudWatch dashboards and a runbook for scaling incidents
Multi-phase engagement