My team and I run LLM serving infrastructure for enterprises and the public sector, and I keep watching strong teams hit the same wall: scaling an AI product is nothing like prototyping one. Only about one in eight AI pilots reaches production, per IDC,[1] and MIT found 95 percent deliver no measurable ROI.[2] Rarely because of the model. Usually because nobody planned for production before or during the pilot.
Here are 6 things to consider, with a full checklist at the end.
1. Collect non-functional requirements. Yes, for the pilot.
Expected load, latency, concurrency: AI products need them like any system, but a chat demo hides them. Some of the most important metrics are time to first token, end-to-end response time, prompt and output length, and number of concurrent users or requests.
These parameters decide your setup. Voice applications require low latency, which means capacity headroom and often dedicated instances. Latency-sensitive workloads on shared infrastructure fail exactly where users tolerate it least.
Know your usage patterns. Do all your users get active first thing in the morning? Which tasks can wait? You pay for the peak.
Expected output quality, though not a classic NFR, belongs on the same list. It drives your model choice and with it your infrastructure costs and operational setup. Demanding 100 percent accuracy ignores the non-deterministic nature of AI. Define acceptable quality per feature and measure it continuously.
Example. 5,000 employees × 10% active at the same time between 9-10AM = 500 concurrent requests, ten times what the daily average suggests. Size for the average and every morning starts with timeouts.
2. Choose the model for the job, not the leaderboard
The largest model is usually just the most expensive default: a chainsaw for chopping carrots. Start from what the task requires.
Architecture matters as much as size. A dense model activates every parameter on every request; a mixture-of-experts model only a fraction. That sets your speed and hardware bill. Judge models by cost to run and token efficiency, not solely by their leaderboard rank.
Evaluate on your own data. Small internal evals from real failure cases beat public benchmarks and survive model switches. See the Applied LLMs guide.[3]
When in doubt, start with verifiable tasks. Per a16z,[4] the enterprise use cases that succeed have checkable output: code runs or it does not, a support ticket is resolved or escalates to a human. It is also easier to demonstrate the value of an AI rollout in these scenarios.
Example. Ticket classification: small model = 94% accuracy at €0.05/M tokens; frontier model = 96% at €15/M. That is +2 points for 300× the price. Sometimes worth it, but you only know after running both on your own data.
3. Know what each deployment option commits you to
Token APIs give you the fastest start. Set quotas, respect rate limits, monitor cost from day one. According to Gartner,[5] agents burn anywhere from 5 to 30 times more tokens per task than a chatbot. Be careful extrapolating pilot costs: for example, a production agent turns one task into ten or twenty calls, each growing its context, so a budget built on pilot economics is off by a multiple, not a margin.
Dedicated capacity is the way to go for predictable load and strict latency. The downside is that you pay whether it is busy or idle. Offset this by scheduling non-time-critical workloads into off-peak windows.
Self-hosting is the last resort, for when sustained utilization tips the math or regulation leaves no alternative. Hardware gets outdated within two or three years, and running inference well takes scarce, expensive specialists. Unless your utilization is high and sustained, buy serving instead of building it. Customers in regulated industries choose our LLM Serving for exactly this reason: open-weight models on European infrastructure, with dedicated, tenant-isolated instances where needed.
Example. Dedicated instance at €10,000/month ÷ API price of €2/M tokens = 5B tokens/month break-even. Below that, you pay for idle hardware; above it, every token beats list price. Utilization decides.
4. Estimate the costs before they become a rollout blocker
Every AI feature carries a variable cost on each use, so margin is tied directly to usage. A demo that delights in the boardroom can quietly destroy the margin at scale. The CFO's Playbook for AI Unit Economics[6] offers a simple test for whether a team has this under control: can they say what each feature costs in tokens? If not, there is no cost visibility, and no basis for pricing.
Define your unit, whether cost per query, per session, or per resolved task. Ranges are enough. Budget for adoption success: though token prices fall, bills rise because usage outruns price.
Behind every token price sits hardware, and hardware is the dominant cost. Your requirements drive architecture, architecture drives cost. Skip step one and every estimate is fiction. Napkin math suffices, and for large-scale operations you can check your assumptions with your AI provider.
Example. Chatbot: 1,000 users × 10 queries × 2,000 tokens × 30 days = 0.6B tokens/month → several hundred euros. Agentic version: × 15 calls × 2× context = 18B tokens/month → five figures. Same users, same feature, ~30× the cost.
5. Build the boring parts
None of this is AI-specific, yet the faster and easier it is to build software, the more these best practices are ignored. Load test at one, two, and five times expected peak. Before launch.
Budget resilience upfront: fallbacks, high availability, real SLAs. Keep a staging environment. A safe place to experiment is what keeps you fast after launch.
Monitor cost per query, latency percentiles, and quality over time. Models drift; some providers ship silent updates. Stanford's study of 51 enterprise deployments[7] found the winners invest mostly outside the model: process, data, human oversight.
Example. 10 concurrent requests → P95 = 0.9s. 60 concurrent → P95 = 8s, not 1.5s, because saturated GPUs queue. Throughput has a cliff; a load test finds it before your launch does.
6. Security, data, and the rules
Classify your data before the pilot. It constrains provider choice, hosting model, and documentation duties. Choose a provider that complies with the applicable regulations.
Set the necessary guardrails. Consider AI-native threats like prompt injection, hallucination, and data leakage. Design for this instead of retrofitting. For example, in SmartChat we combine UX that protects against accidental data leaks with sovereignly hosted models for sensitive work. The goal should be to avoid shifting the responsibility for data protection onto the user.
Example. A pilot summarizes contracts through a US-hosted API. Legal later classifies contracts as confidential → provider, hosting, and logging all change mid-project = weeks of migration and a stalled rollout. Classifying the data first = one meeting.
The checklist
I have compressed everything above into a single-page checklist below, also available as a PDF (opens in new tab). Share it with the team building the pilot and the team that will run the finished product. Asking these questions early will save you time and money down the line.
The teams that get somewhere with AI don't experiment less. Their experiments are designed to survive success.