Enterprise AI in production
The demo worked. Now ship it.
Five things that break between a good demo and a product people use every day, and what to plan for each.
Akshay Sharma6 October 2026 · 3 min read
A demo is a best case. Someone who understands the system picks a good input and shows a good output. Production is everyone else: hurried users, incomplete data, and a thousand requests an hour. Most of my work as a product manager has happened in the gap between the two.
I talked about this at length on The Voice of Visionaries podcast. This is the written version: what I have seen break, and what I now plan for.
1. Real inputs are messier than test inputs
In a demo, the question is clear and the document is complete. Real users paste half a document, ask two questions at once, or leave out the one detail that matters.
What to do. Collect real inputs as early as you can, even from five pilot users, and build your tests from those. Sort them into three piles: handled well, handled badly, and should have been refused. The third pile is usually the surprise.
2. Cost appears at scale
A feature that costs a few cents a request looks free in a demo. Multiply by real volume, add retries, long inputs and a second model checking the first, and it becomes a line in the budget that someone has to defend.
On cloud cost products I learned that teams decide faster when the cost of each option is in front of them. Do the same for your own feature.
What to do. Before launch, calculate the cost of one successful task, not one request. Include failures you had to retry and outputs a person had to fix.
3. Speed changes behaviour
If a response takes fifteen seconds, people stop waiting and go back to the old way. They do not complain. They just stop using it, and the usage chart tells you a month later.
What to do. Decide what speed the workflow needs, then design for it. Show partial results as they arrive. Run slow steps in the background and notify the user. Use a smaller model where it is good enough.
4. Quality drifts
Data changes, policies change, and users find new ways to use the product. A feature that passed its tests in March can be quietly worse by June, and nothing in the code will have changed.
What to do. Track quality in production weekly. Sample real outputs and have someone score them. Give users a one-click way to flag a bad answer, and make sure flags go to a person.
5. Adoption is not automatic
The biggest surprise for most teams is that a working feature can go unused. People need to know it exists, trust it, and find it at the moment they need it. I have seen onboarding and clear guides lift adoption by 20% with no change to the underlying product.
What to do. Give adoption its own plan: where the feature appears in the workflow, how a first-time user learns what it is good at, and what you will measure in the first month.
What changes between demo and production
| Demo | Production | |
|---|---|---|
| Inputs | Chosen by the team | Whatever users send |
| Success | It worked once | It works every day |
| Cost | Ignored | Measured per successful task |
| Failure | Hidden | Designed for |
| Users | The builders | People with other things to do |
A pre-launch checklist
Before any AI feature goes to general release, I want a written answer to each of these.
- What is the quality bar, and what did we measure against it?
- What does a successful task cost?
- How long does a typical response take, and is that fast enough for the workflow?
- What does the user see when the AI is wrong or unavailable?
- Who reviews flagged outputs, and how quickly?
- How will we know in a month whether people use it?
If any answer is "we will find out after launch", that is the thing most likely to hurt you.
None of this is glamorous, and none of it shows up in a demo. It is most of what it takes to get an AI feature into production and keep it working there.
Found this useful? Share it, or get the next one through The AI Product Playbook, my LinkedIn newsletter.
