The demo always works. Here’s what breaks between a working prototype and a feature real users depend on.
The number in the title isn’t a citation — it’s a rough, honest estimate from watching this pattern repeat across enough projects to notice the shape of it, not a stat from a published study. But talk to enough teams that shipped an “AI-powered” feature in the last couple of years, and a genuinely large fraction of them will tell you the same story: it demoed beautifully, shipped to real users, and within two quarters was either quietly disabled, rewritten from scratch, or downgraded to something far less “AI” than the original pitch. Not because the underlying model got worse. Because the gap between “the demo worked” and “this survives being depended on” turned out to be almost entirely made of ordinary production engineering — the same categories of failure that sink any feature, just wearing an unfamiliar costume because it happens to involve an LLM call.
This post is that gap, specifically for Laravel apps, because the failure modes are concrete and repeatable enough to name: cost that scales linearly with usage and nobody modeled it before launch, latency that was invisible in a demo and unbearable at real traffic, silent quality drift nobody’s watching for, a feature built on a foundation with no plan for what happens when the model changes underneath it, and — the one that kills more shipped AI features than any technical failure — a feature that was genuinely impressive and solved a problem nobody actually had.
Cost Nobody Modeled Before Launch
The demo runs on a laptop, against a handful of test cases, over the course of an afternoon. Nobody’s watching the API bill during a demo, because a few dozen calls cost close to nothing regardless of the model or the provider. The bill becomes a real number the moment the feature has real usage — and the specific way this goes wrong in Laravel apps is almost always the same shape: a feature that calls an LLM per row, per request, or per user action, with no caching layer, built by someone who correctly estimated the per-call cost and never multiplied it by realistic volume.
// The pattern that quietly becomes a five-figure monthly bill
class Product extends Model
{
public function getAiDescriptionAttribute(): string
{
// Called every time this accessor is touched — including in a
// loop over a paginated product list, including on every page
// load, including for products whose description never changes
return Prism::text()
->using(Provider::OpenAI, 'gpt-5')
->withPrompt("Write a product description for: {$this->name}")
->asText()
->text;
}
}
This is a completely ordinary accessor, written the ordinary way an Eloquent accessor gets written — and that ordinariness is exactly the trap. Nobody would write a database-query-per-page-load accessor without noticing the N+1 problem, because N+1 queries are a familiar, named failure mode every Laravel developer’s been taught to watch for. An LLM call inside an accessor is the same structural mistake — an expensive operation triggered on every access, multiplied by however many rows render on a page — but it doesn’t get caught in review nearly as often, because “AI call in a loop” doesn’t pattern-match to “N+1” in most developers’ trained instincts yet, even though it’s mechanically identical and often more expensive per occurrence than the database query it resembles.
The fix is the same fix as any other expensive-operation-in-a-loop problem: generate once, store the result, and treat the LLM call as a write operation that happens at content-creation time, not a read operation that happens on every access. A cached, normalized-hash lookup (covered in prior posts on this exact cost problem) helps for genuinely repeated queries — but the more fundamental fix here is recognizing that “AI-generated content” is content, and content gets generated once and persisted, the same as any other field a user or a background job populates.
Latency That Was Invisible in the Demo
A demo is a controlled environment: one request at a time, a fast connection, a patient audience watching a screen share who already knows the feature involves AI and has mentally budgeted a few extra seconds for it. Production is none of those things — concurrent requests competing for the same rate limits, real users with zero patience for a spinner they didn’t sign up to wait for, and a support-ticket-classification feature (or whatever the AI feature actually is) sitting in the middle of a workflow where a human is now blocked on an API call that used to be instant.
// The demo felt fine. Fifty simultaneous submissions during a traffic spike
// means fifty requests all blocked on the same external API, and Laravel's
// default synchronous request handling means every one of those fifty
// support agents is staring at a spinner for however long OpenAI takes
// that particular moment — which is not a number anyone controls
public function store(Request $request): RedirectResponse
{
$classification = Prism::structured()
->using(Provider::OpenAI, 'gpt-5')
->withSchema($schema)
->withPrompt("Classify: {$request->body}")
->asStructured();
Ticket::create([...]);
return redirect()->route('tickets.index');
}
This is the exact synchronous-call-in-the-request-cycle mistake covered in prior posts on AI integration patterns — worth repeating here specifically because it’s disproportionately common in the first version of a feature, precisely because the first version is usually built and demoed by one person, alone, at a desk, where “it feels fast enough” is actually true, locally, under exactly the load conditions that never repeat once fifty real users hit it during a genuine traffic spike. The fix — queue it, broadcast the result — isn’t a new lesson. It’s the same lesson that’s always applied to any unpredictable external dependency, arriving late because the feature didn’t feel like it needed the lesson until the first time it actually got tested under real concurrent load.
Silent Quality Drift Nobody’s Watching For
A database query either returns the right rows or it doesn’t — correctness is binary, and a broken query usually fails loudly and immediately. An LLM’s output quality is not binary, and a subtle degradation — a provider’s model update changing its behavior slightly, a prompt that was tuned against one model version behaving differently against the version quietly upgraded underneath it, a classification accuracy that drifts a few percentage points over months — produces no error, no exception, no failed test. It just slowly gets worse, and because nothing about “slightly worse quality” trips a monitoring alert built for the errors a normal Laravel app watches for, the first real signal is usually a human noticing a pattern in complaints, weeks after the drift actually started.
// Nothing here fails loudly when quality degrades — worth deliberately
// building a way to notice, since the AI call itself never will
class LogAiClassification
{
public function handle(TicketClassified $event): void
{
AiClassificationLog::create([
'ticket_id' => $event->ticket->id,
'model' => $event->modelUsed,
'classification' => $event->result,
'confidence' => $event->confidence ?? null,
]);
}
}
// A cheap, periodic check worth having specifically because nothing
// else will surface this kind of degradation on its own
class MonitorClassificationDrift extends Command
{
public function handle(): void
{
$recentAgreementRate = AiClassificationLog::query()
->whereNotNull('human_override')
->where('created_at', '>=', now()->subWeek())
->selectRaw('AVG(CASE WHEN classification = human_override THEN 1 ELSE 0 END) as rate')
->value('rate');
if ($recentAgreementRate !== null && $recentAgreementRate < 0.85) {
Log::warning('AI classification agreement rate dropped below threshold', [
'rate' => $recentAgreementRate,
]);
// Notify whoever owns this feature — this is the alert that
// catches quality drift, since nothing about the LLM call
// itself will ever throw an exception for "got slightly worse"
}
}
}
Building this kind of check feels like overhead for a feature that “just works” at launch — which is exactly why it usually doesn’t get built until after the first real quality incident, at which point the team is debugging degraded output with no historical data showing when the drift actually started, only a support queue full of complaints and a hunch about when things started feeling off. The honest lesson: an AI feature needs an observability story from day one, the same way a payment feature does, because both share the property that silent, gradual wrongness is worse than a loud failure, and neither one announces itself the way a database exception does.
No Plan for What Happens When the Model Changes Underneath You
A feature tuned carefully against a specific model version — a prompt refined over weeks to get the classification accuracy right, a temperature setting dialed in through trial and error — is implicitly making a bet that the model stays put. Providers deprecate model versions, sometimes with real advance notice and sometimes with less than a team would like, and a feature with the model name hardcoded in one place, with no test suite verifying behavior against real inputs, discovers the bet was wrong at the worst possible time: the deprecation notice arrives, the swap to the replacement model happens under deadline pressure, and the prompt that was carefully tuned against the old model’s specific behavior now produces subtly different output against the new one, with no test in place that would have caught the regression before it reached production.
// A regression test suite specifically for prompt/model behavior —
// not a nice-to-have, the actual safety net for a model swap
it('classifies a clearly urgent ticket as high priority', function () {
$result = app(TicketClassifier::class)->classify(
'URGENT: payment system is down for all customers, losing revenue right now'
);
expect($result->urgency)->toBe(Urgency::High);
});
it('classifies a routine question as low priority', function () {
$result = app(TicketClassifier::class)->classify(
'What are your business hours on weekends?'
);
expect($result->urgency)->toBe(Urgency::Low);
});
// A representative set of real, anonymized historical tickets, with known-
// correct classifications, run against any candidate model swap BEFORE
// it ships — this is the test suite that catches "the new model is
// subtly worse at this specific thing our prompt depended on"
it('matches expected classifications across a representative ticket sample', function () {
$sample = TicketFixtures::representativeSample();
$accuracy = collect($sample)->map(function ($ticket) {
$result = app(TicketClassifier::class)->classify($ticket->body);
return $result->urgency === $ticket->expectedUrgency;
})->filter()->count() / count($sample);
expect($accuracy)->toBeGreaterThan(0.9);
});
This is the same discipline as any dependency a team doesn’t fully control — pin versions where the provider allows it, maintain a real regression suite against representative inputs, and treat a model swap as a deploy that needs testing, not a config value flipped under deadline pressure the day a deprecation notice arrives. Teams that built this suite from the start swap models calmly. Teams that didn’t discover, during an actual forced migration, that they have no way to verify the new model didn’t quietly break the exact behavior their users depended on.
The Feature Nobody Actually Needed, Executed Well
The failure mode that isn’t really a technical failure at all: an AI feature built because the technology was available and impressive, solving a problem framed around what the model could do rather than what users were actually asking for. This is the hardest one to see coming, because it doesn’t fail on any of the axes above — the cost is modeled correctly, the latency is handled properly, the quality is monitored, the model-swap plan exists — and it still gets ripped out, because usage data eventually shows almost nobody engages with it, and the ongoing cost and maintenance burden of a well-built feature nobody uses is still a cost with no offsetting benefit.
The tell, in hindsight, across most of these: the feature was pitched and prioritized around “we could add AI-powered X” rather than “users are asking for X and AI happens to be the right way to build it.” The order of those two clauses matters more than almost anything covered in the technical sections above — a feature built to solve a real, previously-articulated user need, using AI as the implementation detail, survives scrutiny about whether it’s worth keeping. A feature built because AI made a previously-hard thing suddenly technically possible, with the actual user need reverse-engineered afterward to justify it, is the one that’s genuinely at risk of being correctly identified, six months later, as effort spent on something nobody asked for.
The One Rule
None of the five failure modes here are AI-specific in the way they’re usually discussed — they’re ordinary production engineering failures (uncontrolled cost scaling, blocking latency, silent quality regression, unmanaged third-party dependency risk, and building the wrong thing well) that happen to be easier to miss when the unfamiliar-feeling technology involved makes a team’s normal instincts for “wait, should we cache this” or “wait, should this be queued” temporarily go quiet. The demo always works, because a demo is specifically the condition under which none of these five failure modes have had time or scale to appear yet. The feature that survives being depended on for longer than six months is the one built by a team that applied the same production discipline to the LLM call that they’d have applied automatically to a database query, a third-party API, or a payment provider — and treated “there’s an AI in it” as an implementation detail, not an exemption from the engineering discipline that was never actually about the technology in the first place.