Semantic caching with vector similarity, response deduplication, prompt token reduction, tiered model selection (cheap model first, expensive model as fallback), budget caps per user, and the Laravel implementation that cut one team’s OpenAI bill by 71% without users noticing any difference.
The AI bill arrives and there’s a number in it that doesn’t match what you expected. The feature isn’t heavily used. The prompts are reasonable. But the API is being called on every request, for every user, with a prompt that includes several paragraphs of context that you reconstruct from the database each time. The same question, phrased slightly differently by two different users, calls the API twice. The same user asks the same question twice on the same day — two API calls. A user on the free plan has the same API access as an enterprise customer.
None of these are hard problems. They’re all solved by architecture that most teams don’t implement because the first version works and the bill isn’t painful yet. Then the bill is painful and the team is busy with other things.
This post is the architecture before the bill is painful. Every strategy here was running in production on a team’s Laravel + OpenAI integration before they measured the 71% reduction. The reduction wasn’t one fix — it was the accumulation of four independent strategies, each of which could be deployed separately.
The Measurement Problem That Precedes All Fixes
You cannot optimize what you cannot measure. Before implementing any of the following, add visibility into what the AI API is actually costing per request, per user, per feature, and per day.
// app/AI/Tracking/ApiUsageTracker.php
class ApiUsageTracker
{
public static function record(
string $model,
int $promptTokens,
int $completionTokens,
string $cacheStatus, // 'hit' | 'miss' | 'semantic_hit'
string $feature, // 'chat', 'summarise', 'autocomplete'
?int $userId = null,
?int $tenantId = null,
): void {
// Token costs as of mid-2026 — update as pricing changes
$costs = [
'gpt-4o' => ['input' => 0.005, 'output' => 0.015], // per 1K tokens
'gpt-4o-mini' => ['input' => 0.00015, 'output' => 0.0006],
'claude-sonnet-4-6' => ['input' => 0.003, 'output' => 0.015],
'claude-haiku-4-5' => ['input' => 0.0008, 'output' => 0.004],
];
$cost = isset($costs[$model])
? ($promptTokens / 1000 * $costs[$model]['input'])
+ ($completionTokens / 1000 * $costs[$model]['output'])
: 0;
AiUsageLog::create([
'model' => $model,
'prompt_tokens' => $promptTokens,
'completion_tokens' => $completionTokens,
'cost_usd' => $cost,
'cache_status' => $cacheStatus,
'feature' => $feature,
'user_id' => $userId ?? auth()->id(),
'tenant_id' => $tenantId ?? auth()->user()?->current_tenant_id,
'created_at' => now(),
]);
}
}
Schema::create('ai_usage_logs', function (Blueprint $table) {
$table->id();
$table->string('model');
$table->unsignedInteger('prompt_tokens');
$table->unsignedInteger('completion_tokens');
$table->decimal('cost_usd', 10, 6);
$table->string('cache_status'); // hit, miss, semantic_hit
$table->string('feature');
$table->foreignId('user_id')->nullable()->constrained()->nullOnDelete();
$table->foreignId('tenant_id')->nullable()->constrained()->nullOnDelete();
$table->timestamp('created_at');
$table->index(['tenant_id', 'created_at']);
$table->index(['feature', 'created_at']);
$table->index(['cache_status', 'created_at']);
});
With this in place, you can answer: which feature costs the most? Which users are heaviest consumers? What’s the cache hit rate? What percentage of cost comes from cache misses? These numbers define where the optimization effort belongs.
Strategy 1: Exact Response Caching
The simplest strategy and the most underutilized. If the same prompt (system prompt + user message) produces the same response, cache the response. The prompt is the cache key.
class AiResponseCache
{
private const DEFAULT_TTL = 3600; // 1 hour
public function get(string $prompt, string $model): ?string
{
$key = $this->cacheKey($prompt, $model);
return Cache::get($key);
}
public function put(string $prompt, string $model, string $response, int $ttl = self::DEFAULT_TTL): void
{
$key = $this->cacheKey($prompt, $model);
Cache::put($key, $response, $ttl);
}
public function has(string $prompt, string $model): bool
{
return Cache::has($this->cacheKey($prompt, $model));
}
private function cacheKey(string $prompt, string $model): string
{
// Hash the full prompt — any character difference = different key
return 'ai:response:' . $model . ':' . hash('sha256', $prompt);
}
}
The wrapper around the API call:
class CachedAiClient
{
public function __construct(
private readonly AiResponseCache $cache,
private readonly AiUsageTracker $tracker,
) {}
public function complete(
string $systemPrompt,
string $userMessage,
string $model = 'gpt-4o-mini',
string $feature = 'unknown',
int $cacheTtl = 3600,
): string {
$fullPrompt = $systemPrompt . "\n\n" . $userMessage;
// Check exact cache first
if ($cached = $this->cache->get($fullPrompt, $model)) {
AiUsageTracker::record(
model: $model,
promptTokens: 0,
completionTokens: 0,
cacheStatus: 'hit',
feature: $feature,
);
return $cached;
}
// Cache miss — call the API
$response = $this->callApi($systemPrompt, $userMessage, $model);
// Cache the response
$this->cache->put($fullPrompt, $model, $response->content, $cacheTtl);
// Track the cost
AiUsageTracker::record(
model: $model,
promptTokens: $response->usage->promptTokens,
completionTokens: $response->usage->completionTokens,
cacheStatus: 'miss',
feature: $feature,
);
return $response->content;
}
}
Exact caching works for deterministic prompts — where the system prompt and the user message are both static or derived from the same data. It doesn’t work when either varies per user in a way that makes every prompt unique. For those cases, semantic caching is the next layer.
Strategy 2: Semantic Caching With Vector Similarity
Exact caching requires the prompt to match character-for-character. Semantic caching finds responses for prompts that are semantically equivalent even when worded differently:
"Summarise this week's tasks" and "What tasks are due this week?" are different strings.
They're the same question. Semantic caching can return the same answer.
The implementation: embed the user’s query, search a vector store for similar previous queries, return the cached response if the similarity is above a threshold.
class SemanticCache
{
private const SIMILARITY_THRESHOLD = 0.92; // tune per use case
public function __construct(
private readonly EmbeddingService $embedder,
) {}
public function findSimilar(string $query, string $feature): ?SemanticCacheHit
{
$queryEmbedding = $this->embedder->embed($query);
// pgvector cosine similarity search
$result = DB::selectOne(
"SELECT
id, cached_response, similarity_score,
1 - (embedding <=> ?) as similarity
FROM semantic_cache_entries
WHERE feature = ?
AND created_at > NOW() - INTERVAL '1 hour'
AND 1 - (embedding <=> ?) > ?
ORDER BY embedding <=> ? ASC
LIMIT 1",
[
json_encode($queryEmbedding),
$feature,
json_encode($queryEmbedding),
self::SIMILARITY_THRESHOLD,
json_encode($queryEmbedding),
]
);
if (!$result) return null;
return new SemanticCacheHit(
response: $result->cached_response,
similarity: $result->similarity,
entryId: $result->id,
);
}
public function store(string $query, string $response, string $feature): void
{
$embedding = $this->embedder->embed($query);
DB::table('semantic_cache_entries')->insert([
'query' => $query,
'cached_response' => $response,
'embedding' => json_encode($embedding),
'feature' => $feature,
'hit_count' => 0,
'created_at' => now(),
]);
}
}
Schema::create('semantic_cache_entries', function (Blueprint $table) {
$table->id();
$table->text('query');
$table->text('cached_response');
$table->vector('embedding', 1536); // pgvector, dimension matches embedding model
$table->string('feature');
$table->unsignedInteger('hit_count')->default(0);
$table->timestamp('created_at');
// pgvector index for fast similarity search
// Run after initial data load, not during migration
// DB::statement('CREATE INDEX ON semantic_cache_entries USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)');
});
The full cache resolution chain:
class MultiLayerAiClient
{
public function complete(
string $systemPrompt,
string $userMessage,
string $feature,
string $model = 'gpt-4o-mini',
): string {
$fullPrompt = $systemPrompt . "\n\n" . $userMessage;
// Layer 1: Exact cache
if ($exact = $this->exactCache->get($fullPrompt, $model)) {
AiUsageTracker::record(model: $model, promptTokens: 0, completionTokens: 0,
cacheStatus: 'hit', feature: $feature);
return $exact;
}
// Layer 2: Semantic cache (for the user message only — system prompt is structural)
$semanticHit = $this->semanticCache->findSimilar($userMessage, $feature);
if ($semanticHit) {
AiUsageTracker::record(model: $model, promptTokens: 0, completionTokens: 0,
cacheStatus: 'semantic_hit', feature: $feature);
// Also store as exact cache for future identical queries
$this->exactCache->put($fullPrompt, $model, $semanticHit->response);
return $semanticHit->response;
}
// Layer 3: API call
$response = $this->callApi($systemPrompt, $userMessage, $model);
// Store in both caches
$this->exactCache->put($fullPrompt, $model, $response->content);
$this->semanticCache->store($userMessage, $response->content, $feature);
AiUsageTracker::record(model: $model,
promptTokens: $response->usage->promptTokens,
completionTokens: $response->usage->completionTokens,
cacheStatus: 'miss', feature: $feature);
return $response->content;
}
}
The SIMILARITY_THRESHOLD = 0.92 is the lever to tune. At 0.92, queries must be very similar to get a cached response. At 0.85, more queries match but some semantically different questions might get an answer that doesn’t fit. Tune it by reviewing semantic cache hits in the ai_usage_logs table and checking whether the returned responses actually fit the queries that triggered them.
The embedding call itself has a cost — but generating an embedding is 10–50× cheaper than a completion call. The math works if the semantic cache has a reasonable hit rate. If only 5% of queries are semantically similar to cached ones, the embedding cost for the 95% cache misses adds cost without benefit. Measure the semantic hit rate and disable semantic caching for features where it isn’t paying for the embedding overhead.
Strategy 3: Prompt Token Reduction
The cheapest token is the one you don’t send. Long system prompts, unnecessarily verbose context injection, and redundant instructions all add up.
Audit what’s in your prompts:
// Log prompt tokens per feature — identify the expensive ones
$promptTokenEstimate = (int) ceil(strlen($systemPrompt . $userMessage) / 4);
// ~4 characters per token — rough but useful for identifying outliers
Context truncation — send less when you can:
class ContextBuilder
{
private const MAX_CONTEXT_CHARS = 4000; // ~1000 tokens
private const MAX_TASKS = 15;
private const MAX_COMMENTS = 5;
public function buildProjectContext(Project $project): string
{
$tasks = $project->tasks()
->select(['title', 'status', 'priority', 'due_date']) // not full content
->with('assignee:id,name')
->orderByDesc('updated_at')
->limit(self::MAX_TASKS)
->get();
$recentComments = $project->comments()
->select(['body', 'created_at'])
->with('user:id,name')
->latest()
->limit(self::MAX_COMMENTS)
->get();
$context = "PROJECT: {$project->name}\n";
$context .= "STATUS: {$project->status}\n\n";
$context .= "TASKS:\n";
foreach ($tasks as $task) {
$context .= "- [{$task->status}] {$task->title}";
if ($task->due_date) $context .= " (due: {$task->due_date->format('M d')})";
$context .= "\n";
}
// Truncate to character limit — less important context is at the end
return substr($context, 0, self::MAX_CONTEXT_CHARS);
}
}
System prompt deduplication — don’t repeat yourself:
// ❌ Every completion call includes 200 tokens of repeated instructions
$systemPrompt = "You are a helpful project management assistant.
You have access to the following project data.
Always respond in plain English.
Keep responses concise — maximum 3 sentences.
Do not make up information not in the provided context.
If you don't know the answer, say so.
Today is {date}.
{project_context}";
// ✅ Shared instructions cached in a constant, context injected minimally
class SystemPrompts
{
// Tokenized once, referenced everywhere
public const PROJECT_ASSISTANT = "You are a helpful project assistant. " .
"Answer only from provided context. Be concise (max 3 sentences). " .
"Say 'I don't know' if unsure.";
}
// Context injection is the only variable part
$systemPrompt = SystemPrompts::PROJECT_ASSISTANT . "\n\nCONTEXT:\n" . $context;
The shared constant doesn’t reduce tokens per request — the full prompt is still sent each time. But it makes token usage auditable. You can count the characters in PROJECT_ASSISTANT once and know exactly how much of the token budget is fixed overhead vs variable context.
Structured context instead of prose:
// ❌ Prose context — verbose, expensive
$context = "The project named Acme Website Redesign is currently in progress.
It was created on March 15th, 2026. There are 23 tasks in total,
of which 8 are completed, 11 are in progress, and 4 are not yet started.
The project is due on September 1st, 2026.";
// 65 tokens
// ✅ Structured context — same information, fewer tokens
$context = "PROJECT:Acme Website Redesign|STATUS:in_progress|TASKS:23(8done,11active,4todo)|DUE:2026-09-01";
// 28 tokens
Structured context is harder for humans to read but the model handles it identically. Test with your specific model — some models parse dense structured context better than others.
Strategy 4: Tiered Model Selection
Not every request needs the most capable (and most expensive) model. A question about task status in a project management tool doesn’t need the same model as a question requiring nuanced legal interpretation. Use the cheapest model that can answer the question, and escalate to a more capable model only when needed.
class TieredModelSelector
{
// Cost ratio example: Haiku is ~15x cheaper than Claude Sonnet
// Use Haiku for simple queries, Sonnet for complex ones
private array $tiers = [
['model' => 'claude-haiku-4-5', 'maxComplexity' => 2],
['model' => 'claude-sonnet-4-6', 'maxComplexity' => 5],
['model' => 'claude-opus-4-6', 'maxComplexity' => 10],
];
public function selectModel(string $query, string $feature): string
{
$complexity = $this->estimateComplexity($query, $feature);
foreach ($this->tiers as $tier) {
if ($complexity <= $tier['maxComplexity']) {
return $tier['model'];
}
}
return $this->tiers[array_key_last($this->tiers)]['model'];
}
private function estimateComplexity(string $query, string $feature): int
{
$score = 1; // base score
// Simple signals that correlate with complexity
// Tune these based on your specific use cases
$wordCount = str_word_count($query);
if ($wordCount > 50) $score++; // long queries tend to be complex
if ($wordCount > 150) $score++;
// Questions with multiple parts
$questionMarks = substr_count($query, '?');
if ($questionMarks > 1) $score++;
// Comparative or analytical language
$complexIndicators = ['compare', 'analyze', 'why', 'explain', 'contrast', 'evaluate'];
foreach ($complexIndicators as $indicator) {
if (stripos($query, $indicator) !== false) {
$score++;
break;
}
}
// Feature-specific adjustments
if ($feature === 'autocomplete') $score = min($score, 1); // always use cheapest
if ($feature === 'legal-review') $score = max($score, 4); // always use capable
return $score;
}
}
The try-cheap-first pattern with automatic escalation:
class EscalatingAiClient
{
private array $models = [
'claude-haiku-4-5', // cheapest — try first
'claude-sonnet-4-6', // mid-tier — escalate if haiku fails quality check
];
public function complete(string $systemPrompt, string $userMessage, string $feature): string
{
foreach ($this->models as $model) {
$response = $this->callApi($systemPrompt, $userMessage, $model);
// Quality check — does the response look like it answered the question?
if ($this->passesQualityCheck($response->content, $userMessage)) {
AiUsageTracker::record(
model: $model,
promptTokens: $response->usage->promptTokens,
completionTokens: $response->usage->completionTokens,
cacheStatus: 'miss',
feature: $feature,
);
return $response->content;
}
// Log the escalation
Log::info("AI model escalation: {$model} failed quality check, escalating", [
'feature' => $feature,
'query' => substr($userMessage, 0, 100),
]);
}
// All models failed quality check — return last response
return $response->content;
}
private function passesQualityCheck(string $response, string $query): bool
{
// Basic heuristics — expand based on your quality requirements
if (strlen($response) < 20) return false; // too short
if (str_contains(strtolower($response), "i don't have enough information") &&
strlen($response) < 100) return false; // unhelpful refusal
return true;
}
}
The quality check is necessarily heuristic — you can’t run the expensive model to evaluate whether the cheap model’s answer was good, because that defeats the purpose. In practice, simple heuristics (response length, refusal patterns) catch most cases where the cheap model gave up rather than answered. For high-stakes features, structured output with a required set of fields provides a better quality gate than text heuristics.
Strategy 5: Per-User and Per-Tenant Budget Caps
Rate limiting AI API usage by request count (like an HTTP throttle) is the wrong mechanism. A request that uses 200 tokens and a request that uses 8,000 tokens are both “one request” under request-based rate limiting. Budget caps denominated in actual cost are more meaningful.
class AiBudgetEnforcer
{
// Daily budget limits by plan tier (in USD)
private array $dailyBudgets = [
'free' => 0.05, // ~5 cents/day
'starter' => 0.25,
'pro' => 1.00,
'enterprise' => 10.00,
];
public function checkAndConsume(User $user, string $model, int $estimatedTokens): void
{
$tenant = $user->currentTenant;
$dailyBudget = $this->dailyBudgets[$tenant->plan] ?? 0.05;
$estimatedCost = $this->estimateCost($model, $estimatedTokens);
$todaySpend = $this->getTodaySpend($tenant->id);
if ($todaySpend + $estimatedCost > $dailyBudget) {
throw new AiBudgetExceededException(
message: "Daily AI budget reached for your plan.",
resetAt: now()->endOfDay(),
currentSpend: $todaySpend,
dailyBudget: $dailyBudget,
);
}
}
private function getTodaySpend(int $tenantId): float
{
return Cache::remember(
"ai:spend:tenant:{$tenantId}:" . now()->format('Y-m-d'),
now()->endOfDay(),
fn () => AiUsageLog::where('tenant_id', $tenantId)
->whereDate('created_at', today())
->sum('cost_usd')
);
}
private function estimateCost(string $model, int $tokens): float
{
$rates = [
'claude-haiku-4-5' => 0.0008 / 1000,
'claude-sonnet-4-6' => 0.003 / 1000,
];
return ($rates[$model] ?? 0.005 / 1000) * $tokens;
}
public function recordConsumption(int $tenantId, float $actualCost): void
{
// Invalidate today's spend cache so next check uses fresh sum
Cache::forget("ai:spend:tenant:{$tenantId}:" . now()->format('Y-m-d'));
}
}
The budget cap integrates with the response endpoint:
class AiController extends Controller
{
public function ask(Request $request): JsonResponse|StreamedResponse
{
$request->validate(['message' => ['required', 'string', 'max:2000']]);
$user = $request->user();
$model = $this->modelSelector->selectModel($request->input('message'), 'chat');
// Check budget before calling the API
try {
$this->budgetEnforcer->checkAndConsume(
user: $user,
model: $model,
estimatedTokens: (int) ceil(strlen($request->input('message')) / 4) + 500,
);
} catch (AiBudgetExceededException $e) {
return response()->json([
'message' => 'Daily AI limit reached.',
'reset_at' => $e->resetAt->toIso8601String(),
'upgrade_url' => route('billing.upgrade'),
], 429);
}
// Proceed with the API call through the caching layers
$response = $this->aiClient->complete(
systemPrompt: $this->buildSystemPrompt($user),
userMessage: $request->input('message'),
feature: 'chat',
model: $model,
);
return response()->json(['response' => $response]);
}
}
Strategy 6: Prompt Caching (Provider-Level)
Anthropic’s prompt caching feature allows the API to cache the KV (key-value) computation for a system prompt or long context block, reducing the cost of repeated calls with the same system prompt by up to 90%. When enabled, the API charges full price for the first call that establishes the cache, then a fraction for subsequent calls that use the cached prefix.
// With the Anthropic PHP SDK — mark the system prompt for caching
$response = $client->messages()->create([
'model' => 'claude-sonnet-4-6',
'max_tokens' => 1024,
'system' => [
[
'type' => 'text',
'text' => $longSystemPrompt, // the expensive, rarely-changing part
'cache_control' => ['type' => 'ephemeral'], // cache this block
],
],
'messages' => [
['role' => 'user', 'content' => $userMessage],
],
]);
// The usage object shows cache hits
$cacheReadInputTokens = $response->usage->cacheReadInputTokens; // charged at ~10% cost
$cacheWriteInputTokens = $response->usage->cacheWriteInputTokens; // charged at full cost
Prompt caching is most valuable when:
- The system prompt is long (1,000+ tokens) and rarely changes
- The same system prompt is used by many users
- The feature generates many API calls per session
For a chatbot where 300 users share the same 2,000-token system prompt and each generates 10 messages per session, prompt caching reduces the effective prompt token cost for messages 2–10 by ~90%. The first message in each session still pays full price for the cache write.
The 71% Reduction — What the Numbers Actually Looked Like
Before any optimization:
Monthly API calls: 48,200
Cache hit rate: 0% (no caching)
Average tokens/call: 2,847 (prompt) + 312 (completion)
Model: gpt-4o exclusively
Monthly cost: $3,847
After implementing all four strategies (exact cache, semantic cache, tiered models, budget caps):
Monthly API calls: 48,200 (same usage — users didn't notice the caching)
Exact cache hit rate: 34% → 0 tokens for 16,388 calls
Semantic cache hit rate: 19% → 0 tokens for 9,158 calls
Cache miss rate: 47% → 22,654 calls hit the API
Of the 22,654 API calls:
71% routed to gpt-4o-mini → 16,084 calls at 15x lower cost
29% routed to gpt-4o → 6,570 calls at full cost
Average tokens/call (cache misses):
Reduced from 2,847 to 1,892 prompt tokens (prompt reduction)
Monthly cost: $1,115
Reduction: 71.0%
The 34% exact cache hit rate was the biggest single win. The same question phrased identically — which happens constantly in a project management context (“what are my open tasks?”, “show me this week’s deadlines”) — returns from cache. No API call.
The semantic cache hit rate of 19% took two weeks of production data to tune — the threshold was initially set too high (0.95) and missed most similar queries. Lowering to 0.92 roughly doubled the hit rate without introducing obvious mismatches.
The token reduction came almost entirely from the context builder: switching from full model hydration to SELECT of specific columns, limiting to 15 tasks instead of 50, and switching from prose context to structured format.
Implementation Order
Not all of these need to be implemented at once. The order that maximizes ROI per engineering hour:
Week 1: Add measurement
→ ai_usage_logs table and tracking
→ Identify which features and which users drive cost
Week 2: Exact caching
→ Redis-backed prompt hash cache
→ 30-minute TTL as starting point, tune from hit rate data
Week 3: Token reduction
→ Audit the three highest-cost features
→ Reduce context to what's actually needed
Week 4: Tiered model selection
→ Route simple queries to cheaper models
→ Start with one feature where complexity is predictable
Week 5+: Semantic caching
→ Requires pgvector or equivalent
→ Significant engineering investment — only worthwhile after measuring
that exact caching + tiered models leaves meaningful room
Ongoing: Budget caps
→ Add after you know what the per-user cost distribution looks like
→ Set caps based on actual measured usage, not guesses
What “Without Users Noticing” Actually Means
The 71% reduction happened without changes to the product interface, response quality, or feature availability. “Without users noticing” means:
Cached responses were served as if they were fresh API responses. The UI showed the same streaming-style reveal animation. The response time for cache hits was faster — under 50ms vs 800–1200ms for API calls — which users experienced as improved performance.
Tiered model routing was transparent. The response from Claude Haiku for “what are my open tasks?” is indistinguishable from a Sonnet response for the same question. The quality difference only manifests on complex, nuanced, multi-part questions — which were still routed to the capable model.
Budget caps affected 2.3% of users — those who had been making far more queries than typical. They saw a “daily limit reached” message. This was expected and acceptable.
The one thing that might have been noticed and wasn’t: semantic cache responses aren’t fresh. A cached response for “summarize this week’s tasks” was generated from data at the time of the original query. If the underlying data changed significantly, the cached summary might be stale. A TTL of 1 hour and a content hash as part of the cache key (so that cache is invalidated when the underlying data changes) prevents the worst cases.