Horizontal scaling, Redis queue distribution, Octane with FrankenPHP, database read replicas, CloudFront CDN, rate limiting per endpoint, and the Pulse alerts that fired at 11pm so I could sleep — a real architecture story from a real traffic event with real numbers.
The year before last Black Friday, the app went down at 2:14pm. Not a graceful degradation — a hard crash. PHP-FPM workers saturated, MySQL connections exhausted, the queue backed up to 40,000 jobs, and the first three minutes of the biggest traffic window of the year produced zero successful checkouts. The incident lasted 22 minutes. The revenue impact was measurable and uncomfortable.
The year after, the same traffic hit an architecture that had been rebuilt to absorb it. Peak concurrency was 4.3× higher than the previous year. Response time at peak was 94ms average. Zero application errors. One Pulse alert fired at 11:03pm — a non-critical queue depth warning — and I was back asleep by 11:07pm.
This post is the architecture that made the second year work, with the specific numbers from both events so the before/after is concrete.
The Numbers From Both Years
Year 1 (crash):
Peak concurrent users: 847
Peak requests/minute: 4,200
PHP-FPM workers: 24 (single server)
MySQL connections at peak: 148 / 150 (99% saturated)
Queue depth at peak: 40,847 jobs
Response time at peak: timeout (503s)
Downtime: 22 minutes
Checkout errors during peak: 100%
Year 2 (no errors):
Peak concurrent users: 3,640 (4.3× higher)
Peak requests/minute: 18,200 (4.3× higher)
PHP-FPM workers: 0 (Octane, persistent workers)
MySQL connections at peak: 31 / 200 (15% utilised)
Queue depth at peak: 2,847 jobs
Average response time: 94ms
Downtime: 0 minutes
Checkout errors during peak: 0
The most significant difference isn’t traffic volume — it’s that MySQL connections went from 99% saturated to 15% utilised despite 4.3× the traffic. That’s the read replica configuration at work.
Layer 1: Octane With FrankenPHP
The first-year crash started with PHP-FPM worker exhaustion. PHP-FPM spawns a process per request. Under high concurrency, worker pool saturation means new requests queue, then time out, then return 502/503. Adding more workers helps until memory limits the process count.
Octane with FrankenPHP eliminates the per-request process overhead entirely. FrankenPHP is a PHP runtime built on Go and Caddy that runs your Laravel application in persistent workers — the application boots once, handles thousands of requests in the same process, and never pays the bootstrap cost per request.
# Install Octane with FrankenPHP
composer require laravel/octane
php artisan octane:install --server=frankenphp
# Run in production
php artisan octane:frankenphp --workers=8 --max-requests=500
The --workers=8 value is set to the number of CPU cores, not the number of expected concurrent users. Octane workers are event-loop-driven, not process-per-request. 8 workers on an 8-core server can handle hundreds of concurrent requests if those requests are I/O-bound (database queries, Redis calls, API calls), because each worker handles a request asynchronously.
The --max-requests=500 value recycles each worker after 500 requests. This prevents memory leaks in application code from accumulating indefinitely. After 500 requests, the worker gracefully exits and a new one starts, ensuring memory usage stays bounded.
The critical Octane configuration for a stateful application:
// config/octane.php
return [
'server' => 'frankenphp',
'workers' => env('OCTANE_WORKERS', 8),
// Flush these between requests
// Anything stateful that could leak between requests
'flush' => [
\Illuminate\Auth\AuthManager::class,
\Illuminate\Cache\RateLimiter::class,
\Illuminate\Http\Request::class,
],
// Warm these on worker boot — expensive to create, cheap to reuse
'warm' => [
\App\Services\TenantService::class,
],
];
The flush array is the most important Octane configuration for correctness. Anything that holds per-request state that must not persist to the next request must be in this list. Authentication state, the current request, rate limiter state — all of these need to be reset between requests. Missing an entry from flush causes state from one request to bleed into another, which produces bugs that are extremely hard to reproduce in development.
Memory profiling before deploying Octane:
# Run the memory leak detector before Black Friday
php artisan octane:start --workers=1
ab -n 10000 -c 10 http://localhost:8000/api/products
# Monitor memory per worker
watch -n1 'ps aux | grep "octane\|frankenphp" | grep -v grep'
If worker memory grows unboundedly over 10,000 requests, there’s a leak. Common sources: static class properties, event listeners that accumulate, singletons that append to internal collections. Fix these before deploying Octane to production.
Layer 2: Horizontal Scaling With Load Balancing
One server cannot absorb 4.3× traffic growth without vertical limits becoming a ceiling. The architecture uses AWS Elastic Load Balancer distributing traffic across three application servers in an Auto Scaling Group.
CloudFront
│
Application Load Balancer
/ | \
App-01 App-02 App-03
(8 CPU) (8 CPU) (8 CPU)
Octane Octane Octane
The Auto Scaling configuration:
# Auto Scaling policy — add a server when CPU > 70% for 2 minutes
ScalingPolicy:
Type: TargetTrackingScaling
TargetValue: 70.0
PredefinedMetricType: ASGAverageCPUUtilization
ScaleInCooldown: 300 # 5 minutes before scaling down
ScaleOutCooldown: 60 # 1 minute before scaling up
# Min 2, Max 8 servers
MinSize: 2
MaxSize: 8
DesiredCapacity: 3 # normal day
For Black Friday specifically, the desired capacity was manually set to 5 the night before, ensuring the additional servers were warm and ready before traffic arrived. Auto scaling handles unexpected surges; pre-warming handles known events.
Sticky sessions are disabled. With Octane’s in-memory application state, sticky sessions would mean a server restart loses all users routed to that server. Instead, all session state is stored in Redis:
// config/session.php
'driver' => 'redis',
'connection' => 'session', // dedicated Redis connection for sessions
// config/database.php
'redis' => [
'session' => [
'url' => env('REDIS_SESSION_URL'),
'host' => env('REDIS_SESSION_HOST', '127.0.0.1'),
'port' => env('REDIS_SESSION_PORT', '6379'),
'database' => env('REDIS_SESSION_DB', '2'),
],
],
With sessions in Redis, any server can handle any request from any user. The load balancer distributes round-robin without session affinity.
Layer 3: Database Read Replicas
The first-year crash was caused partly by MySQL connection exhaustion. With 24 PHP-FPM workers on one server holding connections, 150 connection limit was hit instantly under peak load.
The second-year architecture uses three database nodes: one primary (writes), two read replicas (reads). Laravel’s database layer routes writes to the primary and reads to replicas automatically:
// config/database.php
'connections' => [
'mysql' => [
'write' => [
'host' => env('DB_WRITE_HOST'),
],
'read' => [
['host' => env('DB_READ_HOST_1')],
['host' => env('DB_READ_HOST_2')],
],
'sticky' => true, // read your own writes within a request
'driver' => 'mysql',
'database' => env('DB_DATABASE'),
'username' => env('DB_USERNAME'),
'password' => env('DB_PASSWORD'),
],
],
The sticky option ensures that if a request writes to the primary, subsequent reads in the same request also go to the primary. This prevents the read-after-write inconsistency where a user creates an order and immediately sees a “no orders” page because the read replica hasn’t caught up yet.
For product catalog queries, price lookups, and search — the bulk of read traffic on Black Friday — reads are distributed across two replicas. Each replica can handle the full read load independently, so if one fails, the other absorbs everything automatically.
Connection pooling with PgBouncer/ProxySQL:
App servers → ProxySQL → Primary MySQL (writes)
↘ Read Replica 1 (reads)
↘ Read Replica 2 (reads)
ProxySQL sits between the application and MySQL and pools connections. Instead of each Octane worker maintaining a persistent database connection, connections are pooled and reused. Three application servers × 8 Octane workers × 2 connections each = 48 potential connections, but ProxySQL serves them from a pool of 30, because at any moment most workers are not in a database call.
This is why MySQL connections at peak were 31/200 (15%) rather than exhausted. The connection pool absorbed the per-worker connection overhead.
Layer 4: Redis Queue Distribution
The first year’s 40,847-job queue backlog happened because a single queue worker was processing events synchronously on the same server handling web requests. When web traffic consumed all server resources, queue processing stalled, and jobs backed up.
The second-year architecture separates queue workers onto dedicated servers:
App servers (web traffic) → Redis → Queue servers (dedicated)
App-01, App-02, App-03 Queue-01, Queue-02
// config/queue.php
'connections' => [
'redis' => [
'driver' => 'redis',
'connection' => 'queue',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
],
Queue priority configuration for Black Friday:
// routes/console.php — queue worker process configuration
// Queue-01: handles payment and order jobs with highest priority
// Queue-02: handles notifications and analytics with lower priority
// High-priority queue — payment processing, inventory updates
// These must never stall, even under load
Schedule::command('queue:work redis --queue=payments,orders --sleep=0 --tries=3 --timeout=30')
->everyMinute()
->onOneServer();
// Standard queue — emails, notifications, analytics events
Schedule::command('queue:work redis --queue=default,notifications --sleep=1 --tries=3')
->everyMinute()
->onOneServer();
In practice, Horizon manages the queue workers rather than Artisan commands directly:
// config/horizon.php — Black Friday configuration
'environments' => [
'production' => [
// Critical path — payment processing
'supervisor-payments' => [
'connection' => 'redis',
'queue' => ['payments', 'orders', 'inventory'],
'balance' => 'simple',
'processes' => 20,
'tries' => 3,
'timeout' => 30,
'memory' => 256,
],
// Standard async — emails, webhooks, analytics
'supervisor-default' => [
'connection' => 'redis',
'queue' => ['default', 'notifications', 'analytics'],
'balance' => 'auto',
'minProcesses' => 5,
'maxProcesses' => 40,
'tries' => 3,
'timeout' => 60,
'memory' => 128,
],
],
],
Horizon’s balance: auto on the standard supervisor lets it allocate workers between queues based on depth. When notification emails are backed up, it adds workers to the notifications queue. When they drain, it redistributes. This prevented the queue from getting stuck behind one category of jobs.
Layer 5: CloudFront CDN
Static assets (JS, CSS, images) were served from S3 via CloudFront before the first Black Friday. Product images, marketing assets, and the compiled frontend — none of those hit the application server.
For the second year, the CDN layer was extended to cache API responses for unauthenticated requests:
// Cacheable API responses — product catalog, category listings
class ProductController extends Controller
{
public function index(Request $request): JsonResponse
{
// Cache at the CDN layer for authenticated users only by not caching if auth header present
$response = response()->json($this->getProducts($request));
if (!$request->bearerToken()) {
// Unauthenticated: CDN can cache this response
$response->header('Cache-Control', 'public, max-age=60, s-maxage=300');
// s-maxage=300: CDN caches for 5 minutes
// max-age=60: browser caches for 1 minute
} else {
// Authenticated: no caching (user-specific data)
$response->header('Cache-Control', 'private, no-cache');
}
return $response;
}
}
The CloudFront behaviour configuration:
CloudFront Behaviour:
Path: /api/products*
Cache Policy:
Cache based on headers: Authorization
- Request with Authorization: → cache miss, forward to origin
- Request without Authorization: → serve from CloudFront cache
TTL: Min=0, Default=60, Max=300
Compress: Yes
Viewer Protocol: HTTPS only
During Black Friday, 73% of product listing API requests were served from CloudFront without touching the application server. The “browse phase” — when millions of users are looking at products before they decide to buy — was handled entirely at the CDN edge. Only add-to-cart and checkout requests reached the application.
Cache invalidation on price changes:
// When a product price updates, invalidate the CDN cache
class ProductPriceUpdated
{
public function handle(Product $product): void
{
// Invalidate the specific product's cache path
CloudFront::createInvalidation([
'Paths' => [
'Quantity' => 2,
'Items' => [
"/api/products/{$product->id}",
"/api/products/{$product->slug}",
],
],
]);
}
}
Layer 6: Per-Endpoint Rate Limiting
The first year had one rate limit: 60 requests per minute per IP. This protected against obvious abuse but did nothing to prevent legitimate users from accidentally hammering expensive endpoints.
The second year added per-endpoint rate limiting calibrated to what each endpoint actually cost:
// AppServiceProvider::boot()
use Illuminate\Cache\RateLimiting\Limit;
// Product search — database-intensive, limit more aggressively
RateLimiter::for('search', function (Request $request) {
return $request->user()
? Limit::perMinute(30)->by($request->user()->id)
: Limit::perMinute(10)->by($request->ip());
});
// Add to cart — write operation, moderate limit
RateLimiter::for('cart', function (Request $request) {
return Limit::perMinute(60)->by($request->user()?->id ?? $request->ip());
});
// Checkout — expensive, rate limit tightly
RateLimiter::for('checkout', function (Request $request) {
return [
Limit::perMinute(5)->by($request->user()?->id ?? $request->ip()),
Limit::perHour(20)->by($request->user()?->id ?? $request->ip()),
];
});
// Product listings — served from CDN, but still limit the origin
RateLimiter::for('catalog', function (Request $request) {
return $request->user()
? Limit::perMinute(120)->by($request->user()->id)
: Limit::perMinute(30)->by($request->ip());
});
The checkout rate limiter uses two limits: 5 per minute (burst protection) and 20 per hour (prevents automated checkout scripting over a longer window). Legitimate users don’t check out more than 20 times in an hour. Bots and bad actors do.
Layer 7: Laravel Pulse Monitoring
Pulse provided real-time visibility into what was happening during the event. The dashboard showed:
Pulse dashboard metrics (Black Friday peak, 3:00–4:00pm):
Requests/minute: 18,240
Slow queries (>100ms): 14 (0.07% of requests)
Exceptions: 0
Queue depth (default): 2,847
Queue depth (payments): 23
Octane memory (App-01): 284MB / 512MB
Octane memory (App-02): 291MB / 512MB
Octane memory (App-03): 278MB / 512MB
Cache hit rate: 94%
The Pulse alert that fired at 11:03pm:
// config/pulse.php — alert configuration
'thresholds' => [
'queue_depth' => [
'default' => 5000, // alert when default queue exceeds 5000 jobs
'payments' => 100, // alert when payments queue exceeds 100 jobs
],
],
At 11:03pm the default queue hit 5,200 jobs — a batch of notification emails queued after a flash sale announcement. Not a critical alert (payments queue was at 23), but Pulse sent a Slack notification. I looked at the Horizon dashboard on my phone, saw the default queue clearing at normal rate, and went back to sleep. By 11:07pm it was below 1,000.
The Pulse recorder configuration for Black Friday:
// config/pulse.php
'recorders' => [
\Laravel\Pulse\Recorders\Exceptions::class => [
'enabled' => true,
'sample_rate' => 1, // record all exceptions
],
\Laravel\Pulse\Recorders\Queues::class => [
'enabled' => true,
'queues' => ['payments', 'orders', 'default', 'notifications'],
],
\Laravel\Pulse\Recorders\SlowRequests::class => [
'enabled' => true,
'sample_rate' => 0.1, // sample 10% of requests
'threshold' => 1000, // flag requests over 1 second
],
\Laravel\Pulse\Recorders\SlowQueries::class => [
'enabled' => true,
'threshold' => 100, // flag queries over 100ms
],
],
Sampling at 10% for slow request tracking keeps the Pulse database from becoming a bottleneck during peak traffic. At 18,000 requests/minute, sampling 100% would generate 18,000 Pulse writes per minute — itself a significant write load. 10% sampling gives representative data with 10% of the database overhead.
Layer 8: The Pre-Event Preparation Checklist
The architecture was only part of the preparation. The operational preparation was equally important:
# 48 hours before Black Friday:
# 1. Scale up the infrastructure manually
aws autoscaling set-desired-capacity --auto-scaling-group-name prod-asg --desired-capacity 5
# 2. Warm the CDN cache by crawling key pages
php artisan cache:warm-cdn --pages=products,categories,homepage
# 3. Pre-generate report files that would be generated during peak
php artisan reports:pre-generate --date=today
# 4. Increase Redis maxmemory for the expected cache growth
redis-cli CONFIG SET maxmemory 4gb
redis-cli CONFIG SET maxmemory-policy allkeys-lru
# 5. Put the database in the highest performance tier
aws rds modify-db-instance --db-instance-identifier prod-primary --db-instance-class db.r6g.2xlarge
# 6. Verify all health checks pass
php artisan health:check --verbose
# 7. Test the rollback procedure (if needed during the event)
php artisan deploy:rollback --dry-run
The night before:
# 24 hours before:
# Disable non-critical background jobs that generate unnecessary load
php artisan schedule:pause --tags=analytics,reporting --until="2026-11-30 00:00:00"
# Pre-load the product catalog into Redis cache
php artisan cache:prime-product-catalog
# Set up enhanced monitoring
php artisan pulse:rotate # fresh Pulse data store for clean metrics
# Verify queue workers are healthy
php artisan horizon:status
The deployment freeze:
No code deployments between 72 hours before and 24 hours after Black Friday. The last deployment was Wednesday. Friday’s traffic hit code that had been running in production for 72 hours with no changes. This is the most important preparation item — not because it prevents bugs, but because rollbacks during peak traffic add risk on top of load.
What Failed and Why It Didn’t Matter
No architecture works perfectly. Two things didn’t work as designed:
The search index fell behind. Meilisearch’s index sync, which normally kept product search results within 30 seconds of database changes, fell behind by 4–6 minutes during peak. Users who viewed a product, added it to cart, then searched for the same product by name saw slightly stale results. The product was still purchasable — checkout bypassed the search index — but the result ranked differently. Impact: non-zero but not measurable in lost sales.
One queue server ran out of file descriptors. At 3:47pm, Queue-02 hit its ulimit for open file descriptors (default 1024 on the server) due to each Horizon worker holding Redis connections and log file handles. Horizon restarted the affected workers automatically. Queue depth on that server briefly spiked before recovering. The total impact was a 12-second delay on outgoing notification emails. Payment processing was on Queue-01 and unaffected.
The fix for the file descriptor issue is now in the server bootstrap:
# /etc/systemd/system/horizon.service
[Service]
LimitNOFILE=65536 # 64x the default
Neither failure caused user-visible errors. The architecture had enough redundancy that a partial failure in one layer didn’t propagate to the user layer.
The Infrastructure Cost for Peak Day
Running a higher-performing infrastructure costs more. The specific numbers:
Infrastructure cost comparison (per day):
Normal day:
App servers (2× t3.xlarge): $0.40
Queue servers (1× t3.large): $0.10
RDS (1× db.r6g.large): $0.18
Redis (cache.r6g.large): $0.22
CloudFront: $0.80
Total/day (normal): $1.70
Black Friday (scaled up):
App servers (5× c6g.2xlarge): $3.40
Queue servers (2× c6g.xlarge): $0.76
RDS (1× db.r6g.2xlarge + 2 replicas): $1.68
Redis (cache.r6g.xlarge): $0.39
CloudFront (10× normal traffic):$8.00
Total/day (Black Friday): $14.23
Incremental cost for Black Friday infrastructure: ~$12.53 vs a normal day
The incremental $12.53 for the infrastructure to handle Black Friday without downtime was the most efficient money spent in the entire year. The first year’s 22-minute downtime cost more in lost revenue than a year of incremental infrastructure overhead.
The Architecture Diagram
Users
│
▼
CloudFront (CDN)
│ ├── Static assets → S3 (100% cache)
│ ├── Unauthenticated API → Cache (73% cache hit rate)
│ └── Authenticated / Cache miss → Origin
│
▼
Application Load Balancer
├── App-01 (Octane/FrankenPHP, 8 workers)
├── App-02 (Octane/FrankenPHP, 8 workers)
└── App-03 (Octane/FrankenPHP, 8 workers)
│
┌─────┼──────────┐
│ │ │
▼ ▼ ▼
Redis Redis ProxySQL
(Cache) (Sessions) │
(Queue) ┌───┴───┐
│ │
Primary Replica
Write Read×2
│
Queue Servers
├── Queue-01 (payments, orders)
└── Queue-02 (default, notifications)
│
Horizon
│
Redis (Queue)
What Year 3 Looks Like
The two failures from Year 2 are already addressed. The file descriptor limit is in the server bootstrap. The search index lag is addressed by switching from Meilisearch to a read replica of the production database for search queries that don’t require relevance ranking — the latency is acceptable and there’s no sync process to fall behind.
The architectural change planned for Year 3: moving the checkout flow to Lambda via the Vapor queue integration, so the checkout service scales independently of the web tier. During non-peak times, Vapor is cheaper than dedicated servers. During peak, it scales to zero delay. The web servers handle browsing; Lambda handles transaction processing.
The lesson that took two years to fully absorb: the architecture that handles 10× your normal traffic isn’t 10× the complexity of the one that handles 1×. It’s the same stack with four specific additions — stateless workers, distributed sessions, read replicas, and a CDN layer for cacheable responses — and the operational discipline to pre-warm everything before the event rather than reacting during it.