How to Prepare Your Server and API for 100k Requests Per Second

Scaling an API to handle 100,000 requests per second (RPS) is a milestone that separates hobby projects from production-grade systems. Whether you're preparing for a product launch, a marketing campaign, or simply building headroom, this guide covers the architectural patterns that matter.

Step 1: Implement aggressive edge caching

The fastest request is the one that never reaches your origin server. Place a CDN (Cloudflare, Fastly, AWS CloudFront) in front of your API and cache every response that is identical for all users:

• GET endpoints with public data — product listings, category pages, static configs. Set Cache-Control: public, max-age=300.
• API responses with Vary headers — if the response varies by Accept-Language, set the Vary header so the CDN caches per-variant.
• Stale-while-revalidate — serve stale content for up to 60 seconds while the CDN fetches a fresh copy in the background. This eliminates cache thundering herds.

A well-configured CDN can absorb 90%+ of incoming traffic. Your origin only needs to handle the remaining 10,000 RPS — a much more tractable problem.

Step 2: Optimize database and connection pooling

At 100k RPS, even a fast 5ms database query means your backend needs to sustain 500 concurrent database connections. Without connection pooling, you'll exhaust your database's max_connections limit instantly.

• Use a connection pooler like PgBouncer (PostgreSQL) or ProxySQL (MySQL) to multiplex thousands of application connections over a smaller pool of database connections.
• Implement read replicas for read-heavy workloads. Route SELECT queries to replicas; reserve the primary for writes.
• Add a Redis or Memcached layer for frequently accessed data (user sessions, feature flags, hot product records). Cache hit rates above 95% are the target.

Step 3: Horizontal scaling and auto-scaling

Vertical scaling (bigger servers) hits diminishing returns quickly. Instead:

• Deploy your API as stateless containers behind a load balancer (Kubernetes, ECS, Cloud Run).
• Configure Horizontal Pod Autoscaler (HPA) to scale based on CPU utilization (target: 60'“70%) or custom metrics like request queue depth.
• Pre-warm instances before known traffic spikes — auto-scalers react, but they aren't instant. A 30-second cold start during a traffic surge means dropped requests.

Step 4: Protect the edge (WAF, rate limiting, challenge pages)

Not all 100k RPS will be legitimate. Bots, scrapers, and DDoS attacks will target any high-traffic API. Ensure your edge layer includes:

• A Web Application Firewall (WAF) with rules for SQL injection, XSS, and known bot signatures.
• Adaptive rate limiting per IP, per API key, and per user session.
• Challenge pages (CAPTCHA, JS challenge) for suspicious traffic patterns — but beware that advanced bots can spoof TLS fingerprints and exploit HTTP/2 multiplexing to bypass naive challenges.

Step 5: Validate with realistic load testing

Architecture diagrams are theory. Load testing is evidence. Use a combination of k6 for regression testing and an authorized IP stresser for adversarial chaos testing:

1. Run a gradual ramp test (k6) from 0 to 100k RPS over 10 minutes. Monitor p99 latency, error rate, and CPU utilization.
2. Run a spike test (Boota) — instant jump from 0 to 100k RPS. Observe auto-scaler reaction time and request drop rate during the cold-start window.
3. Run a sustained load test at 100k RPS for 30+ minutes. Watch for memory leaks, connection pool exhaustion, and disk I/O saturation.
4. Run an adversarial test with mixed malicious traffic — DNS floods, UDP amplification, and L7 cache busters — to verify your WAF actually works under pressure.

Always ensure your testing is legally authorized and scheduled during an approved maintenance window.