Scaling Servers for 100k RPS
Handling 100,000 Requests Per Second (RPS) on a modern web API requires more than adding CPU cores. At this scale, the bottleneck is almost never application compute logic — it is Linux kernel network stack limits, TCP socket exhaustion, file descriptor caps, and reverse proxy worker starvation.
Architectural Bottlenecks at 100k RPS
When scaling web servers to absorb massive request concurrency, bottlenecks emerge across multiple architectural tiers:
- Socket Backlog Overflow: If the operating system's listen backlog (
somaxconn) is saturated, incoming TCP SYN packets are silently discarded. - Local Port Exhaustion: Reverse proxies making thousands of outbound connections to backend microservices deplete available ephemeral ports (
ip_local_port_range). - File Descriptor Limits: Sockets are treated as file handles in UNIX. The default
nofilelimit (1024) must be increased to over 1,000,000.
System Scaling & Connection Pool Tuning Matrix
| Component | Default Limit | 100k RPS Production Value | Architecture Fix |
|---|---|---|---|
| PostgreSQL max_connections | 100 connections | 2,000 (via PgBouncer pooler) | Deploy PgBouncer in transaction mode |
| Redis Cache Hit Ratio | Unmonitored | > 98.5% cache hit rate | Cache dynamic GET endpoints with TTL 5s |
| Linux conntrack_max | 65,536 entries | 2,097,152 entries | Tune net.netfilter.nf_conntrack_max sysctl |
NIC Hardware Ring Buffers & Network Offloading
At 100,000 requests per second, physical Network Interface Cards (NICs) process hundreds of thousands of Ethernet frames each second. Expanding RX/TX ring buffers to 4096 entries and enabling Generic Segmentation Offload (GSO) and TCP Segmentation Offload (TSO) prevents packet drops at the network driver level.
Reverse Proxy Architecture & Epoll Event Loops
Utilizing Nginx with epoll I/O multiplexing, multi_accept enabled, and open file descriptor caching allows a single reverse proxy node to handle 100,000 concurrent sockets with minimal memory footprint.
Benchmarking Your 100k RPS Capacity with Boota
Once your kernel and Nginx configurations are applied, validate your resilience using Boota Web Stresser. Execute an incremental load ramp from 10k to 100k RPS while monitoring CPU softirq load and connection drop rates.
Frequently Asked Questions (FAQ)
What is epoll and why is it essential for high RPS?
epoll is an O(1) I/O event notification facility in Linux. Unlike select() or poll() which scale linearly with socket count (O(n)), epoll allows Nginx to monitor 100,000+ open sockets with minimal CPU overhead.
Why do I see HTTP 502 Bad Gateway under stress?
HTTP 502 typically indicates that Nginx exhausted available local ephemeral ports when connecting to backend application upstream processes. Enabling keepalive pools in upstream blocks resolves this.