Rate Limiting and Throttling Patterns for System Design

Rate limiting is a technique used to control the number of requests a user, IP address, or service can make to an API or system within a specific time window. It acts as a traffic control mechanism, preventing resource exhaustion and ensuring fair usage across all clients.

Unlike simple throttling (which just delays requests), rate limiting can reject excessive requests entirely. This makes it crucial for protecting against DDoS attacks, preventing abuse, and maintaining service quality under high load conditions.

Modern systems implement rate limiting at multiple layers to create defense in depth. Companies like Twitter limit API calls per user, GitHub limits Git operations per repository, and cloud providers limit API requests per account. Without proper rate limiting, a single misbehaving client can bring down an entire service.

There are four main rate limiting algorithms, each with different characteristics for handling burst traffic and maintaining fairness:

The token bucket algorithm is the most popular choice for rate limiting because it naturally handles burst traffic while maintaining long-term rate limits. Here's how it works:

import time
import threading

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity        # Max tokens
        self.tokens = capacity          # Current tokens
        self.refill_rate = refill_rate  # Tokens per second
        self.last_refill = time.time()
        self.lock = threading.Lock()
    
    def allow_request(self, tokens_needed=1):
        with self.lock:
            # Refill tokens based on elapsed time
            now = time.time()
            elapsed = now - self.last_refill
            tokens_to_add = elapsed * self.refill_rate
            
            self.tokens = min(self.capacity, 
                            self.tokens + tokens_to_add)
            self.last_refill = now
            
            # Check if request can be allowed
            if self.tokens >= tokens_needed:
                self.tokens -= tokens_needed
                return True
            return False

This implementation allows burst traffic up to the bucket capacity while ensuring the long-term rate doesn't exceed the refill rate. It's memory efficient (O(1) per user) and handles edge cases like long idle periods gracefully.

Rate limiting can be implemented using different architectural patterns depending on your system's requirements and constraints. Here are the most common approaches used in production systems.

Redis is the most popular choice for distributed rate limiting because of its atomic operations and sub-millisecond latency. Here's a production-ready sliding window implementation:

-- Sliding window rate limiting script
local key = KEYS[1]           -- Rate limit key (e.g., "user:123")
local window = tonumber(ARGV[1])     -- Time window in seconds
local limit = tonumber(ARGV[2])      -- Max requests per window
local now = tonumber(ARGV[3])        -- Current timestamp

-- Remove expired entries
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

-- Count current requests in window
local current = redis.call('ZCARD', key)

if current < limit then
    -- Add current request
    redis.call('ZADD', key, now, now)
    redis.call('EXPIRE', key, window)
    return {1, limit - current - 1}  -- [allowed, remaining]
else
    return {0, 0}  -- [denied, remaining]
end

This Lua script runs atomically on Redis, preventing race conditions in high-concurrency scenarios. The sliding window approach provides more accurate rate limiting than fixed windows, preventing the 'burst at boundary' problem.

Rate limiting can be implemented at multiple layers of your system architecture. Each layer serves different purposes and provides varying levels of protection and granularity.

1. CDN/Edge Level: Cloudflare, AWS CloudFront - Blocks malicious traffic before it reaches your infrastructure
2. Load Balancer: Nginx, HAProxy - Rate limit by IP address or geographic region at the entry point
3. API Gateway: Kong, AWS API Gateway - Centralized rate limiting with authentication context
4. Application Level: Express.js middleware, Django decorators - Business logic aware, user-specific limits
5. Database Level: Connection pooling, query rate limiting - Protects your most critical resource

The key is implementing complementary limits at each layer. For example, you might have a global IP limit at the load balancer (1000 req/min), authenticated user limits at the API gateway (100 req/min per user), and endpoint-specific limits in your application (10 req/min for expensive operations).

In distributed systems, rate limiting becomes more complex because state must be shared across multiple application instances. The naive approach of per-instance limits doesn't work because users can bypass limits by hitting different servers.

Centralized State Approach: Use Redis or similar to maintain shared counters. This provides perfect accuracy but introduces latency and potential single points of failure.

Gossip Protocol Approach: Each instance maintains local counters and periodically shares updates with other instances. This reduces latency but sacrifices accuracy for eventually consistent rate limiting.

Hybrid Approach: Combine local and global limits. Allow some requests locally but check global state for expensive operations. This balances accuracy with performance.

Most modern web frameworks provide built-in rate limiting middleware or easy integration with external solutions:

// Express.js with express-rate-limit
const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100, // Limit each IP to 100 requests per windowMs
  message: {
    error: 'Too many requests from this IP',
    retryAfter: 900 // seconds
  },
  standardHeaders: true, // Return rate limit info in headers
  legacyHeaders: false,
});

// Apply to all requests
app.use(limiter);

// Apply to specific routes with different limits
app.use('/api/auth', rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 5 // Stricter limit for auth endpoints
}));

For distributed systems, integrate with Redis for shared state across instances. This ensures consistent rate limiting regardless of which server handles the request.