How load balancers distribute requests

· Category: System Design

Short answer

Load balancers operate at different network layers to distribute traffic based on connection information or application content.

Steps

  1. Layer 4 load balancers route based on IP address and TCP/UDP ports without inspecting payload content.
  2. Layer 7 load balancers inspect HTTP headers, cookies, and URLs for intelligent routing.
  3. Configure health checks at the appropriate layer to detect backend failures.
  4. Use SSL termination at the load balancer to centralize certificate management.
  5. Apply rate limiting and geographic filtering at the edge.

Tips

  • Layer 4 offers higher throughput and lower latency; layer 7 offers flexibility.
  • Use sticky sessions sparingly since they reduce distribution effectiveness.
  • Enable connection draining to allow in-flight requests to complete during maintenance.
  • Combine both layers in a two-tier architecture for optimal performance.

Common issues

  • Layer 7 processing introducing CPU overhead and latency.
  • Misconfigured health checks marking healthy nodes as unhealthy.
  • SSL termination consuming significant load balancer resources.
  • Asymmetric routing when responses bypass the load balancer.

Example

# Consistent hashing for service discovery
import hashlib

def get_node(key, nodes):
    hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16)
    return nodes[hash_val % len(nodes)]

node = get_node('user-123', ['node-a', 'node-b', 'node-c'])

This snippet implements consistent hashing to distribute keys across nodes, a foundational technique in scalable distributed systems.