How to design a social media news feed

· Category: System Design

Short answer

A news feed aggregates and ranks content from a user's connections, balancing freshness, relevance, and scalability.

Steps

  1. Use a pull model for celebrities with many followers to avoid write amplification.
  2. Use a push model for normal users by precomputing feeds at post time.
  3. Store posts in a distributed data store and fan them out to follower feeds.
  4. Cache hot feeds and merge them with real-time updates at read time.
  5. Rank content using engagement signals, recency, and personalization models.

Tips

  • Combine push and pull in a hybrid model for optimal efficiency.
  • Use edge caches to serve feeds from locations close to users.
  • Implement pagination with cursors for consistent ordering.
  • Pre-warm caches for users with predictable access patterns.

Common issues

  • Fan-out storms when popular users post content.
  • Feed inconsistency between cached and real-time data.
  • High storage costs from precomputed feeds for inactive users.
  • Ranking latency degrading user experience.

Example

# Consistent hashing for service discovery
import hashlib

def get_node(key, nodes):
    hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16)
    return nodes[hash_val % len(nodes)]

node = get_node('user-123', ['node-a', 'node-b', 'node-c'])

This snippet implements consistent hashing to distribute keys across nodes, a foundational technique in scalable distributed systems.