Thunder

"In-memory in-network post store. Serves recent posts from accounts a user follows in sub-millisecond time via Kafka-fed per-user stores."

Thunder solves a core latency problem: at X's scale, querying a database for all recent posts from followed accounts on every feed request would be prohibitively expensive. Thunder pre-loads that data into memory.

Thunder has two parallel processes:

Write path (continuous)

Kafka stream
    │  (post create / delete events)
    ▼
Realtime ingestion
    │  (parse, validate, route by author)
    ▼
Per-user in-memory stores
    ├── Original posts
    ├── Replies & reposts
    └── Video posts
         │
         └── Auto-trim past retention window

Read path (per feed request)

Feed request arrives
    │
    ▼
Following list (from query hydration)
    │
    ▼
Per-author store lookups
    │  (one lookup per followed account)
    ▼
In-network candidate set  →  Home Mixer pipeline

"Memory for latency. Thunder keeps the relevant slice of the post graph warm and pre-indexed at all times. The cost is memory; the benefit is that in-network retrieval is essentially free compared to the Phoenix ML search."

Splitting per-user stores into original posts, replies/reposts, and video posts lets the read path fetch specific content types efficiently without scanning everything.