X Algorithm — Key Concepts
Atomic concept definitions for the X For You feed algorithm. Each entry is a standalone definition; use wikilinks to jump to component notes for full context.
Query hydration
The first stage of every feed request. Runs before any posts are fetched.
Fetches:
- - Recent engagement history sequence (likes, replies, reposts, clicks, in order)
- - Following list and user features (preferences, subscription status, etc.)
Why first: Every downstream stage consumes this data differently. Thunder uses the following list; Phoenix uses engagement history for the user tower; filters use the block/mute lists. Hydrating once upfront gives a single consistent user snapshot for the entire pipeline.
What it is not: It does not fetch post content — that is candidate hydration, which happens after retrieval.
User embedding
A fixed-length dense vector encoding everything the model knows about a user's behaviour.
Built by: The user tower in Phoenix's two-tower retrieval model.
Input: Engagement history sequence — liked posts, replied posts, reposts, video views, etc. — processed via hash-based embeddings.
Encodes: No explicit attributes ("user likes tech"). The tower learns implicit patterns from behaviour. Users with similar patterns end up geometrically close in the embedding space.
Used for:
- 01Retrieval: dot product with all post embeddings → top-K candidates
- 02Ranking: fed as user context into the transformer
Dot product similarity
The mechanism that converts two embedding vectors into a single relevance score.
After L2 normalisation: $$\mathbf{u} \cdot \mathbf{p} = \cos(\theta)$$
Where $\theta$ is the angle between the user vector and the post vector.
- - $\cos(\theta) \to 1.0$: vectors aligned → post is very relevant
- - $\cos(\theta) \to 0.0$: vectors orthogonal → no relationship
- - $\cos(\theta) \to -1.0$: vectors opposed → likely irrelevant
Why not Euclidean distance? Euclidean distance penalises vectors of different magnitudes. Cosine similarity depends only on direction, which is more meaningful for semantic similarity and more efficient for approximate nearest-neighbour search at scale.
Candidate isolation
A deliberate architectural constraint in the Phoenix ranking transformer.
Rule: During inference, candidate posts cannot attend to each other. Each post attends only to the user context.
Consequences:
- - A post's score is independent of what other posts are in the same inference batch
- - Scores are therefore consistent across different requests
- - Scores are cacheable — the same post + user context always produces the same score
Contrast with: Standard transformer attention, where all tokens attend to all others. That would make scores batch-dependent and uncacheable.
Hash-based embeddings
A technique for mapping discrete features (post IDs, author IDs, action types) into continuous vectors without a fixed vocabulary table.
How it works: Apply multiple hash functions to each feature → use the hash outputs as lookup indices into learned embedding tables → sum or concatenate the results.
Advantage over fixed vocab: Handles an unbounded feature space. New post IDs, new authors, new content never cause an out-of-vocabulary error.
Used in: Both the user tower and post tower in Phoenix retrieval, and in the ranking transformer inputs.
Weighted scorer
Combines ~15 action probabilities from the Phoenix transformer into a single scalar ranking score.
$$\text{Score} = \sum_{i} w_i \cdot P(\text{action}_i)$$
- - Positive weights: like, reply, repost, click, share, follow author, video view
- - Negative weights: block author, mute author, report, not interested
Negative weights mean the model actively suppresses content predicted to generate negative reactions — not just content predicted to generate low positive engagement.
Candidate pipeline
Reusable Rust framework defining the traits that all pipeline stages implement:
| Trait | Purpose |
|---|---|
Source | Fetch candidates from a data source |
Hydrator | Enrich candidates with additional features |
Filter | Remove ineligible candidates |
Scorer | Compute ranking scores |
Selector | Sort and select top candidates |
SideEffect | Async side effects (caching, logging) |
Sources and hydrators run in parallel where possible. Error handling and logging are separated from business logic.
Attenuates scores for posts from authors already well-represented in the ranked candidate set.
Purpose: Prevent any single author from monopolising the feed, even if they consistently produce high-scoring content.
Mechanism: After weighted scoring, if multiple posts from the same author are near the top of the ranking, subsequent posts from that author have their scores reduced.
- - X Algorithm — For You Feed — full pipeline
- - Thunder — in-network retrieval
- - Phoenix — out-of-network retrieval and ranking
- - X Algorithm — Pre-Scoring Filters — filter stack
- - X Algorithm — Scoring Pipeline — scorer details
Further concepts to explore
These topics connect directly to the X algorithm and are worth understanding in depth to build a complete mental model.
Machine learning foundations
- - [ ] Transformer architecture — self-attention, multi-head attention, feed-forward sublayers, layer normalisation; the building block of the Phoenix ranker
- - [ ] Two-tower models — how dual-encoder architectures work, why they scale better than cross-encoders for retrieval, training with contrastive loss
- - [ ] Approximate nearest neighbour (ANN) search — HNSW, FAISS, ScaNN; how Phoenix finds top-K post embeddings efficiently at billions-of-vectors scale
- - [ ] Contrastive learning — how the two towers are trained to push relevant user-post pairs together and irrelevant pairs apart in embedding space
- - [ ] Multi-task learning — training a single model to predict multiple labels (like, reply, block, etc.) simultaneously; trade-offs vs separate models
- - [ ] Calibration — whether predicted probabilities (P(like) = 0.7) actually reflect real-world frequencies; why it matters for the weighted scorer
- - [ ] Embedding spaces — geometric properties, isotropy, how clusters form, why cosine similarity works better than Euclidean distance for semantic similarity
Retrieval and recommendation systems
- - [ ] Candidate generation vs ranking — why large-scale recommenders split into cheap retrieval + expensive ranking; explore/exploit trade-offs
- - [ ] Collaborative filtering — user-item matrix factorisation; how it relates to the two-tower approach
- - [ ] Cold start problem — how the system handles new users with no engagement history, or new posts with no interaction signal
- - [ ] Exploration vs exploitation — how the feed balances showing content you're likely to engage with vs surfacing novel content to discover new interests
- - [ ] Feedback loops — how engagement signals used for training create self-reinforcing patterns; filter bubbles and echo chambers
- - [ ] Position bias — users are more likely to engage with posts shown higher in the feed; how training data is debiased to account for this
Distributed systems and infrastructure
- - [ ] Apache Kafka — distributed event streaming; how Thunder consumes post create/delete events; consumer groups, partitions, offsets
- - [ ] gRPC — the protocol Home Mixer uses to expose its
ScoredPostsServiceendpoint; protocol buffers, streaming vs unary RPCs - - [ ] In-memory data structures — why Rust's ownership model makes Thunder's per-user stores safe and fast; trade-offs of in-memory vs persistent storage
- - [ ] Feature stores — how ML features (user embeddings, post embeddings) are computed, stored, and served at low latency
- - [ ] Serving infrastructure — model serving patterns, batching strategies, latency SLAs for real-time feed ranking
Scoring and ranking theory
- - [ ] Learning to rank (LTR) — pointwise, pairwise, listwise approaches; how they compare to X's multi-action prediction approach
- - [ ] Normalised discounted cumulative gain (NDCG) — standard metric for evaluating ranked lists; how it penalises relevant items ranked low
- - [ ] Diversity and serendipity metrics — beyond relevance; how to measure whether a feed feels varied and surprising
- - [ ] A/B testing at scale — how changes to weights, filters, or model architecture are evaluated; interference effects, network effects
Broader context
- - [ ] Grok-1 architecture — the open-source model that Phoenix's transformer is ported from; mixture-of-experts, context length, training data
- - [ ] Responsible recommendation — algorithmic amplification, content moderation interplay with ranking, the role of negative signals (report, not interested)
- - [ ] Twitter's earlier algorithm (pre-xAI) — the original open-source
the-algorithmrepo; what changed when xAI took over and rebuilt with Grok - - [ ] TikTok's recommendation system — a useful comparison; how it handles cold start differently, heavier reliance on content signals vs social graph
- - [ ] YouTube's recommendation system — DeepMind's paper "Deep Neural Networks for YouTube Recommendations" (2016); historical reference for two-stage retrieval + ranking