May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design and AI Engineering, by Roni Das. 989 interactive lessons in two courses.: 770 in System Design and 219 in AI Engineering.
Start LearningSystem design interview guide
Short answer
A news aggregator such as Google News is a pipeline with four steps. Fetchers poll publishers' RSS or Atom feeds, or receive pushes through WebSub, while obeying robots.txt and limiting requests per site. Each article is cleaned and checked for duplicates, from the cheapest test to the most costly: same URL, same text hash, nearly the same text (SimHash), and finally the same event, where story clustering groups articles about one event. A ranking job scores stories about once a minute by freshness, how many publishers cover them, and location and language. Each page view reads these shared lists and re-ranks a few hundred stories for the reader.
in this guide
8 parts open to everyone · 3 with a free account · 3 with the course
You can read 8 of 14 parts. A free account opens the deep dive; the course opens all of it.
This question tests four skills at once, and each one hides a trap. The first is pulling data from systems you do not own. Publishers are slow, change formats, go down, and can block you, so you have to be polite and patient. The second is deciding when two things are the same. An exact match catches very little, because copies differ by a headline or a dateline. A candidate who knows near-duplicate fingerprints, and who also knows that a copy and a different article about one event are two separate problems, stands out. The third is trading freshness against importance in one score, with numbers. The fourth is the fan-out question, and it is the one that most often goes wrong. Many candidates bring the social feed answer, which writes each new post into every follower's feed. News has a different shape: every reader in a region draws from one pool of a few thousand fresh stories. Spotting that, and ranking shared lists once, is what the interviewer most wants to hear.
This page is the free part.
₹1,111 in India$35 everywhere elseonce, for the whole course
The System Design course covers the News Aggregator design across a run of lessons, not one page. These 4 alone are about 70 minutes of step-by-step reading, every one with a quiz.
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
This guide is built on these 6 lessons. Each one teaches one piece of the News Aggregator design above, step by step.
Lifetime access for ₹1,111 in India or $35 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.
This one question is not enough for the interview.
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
Two courses by the author of this page:
770 lessons · 18 free to read
₹1,111 in India · $35 elsewhere, once
Get System DesignYou own this course
219 lessons · 10 free to read
₹1,333 in India · $35 elsewhere, once
Get Generative AI and Agentic AIYou own this course
Save your progress and get the interview checklist PDF
A news aggregator reads articles from thousands of news sites and shows them in one place, grouped by story. Take 200,000 feeds checked every 5 minutes: that is about 670 requests every second, all day, to other people's servers. Say they bring in 1 million new articles a day. Many of those are copies of one wire story, and many more are different articles about one event. A news story is worth most in its first few hours. So the system has three jobs: notice new articles fast without overloading anyone, collapse copies and group the rest into stories, and rank stories so fresh and important ones come first, for 100 million readers.
A news aggregator such as Google News works as a pipeline. First, fetchers collect new articles from publishers. A publisher's feed is a small file (RSS or Atom) that lists its newest articles. The fetchers poll each feed on a schedule. To poll is to ask again and again whether anything changed. Where a publisher supports WebSub, a hub pushes new items to us instead. The fetchers obey robots.txt, a file in which each site lists what crawlers may fetch, and limit how often they hit each site. Second, each article is cleaned and checked for duplicates in four steps, from cheapest to most costly: same id or URL, same text hash (a short code computed from the text, equal only for equal texts), nearly identical text (SimHash, a 64-bit fingerprint where copies differ in at most 3 bits), and same event. The last step, story clustering, keeps every article but groups those about one event into a single story. Third, a ranking job scores each story about once a minute. A good score mixes freshness (score halves every few hours), importance (how many publishers cover it) and location and language. It writes one ranked list per edition and section to a cache. Fourth, each page view reads those shared lists and re-ranks a few hundred candidates for the reader, using their interests. Signed-out readers get a cached global page. Copying every story into each reader's own feed would cost trillions of writes a day, so the design ranks shared lists once and personalizes on read.
Where it shows up
A news aggregator is a common practice problem in system design interview preparation, under names such as 'design Google News' or 'design a news feed aggregator'. It also sits inside other questions: design a web crawler, design a news feed, design a recommendation system, design a search engine. Google described the personalization behind Google News in a 2007 paper, and Google engineers described their near-duplicate check for web pages in another 2007 paper. Both are on this page.
Spec sheeta News Aggregator
worked through below, with the maths
The hard parts of this design are taught step by step in the System Design course: Message Deduplication, Design Twitter/X Feed and Polling.
The simplest way to check for new data, ask repeatedly until something changes.
Don't call us, we'll call you, server-to-server event notifications over HTTP.
One publisher, many subscribers, the pattern that powers event-driven systems at scale.
Decouple producers from consumers with a buffer that holds messages until they're processed.
all part of
System Design Masterclass
770 lessons · about 290 hours · 18 free to read
₹1,111in India, by UPI
$35everywhere else, by card or PayPal
You own this course
Continue with Pollingpay once,
yours for life
Live from the course bench
Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Feed checks
about 670 requests a second
Assume 200,000 feeds (many publishers have one feed per section), each checked every 5 minutes on average. 200,000 / 300 seconds = 667 requests a second, or 57.6 million a day. These are interview assumptions: say them out loud first. Most checks find nothing new and return a short 304 Not Modified answer with no body.
New articles
about 12 a second, peaks near 60
Assume 1,000,000 new articles a day. 1,000,000 / 86,400 seconds = 11.6 a second on average. When big news breaks, assume 5 times that: about 58 a second. As a write rate this is small. The cost is the work done on each article: fetching the page, cleaning it, fingerprinting it and clustering it.
Article storage
about 10 GB a day
Assume 10 KB per article for the cleaned text plus its metadata (title, URL, times, fingerprint, story id). 1,000,000 x 10 KB = 10 GB a day, or 3.65 TB a year for one copy. With 3 copies for safety, about 11 TB a year. Images are not stored; the page links to the publisher's own.
Fingerprint index
240 MB, fits in memory
Keep SimHash fingerprints (8 bytes each) for the last 30 days: 30 x 1,000,000 x 8 bytes = 240 MB. The fast lookup in the deep dive files each fingerprint under 4 keys, so about 960 MB of keys, which still fits on one machine. Each lookup finds about 30,000,000 / 65,536 = 458 candidates per key, so about 1,831 cheap bit comparisons per new article.
Feed page views
about 5,800 a second, peaks near 17,400
Assume 100 million daily readers, each opening the feed 5 times a day: 500 million page views a day. 500,000,000 / 86,400 = 5,787 a second on average. Assume the busiest hour is 3 times the average: 17,361 a second.
Personalize on read
about 8.7 million scores a second at peak
Each signed-in page view re-ranks about 500 candidate stories. 17,361 x 500 = 8.7 million story scores a second at peak. Each score is a few multiplications on data already in memory, spread over all the feed servers.
Precompute every reader instead?
111,000 feed builds a second
Rebuilding all 100 million personal feeds every 15 minutes is 100,000,000 / 900 = 111,111 builds a second. That is 6.4 times the peak page-view rate, and most of those feeds would never be opened before they went stale. This number is the main argument for ranking on read.
Shared top-story lists
500 lists, under 1 MB in total
Assume 50 editions x 10 sections = 500 lists. Each holds 200 story ids of 8 bytes: 1.6 KB. 500 x 1.6 KB = 800 KB. Rebuilt once a minute, that is 500 x 1,440 = 720,000 list writes a day, which is tiny.
Draw two halves. The left half collects and understands articles, and it never stops. The right half serves readers, and it works only when someone opens the page. Between them sit two stores. On the left, a feed scheduler keeps a list of every feed and the time each one is next due. It hands due feeds to a pool of fetchers. Fetchers are grouped by host, the website's domain name, so one host only ever gets a small, fixed number of requests at once. A fetcher sends a conditional request: it asks for the feed only if it changed since the last visit. If the feed changed, the fetcher reads the new items, and fetches the full article page when the feed holds only a summary. It checks robots.txt before fetching any page. Publishers that support WebSub push new items to a callback URL of ours, which joins the polling path. Every new item is written to an article log: Apache Kafka, a durable, ordered list of messages that many workers can read at their own pace. The log decouples fetching from processing. If processing falls behind during breaking news, items wait in the log and nothing is lost. Processing workers read the log. Each one cleans the article text, detects the language, and runs the duplicate checks: exact id, exact text hash, then a SimHash near-duplicate check. Unique articles go to the story clusterer, which either adds the article to an existing story or starts a new one. A classifier, a model trained to sort articles, assigns sections. The results are saved in the article store, a database sharded by article id (split across machines by a hash of the id). The story store holds each story's articles, counts and scores. On the right, a ranking job runs about once a minute. For every edition and section it scores the fresh stories and writes a ranked list of story ids to Redis, an in-memory key-value store. The feed API servers, which answer the app's and website's requests, keep a copy of each list in their own memory. A signed-out reader gets the global page, which a CDN (content delivery network, servers near readers that cache pages) can serve for a few seconds at a time. For a signed-in reader, the feed API takes a few hundred candidates from the relevant lists, re-ranks them with that reader's interest profile, and returns the page. Clicks flow back through a second log into a job that updates each reader's profile.
In a real interview, sketch this on the whiteboard before diving into any single box.
Follow a single article through the system. This is a good order to explain it in an interview.
The scheduler sees that the Technology feed of a news site is due. Its last check, 6 minutes ago, saved the feed's ETag, a version tag the server sent back.
A fetcher for that host sends GET with If-None-Match and the saved ETag. The feed changed, so the server returns 200 with the new file. Had nothing changed, it would return 304 with no body.
The fetcher reads one new item. It has a guid, the RSS item id, so the fetcher checks a fast 'seen ids' set. It is new. The fetcher checks robots.txt and fetches the full article page.
The raw item and page go into the Kafka article log, keyed by host. The fetcher's work is done.
A worker strips menus and ads, detects English, and hashes the text: no exact match. Its SimHash differs from an article seen an hour ago in only 2 of 64 bits, so it is a reprint of a wire story. It is linked to the original and not shown on its own.
A second outlet's own article about the same event is not a copy. The clusterer finds a story from the last 48 hours that shares its people, places and key words, and adds it there. The story now has 12 publishers.
Within a minute the ranking job scores the story: high importance (12 publishers), high freshness (one hour old). It enters the US English Technology list and the US top-stories list in Redis.
A signed-in reader opens the app. The feed API reads both lists from memory, adds candidates from topics the reader follows, re-ranks about 500 stories with the reader's profile and returns the top 30. The story is near the top.
The reader taps the story. The click goes into the click log, and the profile job raises the weight of Technology for that reader.
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Holds every feed: its URL, its publisher, its saved ETag and Last-Modified values, its current poll interval and the time it is next due. A busy feed is checked more often and a quiet one less often. The scheduler hands due feeds to fetchers, grouped by host.
Workers that download feeds and article pages. All requests to one host go through one queue, so the per-host limit holds even with many workers. They cache robots.txt per host, send conditional requests, and slow down when a host answers slowly or with errors.
A web endpoint that receives pushed updates. WebSub is a W3C standard with three roles: the publisher, a hub that the feed names, and a subscriber (us). We subscribe at the hub, prove we own our callback URL, and then receive new content as POST requests. Subscriptions have a lease, a time limit, so a job renews them before they expire.
A durable, ordered log between fetching and processing. It absorbs bursts: when a huge story breaks, the log grows for a few minutes and the workers catch up. Workers can be added or restarted without losing articles. Messages can be read more than once after a crash, so every later step must be safe to repeat.
Runs the duplicate checks from cheapest to most costly: a set of seen ids and canonical URLs, an exact hash of the cleaned text, and a SimHash index for near-copies. A copy is linked to the original article, so its publisher can still be counted, but it is not shown as a separate item.
Groups different articles about one event into a story. It compares a new article with stories from the last day or two that share its people, places, organisations and key words, joins the best match above a threshold, or starts a new story. A slower background job later merges stories that should be one.
Gives each article one or more sections, a language and a region. It uses hints from the publisher (the feed it came from, the RSS category element) and a text model trained on labelled articles. A story takes the sections most of its articles have.
Once a minute, scores the fresh stories for each edition and section and writes a ranked list of story ids to Redis. Readers never trigger a rebuild, so a list never goes missing under load. The feed servers keep their own in-memory copy of each list.
Builds the page. Signed-out readers get the shared list for their edition. Signed-in readers get a few hundred candidates re-ranked with their profile: topic weights learned from their clicks, sources they follow or hide, and stories they already read.
feeds. One row per feed. The scheduler reads the rows whose next_due_at has passed. Saving etag and last_modified is what makes most checks cheap. The websub fields are empty…
Every table for a News Aggregator, its keys, and which store fits it and why.
Ingestion: polling feeds, WebSub push and polite fetching. Most publishers offer a feed. RSS 2.0 and Atom are the two common formats. Both are small text files, written in a format called XML, that list the newest articles with…
9 deep dives on the parts that break at scale, with 8 diagrams: Ingestion: polling feeds, WebSub push and polite fetching; Giving each article one identity; Near-duplicate detection with SimHash.
Poll feeds vs receive WebSub push. Polling works with every feed and is under your control, but new items wait until the next check, and each check costs a request. Push delivers within seconds, but only…
7 decisions an interviewer expects you to defend, each with the alternative and why it loses here.
A big publisher's site starts answering slowly. What does the system do?
6 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design a News Aggregator with the AI tutor.
One of the few detailed public descriptions of a large news aggregator's personalization is Google's 2007 paper by Das, Datar, Garg and Rajaram. At the time Google News had several million unique visitors over a few days and several million stories, where a story is a cluster of news articles. Three methods scored each candidate story: MinHash clustering of users by the stories they clicked, PLSI (a probability model of user interests), and covisitation, which counts two stories clicked by one user within a few hours. Scores were added with weights chosen by live experiments. The candidates could come from the news front end, based on the edition, the reader's language and story freshness. Pages were generated within a second, which left the recommender a few hundred milliseconds, and its counts were time-decayed so recent clicks weighed more. In a test over 5 to 6 months, both weighted mixes got 38% more clicks on average than ranking by popularity alone. For near-duplicates, Manku, Jain and Das Sarma at Google showed in 2007 that Charikar's SimHash, with 64-bit fingerprints and up to 3 differing bits, is reasonable for a store of 8 billion web pages, and showed how to keep those fingerprints in permuted, sorted tables so each lookup is fast. Broder's 1997 paper on resemblance and shingles is the root of the MinHash family. On the ingestion side, the standards are public: RSS 2.0 (every item element optional, a ttl hint in minutes), Atom (RFC 4287, exactly one permanent id per entry), WebSub (a W3C Recommendation with publisher, hub and subscriber roles, leases and signed deliveries), HTTP conditional requests and 304 Not Modified (RFC 9110), and robots.txt (RFC 9309, cache at most 24 hours, a 5xx means fetch nothing). Google's crawling docs describe the politeness rule this page uses: crawl less when a site slows down or returns 5xx or 429. Google's help pages say Google News ranking is done by algorithms using relevance, prominence, authoritativeness, freshness, usability, and location and language, with some sections personalized.
Sources
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Polling
intermediate / messaging event systems
Webhooks
intermediate / messaging event systems
Pub/Sub Pattern
intermediate / messaging event systems
Message Queues
intermediate / messaging event systems
Message Deduplication
intermediate / messaging event systems
Fan-Out/Fan-In
intermediate / messaging event systems
Data Deduplication
advanced / consistency models
Bloom Filters
intermediate / database types storage
Inverted Index
intermediate / database types storage
Cache-Aside Pattern
foundation / caching strategies
Cache Stampede Prevention
foundation / caching strategies
Cache Invalidation
foundation / caching strategies
Time To Live (TTL)
foundation / caching strategies
Content Delivery Network (CDN)
foundation / load balancing proxies
Database Sharding
foundation / database fundamentals
Rate Limiting for Resilience
advanced / reliability resilience
Retry Patterns
advanced / reliability resilience
High Availability
advanced / reliability resilience
Back Pressure
intermediate / microservices architecture
Stream Processing
advanced / stream batch processing
Design Twitter/X Feed
capstone / capstone
Design Google Search
capstone / capstone
interview readiness
Three questions from this guide. Answer them without scrolling up, the way you would have to in the room.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
NEWSystem Design Hands-On
Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.
Free PDF · 79 pages
25 real tech interview questions, answered in full
Five from each of our AI, GenAI, Inference, Maths for ML and Forward Deployed packs. Each with a diagram and a model answer. Free by email.
free tool
Check these numbers yourself
Put in your own guesses for a News Aggregator and see requests per second, storage, bandwidth and servers, with every sum written out.
Open the Capacity EstimatorIt is really nice and good Please join the course