We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Open Datadog, PagerDuty, or Opsgenie in your browser right now. See the bell icon in the top right corner? That bell updates in real time. When someone comments on your incident, the badge count increments before your brain finishes reading the previous comment. No page refresh. No polling. Just instant delivery to every open tab.
Datadog holds roughly 500,000 concurrent connections at peak across its global fleet, and the bell update lands in the user's browser in under 200 ms p95. This is not a nice-to-have. In-app notifications are where engineers spend their day. The bell is the primary triage surface for SEV3 and SEV4 alerts that do not justify a phone call or text message.
The difference between a good in-app notification system and a bad one is whether the engineer trusts the bell. If updates are delayed, if notifications go missing, if the badge count drifts from reality, the engineer stops looking at it and starts polling Slack. Once that happens, your carefully designed notification architecture is dead.
In-app notifications sit on top of two parallel paths: a durable write and a real-time push. Every notification must take both paths or the system breaks in subtle ways.
| Path | What it does | Why it matters |
|---|---|---|
| Persist to database | INSERT into a notifications table with user_id, type, payload, read status | Offline users see the notification when they log in next. Without this, offline gaps are permanent. |
| Publish to | Push the event through a fanout worker to every open tab and mobile session | Active users see the update instantly. Without this, users must refresh to see changes. |
Slack, Linear, GitHub, and every modern incident tool use this persist-then-publish split. Slack's engineering blog describes how their notification system writes to durable storage before fanout so that a user who is offline still sees every alert when they next open the app.
The client side has two pieces: a badge counter on the bell icon that increments for each unread notification, and an optional toast popup that appears briefly at the top of the screen. Both update from the same WebSocket event, and both must be rate-limited to avoid flooding the user's screen during an incident storm.