We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Web Content Delivery
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Web Content Delivery
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
A user uploads a 2 GB video file. Within 5 minutes, it must be playable on any device at any quality level, with thumbnails for scrubbing, captions for accessibility, and watermarks for copyright protection. This is the media processing pipeline, and it ties together everything from the last several lessons: encoding, transcoding, streaming protocols, adaptive bitrate, and delivery.
YouTube processes over 500 hours of uploads per minute. Every upload triggers a DAG of processing jobs: format detection, transcoding into 10+ renditions, thumbnail extraction, caption generation, content moderation (scanning for policy violations), packaging into HLS and DASH, and CDN pre-warming for popular channels. The pipeline runs on hundreds of thousands of machines.
Netflix's pipeline is different: they do not process user uploads. They receive master files from studios and encode each title into over 100 renditions using per-title encoding. A single movie generates 1-2 TB of encoded output. Their encoding pipeline processes every title in their 15,000+ catalog, and re-encodes when codecs improve (they re-encoded the entire catalog when they adopted VP9, and are doing it again for AV1).
The pipeline architecture determines upload-to-playback , processing cost, failure resilience, and how quickly you can add new features (new codec support, AI-powered captioning, content moderation). Get it right and scaling is a configuration change. Get it wrong and every new feature requires rewriting the pipeline.
The DAG above shows the complete media processing pipeline from ingest to player. Each node is an independent, retryable processing stage. The critical path (the slowest chain of dependent stages) determines the upload-to-playback . Transcoding is almost always the bottleneck.
Key design principles:
Fan-out after ingest. Transcoding, thumbnail extraction, captioning, and content moderation all run in parallel. A failure in captioning does not block transcoding. A slow content moderation scan does not delay thumbnail generation.
stages. Every stage writes its output to object storage keyed by a deterministic ID (source file hash + job config). Re-running a stage with the same input produces the same output. This makes retries safe and debugging straightforward.
Event-driven orchestration. Each stage publishes a completion event. Downstream stages subscribe to the events they depend on. The packaging stage waits for all transcoding renditions to complete before generating manifests. The pre-warm stage waits for packaging to complete. SQS, Kafka, or a workflow engine (Temporal, Step Functions) coordinates this.
Dead letter queues for failed jobs. If a stage fails after 3 retries, the job lands in a DLQ for manual investigation. The pipeline keeps processing other uploads. One corrupted video should never block the entire system.
It is easy and good
“It is really nice and good Please join the course”