We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningFoundation · Core Fundamentals
Converting an in-memory data structure into a byte stream for storage or network transmission. Deserialization is the reverse. JSON, Protobuf, and Avro are serialization formats.
In short
Serialization is the process of converting an in-memory data structure (an object, struct, or map) into a flat sequence of bytes that can be written to disk or sent across a network. Deserialization is the reverse, rebuilding the original structure from those bytes. JSON, Protocol Buffers, and Avro are common serialization formats.
In short
Serialization is the process of converting an in-memory data structure (an object, struct, or map) into a flat sequence of bytes that can be written to disk or sent across a network. Deserialization is the reverse, rebuilding the original structure from those bytes. JSON, Protocol Buffers, and Avro are common serialization formats.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Serialization across a run of lessons, not one page. These 4 alone are about 68 minutes of step-by-step reading, every one with a quiz.
Google's language-neutral binary serialization, smaller, faster, and strictly typed.
Schema-evolution-friendly binary serialization, the serialization format built for big data.
JSON but binary, a compact serialization format that is schema-less and fast.
Apache Thrift is a cross-language RPC framework built at Facebook. Learn its IDL and code generation, and see how it compares with gRPC and Protobuf.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Protocol Bufferspay once,
yours for life
A running program holds data as objects scattered across memory, full of pointers and references that only mean something inside that one process. You cannot write a pointer to a file or push it through a TCP socket and expect another machine to understand it. Serialization solves this by flattening the data into a self-contained byte sequence: a string of bytes that carries the values, not the memory addresses.
Deserialization does the opposite. It reads the bytes and reconstructs an equivalent object in the receiver's memory. The two sides have to agree on the format, otherwise the receiver reads garbage. A Python dict serialized to JSON can be deserialized into a JavaScript object because both speak JSON, even though their in-memory representations are completely different.
Serialization shows up everywhere: saving a session to Redis, sending a request body over HTTP, writing rows to a Kafka topic, caching a computed result, or persisting application state to disk. Any time data leaves the boundary of a single process, it gets serialized.
A serializer walks the object graph field by field and emits each value in the chosen wire format. Text formats like JSON write human-readable characters: the number 1000000 becomes the seven characters 1000000. Binary formats like Protocol Buffers write raw bytes using compact encodings, so a small integer might fit in a single byte using variable-length encoding (varints).
Schemas matter. JSON is schemaless and self-describing, so every message repeats the field names ("userId", "createdAt") as text, which is flexible but wasteful. Protobuf and Avro use a separate schema instead. Protobuf gives each field a number, so the wire format carries field 1, field 2 instead of full names, and Avro writes the values in schema order with no field names or numbers at all. That makes the payload far smaller and parsing faster, at the cost of needing the schema to read the data.
Deserialization is where most security and performance problems hide. Parsing untrusted input can blow up memory, and formats that deserialize arbitrary types (Java native serialization, Python pickle) have caused remote code execution vulnerabilities. Treat any byte stream from outside your trust boundary as hostile and validate after parsing.
The main axes are size, speed, schema evolution, and readability. JSON wins on debuggability and ubiquity: every language and tool understands it, and you can read it in a log. It loses on size and speed because numbers and field names are stored as text and parsing requires scanning characters.
Protocol Buffers and Avro win on compact size and decode speed, often 3 to 10 times smaller than equivalent JSON, and they handle schema evolution cleanly. In Protobuf you can add a new optional field with a fresh field number and old readers simply ignore it, which is essential when you cannot upgrade every service at once. The cost is that the bytes are opaque without the schema and you need a code-generation or registry step.
A practical rule: use JSON for public APIs and config where humans read it, use Protobuf or Avro for high-volume internal traffic and event streams where bytes and CPU add up. MessagePack sits in between as a compact binary JSON. Avoid language-native serialization (pickle, Java Serializable) across services because it couples you to one runtime and is a known attack surface.
Consider a microservice that emits one event per user click into Kafka, at 50,000 events per second. With JSON, each event might be 300 bytes including repeated field names like "timestamp" and "eventType". That is 15 MB per second and a lot of CPU spent parsing text on the consumer side.
Switching to Protobuf with a shared schema drops the same event to roughly 60 bytes by replacing field names with numeric tags and using varint encoding for the timestamp and IDs. Throughput stays the same but network and storage costs fall by about 80 percent, and the consumer deserializes faster because it is reading compact binary fields instead of scanning strings.
The team registers the schema in a schema registry. When they later add an optional "experimentId" field, they give it a new field number. Old consumers keep working untouched, new consumers read the extra field, and no coordinated deploy is needed. That smooth evolution is the real reason large event-driven systems pick schema-based binary formats.
Apache Kafka
Stores serialized message bytes; teams typically pair it with Avro plus a Schema Registry for compact, evolvable event payloads.
gRPC
Uses Protocol Buffers as its default serialization format for fast, compact binary request and response messages between services.
Redis
Stores serialized blobs as values; applications serialize objects to JSON or MessagePack before caching them.
MongoDB
Persists documents in BSON, a binary serialization of JSON that adds typed fields like dates and 64-bit integers.
Serialization turns an in-memory object into a byte stream for storage or transmission. Deserialization is the reverse: it reads those bytes and reconstructs an equivalent object in memory. They are two ends of the same contract and must use the same format.
JSON is a serialization format. When you call something like JSON.stringify, you are serializing an object into a JSON text string. Parsing that string back into an object is deserialization.
Protobuf uses a separate schema, so the wire format stores numeric field tags instead of repeating text field names, and it encodes numbers in compact binary (varints) rather than as text characters. That makes payloads roughly 3 to 10 times smaller and faster to parse.
It can be. Parsing untrusted bytes can exhaust memory, and formats that rebuild arbitrary object types (Python pickle, Java native serialization) have led to remote code execution. Never deserialize untrusted input with language-native serializers, and validate parsed data before use.
It is the time spent converting data to and from bytes. It is usually small (well under a millisecond per message) but at high throughput it adds up, which is why high-volume systems prefer fast binary formats like Protobuf or Avro over text-based JSON.
Protocol Buffers
Google's language-neutral binary serialization, smaller, faster, and strictly typed
Apache Avro
Schema-evolution-friendly binary serialization, the serialization format built for big data
MessagePack
JSON but binary, a compact serialization format that is schema-less and fast
Apache Thrift
Apache Thrift is a cross-language RPC framework built at Facebook. Learn its IDL and code generation, and see how it compares with gRPC and Protobuf.
Saving Formats Compared: The Scores Never Moved, the Bytes Did
Pickle, joblib or skops? I saved the features chapter's model, and three extra models, in eleven formats: pickle protocol 4 and 5, joblib with no compression and with zlib, gzip, bz2 and lzma, and skops stored and zipped. All 44 files gave back every prediction to the last bit, in the same process and in a second environment. Size moved a lot: for the chapter model, joblib zlib 3 was 0.45 times pickle, lzma 3 was 0.37 times, and skops stored was 2.15 times, because 43% of the skops file is a text description of the model. skops refused to load the tree models until I named one type by hand. The machine was busy, so I report no load or save times.
JSON
JavaScript Object Notation: a lightweight text format for data interchange using key-value pairs and arrays. The lingua franca of web APIs.
Protocol Buffers
Google's language-neutral, binary serialization format. Smaller and faster than JSON. Defines schemas in .proto files that generate typed code for any language.
gRPC
A high-performance RPC framework by Google using Protocol Buffers and HTTP/2. Much faster than REST for service-to-service communication.
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
Throughput
The number of operations a system can handle per unit of time. Think of it as how many cars a highway can move per hour.
Bandwidth
The maximum amount of data that can be transferred over a network in a given time. It's the width of the pipe, not how fast the water flows.
This page explains the idea. The course covers it step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
See the System Design Masterclass coursecourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Serialization in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
