Serialization

A running program holds data as objects scattered across memory, full of pointers and references that only mean something inside that one process. You cannot write a pointer to a file or push it through a TCP socket and expect another machine to understand it. Serialization solves this by flattening the data into a self-contained byte sequence: a string of bytes that carries the values, not the memory addresses.

Deserialization does the opposite. It reads the bytes and reconstructs an equivalent object in the receiver's memory. The two sides have to agree on the format, otherwise the receiver reads garbage. A Python dict serialized to JSON can be deserialized into a JavaScript object because both speak JSON, even though their in-memory representations are completely different.

Serialization shows up everywhere: saving a session to Redis, sending a request body over HTTP, writing rows to a Kafka topic, caching a computed result, or persisting application state to disk. Any time data leaves the boundary of a single process, it gets serialized.

A serializer walks the object graph field by field and emits each value in the chosen wire format. Text formats like JSON write human-readable characters: the number 1000000 becomes the seven characters 1000000. Binary formats like Protocol Buffers write raw bytes using compact encodings, so a small integer might fit in a single byte using variable-length encoding (varints).

Schemas matter. JSON is schemaless and self-describing, so every message repeats the field names ("userId", "createdAt") as text, which is flexible but wasteful. Protobuf and Avro use a separate schema that assigns each field a number, so the wire format carries field 1, field 2 instead of full names. That makes the payload far smaller and parsing faster, at the cost of needing the schema to read the data.

Deserialization is where most security and performance problems hide. Parsing untrusted input can blow up memory, and formats that deserialize arbitrary types (Java native serialization, Python pickle) have caused remote code execution vulnerabilities. Treat any byte stream from outside your trust boundary as hostile and validate after parsing.

The main axes are size, speed, schema evolution, and readability. JSON wins on debuggability and ubiquity: every language and tool understands it, and you can read it in a log. It loses on size and speed because numbers and field names are stored as text and parsing requires scanning characters.

Protocol Buffers and Avro win on compact size and decode speed, often 3 to 10 times smaller than equivalent JSON, and they handle schema evolution cleanly. You can add a new optional field with a fresh field number and old readers simply ignore it, which is essential when you cannot upgrade every service at once. The cost is that the bytes are opaque without the schema and you need a code-generation or registry step.

A practical rule: use JSON for public APIs and config where humans read it, use Protobuf or Avro for high-volume internal traffic and event streams where bytes and CPU add up. MessagePack sits in between as a compact binary JSON. Avoid language-native serialization (pickle, Java Serializable) across services because it couples you to one runtime and is a known attack surface.

Consider a microservice that emits one event per user click into Kafka, at 50,000 events per second. With JSON, each event might be 300 bytes including repeated field names like "timestamp" and "eventType". That is 15 MB per second and a lot of CPU spent parsing text on the consumer side.

Switching to Protobuf with a shared schema drops the same event to roughly 60 bytes by replacing field names with numeric tags and using varint encoding for the timestamp and IDs. Throughput stays the same but network and storage costs fall by about 80 percent, and the consumer deserializes faster because it is reading fixed binary layouts instead of scanning strings.

The team registers the schema in a schema registry. When they later add an optional "experimentId" field, they give it a new field number. Old consumers keep working untouched, new consumers read the extra field, and no coordinated deploy is needed. That smooth evolution is the real reason large event-driven systems pick schema-based binary formats.