We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · API Design & Protocols
Google's language-neutral, binary serialization format. Smaller and faster than JSON. Defines schemas in .proto files that generate typed code for any language.
In short
Protocol Buffers (protobuf) is Google's language-neutral binary serialization format where you define your data structure once in a .proto schema file, then generate typed code for any language to encode and decode that data. The binary output is smaller and faster to parse than JSON, which is why it is the default wire format for gRPC.
In short
Protocol Buffers (protobuf) is Google's language-neutral binary serialization format where you define your data structure once in a .proto schema file, then generate typed code for any language to encode and decode that data. The binary output is smaller and faster to parse than JSON, which is why it is the default wire format for gRPC.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Protocol Buffers across a run of lessons, not one page. These 4 alone are about 95 minutes of step-by-step reading, every one with a quiz.
Google's language-neutral binary serialization, smaller, faster, and strictly typed.
Google's high-performance RPC framework, binary protocols, HTTP/2 streaming, and auto-generated clients.
JSON Schema describes what valid JSON looks like. Learn type, required, additionalProperties, format email, $ref and oneOf with tested examples.
Apache Thrift is a cross-language RPC framework built at Facebook. Learn its IDL and code generation, and see how it compares with gRPC and Protobuf.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Protocol Bufferspay once,
yours for life
Protocol Buffers is two things working together: a schema language and a binary serialization format. You write a schema in a .proto file that lists the fields of a message, each with a type and a number. From that file, the protoc compiler generates classes in Go, Java, Python, C++, TypeScript, Rust, and more. Your code talks to those generated classes; protobuf handles turning them into bytes and back.
The big difference from JSON is that the field names never travel on the wire. JSON sends {"user_id": 42} as 15 bytes of text every single time. Protobuf sends the field number (1) and the value (42) as a few bytes of binary. Both sides already know from the shared schema that field 1 is user_id, so the name is redundant. That single decision is most of why protobuf payloads are commonly 3 to 10 times smaller than the equivalent JSON.
Protocol Buffers was built at Google around 2001 and open sourced in 2008. It is the foundation under gRPC, and Google uses it internally for billions of messages between services every day.
Each field on the wire is a key followed by a value. The key packs the field number together with a wire type into a single varint. The wire type tells the decoder how to read the value that follows: a varint, a fixed 32 or 64 bit block, or a length-delimited chunk for strings, bytes, and nested messages.
Varints are the trick that makes small numbers cheap. A varint stores an integer in as few bytes as possible, using 7 bits of each byte for data and the top bit as a continue flag. The number 1 takes one byte. The number 300 takes two. You only pay for the magnitude you actually use, instead of a fixed 4 or 8 bytes like most binary formats.
Because the field number is what identifies data, not the position or the name, the format is forgiving about change. A decoder that sees a field number it does not recognize simply skips those bytes. That is how a new server and an old client keep talking. Fields are also optional on the wire, so a field left at its default value is not sent at all, which shrinks sparse messages further.
Reach for Protocol Buffers when you control both ends of a connection and care about size or speed: internal service-to-service calls, mobile clients on slow networks, event streams into Kafka, or anything high throughput. With gRPC you get protobuf, HTTP/2 multiplexing, and generated client stubs as one package, which removes a lot of hand-written serialization code.
The cost is that the bytes are not human readable. You cannot curl an endpoint and eyeball the response, and you cannot debug a payload without the .proto file that describes it. JSON wins for public REST APIs, webhooks, config files, and anything a person needs to read or a browser needs to consume directly.
Schema evolution has firm rules you must respect. Never reuse or renumber a field that has shipped, because old data on disk still carries the old numbers. Add new fields with new numbers, and mark removed field numbers as reserved so nobody accidentally recycles them. Break these rules and you get silent data corruption, not a loud error.
Created protobuf and uses it for nearly all internal service communication, serializing billions of messages a day across its data centers.
gRPC
Uses Protocol Buffers as its default interface definition language and wire format, generating both client stubs and server interfaces from the same .proto file.
Apache Kafka
Commonly stores protobuf-encoded events, with the Confluent Schema Registry tracking .proto versions so producers and consumers stay compatible.
Netflix
Uses protobuf over gRPC for high-throughput internal microservice calls where JSON parsing overhead and payload size would hurt at scale.
Usually yes, for both size and CPU. Protobuf payloads are typically 3 to 10 times smaller because field names are not sent, and decoding binary varints is cheaper than parsing and validating text. The gap is largest for numeric and structured data; for short string-heavy messages the advantage shrinks.
It is the schema. You declare your messages, their fields, each field's type, and a unique field number. The protoc compiler reads this file and generates typed encode and decode code in your chosen languages, so every service shares one source of truth for the data shape.
The number is the field's permanent identity on the wire. The encoded bytes carry field numbers, not names, which is what keeps payloads small and lets old and new code interoperate. Because of this you must never change or reuse a number once it has been deployed.
Protocol Buffers is the data format: how a message is described and serialized to bytes. gRPC is the RPC framework that moves those messages between services over HTTP/2. gRPC uses protobuf by default, but protobuf can be used on its own to serialize data without gRPC.
Yes, if you follow the rules. Add new fields with brand new numbers and they will be ignored by old readers. When you remove a field, mark its number as reserved so it is never reused. Never renumber an existing field, because old stored data still references the old number.
gRPC
A high-performance RPC framework by Google using Protocol Buffers and HTTP/2. Much faster than REST for service-to-service communication.
JSON
JavaScript Object Notation: a lightweight text format for data interchange using key-value pairs and arrays. The lingua franca of web APIs.
REST API
An architectural style for building APIs using standard HTTP methods (GET, POST, PUT, DELETE). Resources are identified by URLs.
WebSocket
A protocol for full-duplex communication over a single TCP connection. Unlike HTTP, the server can push data to the client without being asked.
SSE
Server-Sent Events: a one-way channel where the server pushes updates to the client over HTTP. Simpler than WebSockets when you only need server-to-client streaming.
Rate Limiting
Controlling how many requests a client can make in a given time window. Protects your API from abuse and ensures fair usage.
This page explains the idea. The "Protocol Buffers" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Protocol Buffers lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Protocol Buffers in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
