Protocol Buffers
Google's language-neutral, binary serialization format. Smaller and faster than JSON. Defines schemas in .proto files that generate typed code for any language.
What is Protocol Buffers?
In short
Protocol Buffers (protobuf) is Google's language-neutral binary serialization format where you define your data structure once in a .proto schema file, then generate typed code for any language to encode and decode that data. The binary output is smaller and faster to parse than JSON, which is why it is the default wire format for gRPC.
What Protocol Buffers actually is
Protocol Buffers is two things working together: a schema language and a binary serialization format. You write a schema in a .proto file that lists the fields of a message, each with a type and a number. From that file, the protoc compiler generates classes in Go, Java, Python, C++, TypeScript, Rust, and more. Your code talks to those generated classes; protobuf handles turning them into bytes and back.
The big difference from JSON is that the field names never travel on the wire. JSON sends {"user_id": 42} as 16 bytes of text every single time. Protobuf sends the field number (1) and the value (42) as a few bytes of binary. Both sides already know from the shared schema that field 1 is user_id, so the name is redundant. That single decision is most of why protobuf payloads are commonly 3 to 10 times smaller than the equivalent JSON.
Protocol Buffers was built at Google around 2001 and open sourced in 2008. It is the foundation under gRPC, and Google uses it internally for billions of messages between services every day.
How the encoding works under the hood
Each field on the wire is a key followed by a value. The key packs the field number together with a wire type into a single varint. The wire type tells the decoder how to read the value that follows: a varint, a fixed 32 or 64 bit block, or a length-delimited chunk for strings, bytes, and nested messages.
Varints are the trick that makes small numbers cheap. A varint stores an integer in as few bytes as possible, using 7 bits of each byte for data and the top bit as a continue flag. The number 1 takes one byte. The number 300 takes two. You only pay for the magnitude you actually use, instead of a fixed 4 or 8 bytes like most binary formats.
Because the field number is what identifies data, not the position or the name, the format is forgiving about change. A decoder that sees a field number it does not recognize simply skips those bytes. That is how a new server and an old client keep talking. Fields are also optional on the wire, so a field left at its default value is not sent at all, which shrinks sparse messages further.
When to use it and the trade-offs
Reach for Protocol Buffers when you control both ends of a connection and care about size or speed: internal service-to-service calls, mobile clients on slow networks, event streams into Kafka, or anything high throughput. With gRPC you get protobuf, HTTP/2 multiplexing, and generated client stubs as one package, which removes a lot of hand-written serialization code.
The cost is that the bytes are not human readable. You cannot curl an endpoint and eyeball the response, and you cannot debug a payload without the .proto file that describes it. JSON wins for public REST APIs, webhooks, config files, and anything a person needs to read or a browser needs to consume directly.
Schema evolution has firm rules you must respect. Never reuse or renumber a field that has shipped, because old data on disk still carries the old numbers. Add new fields with new numbers, and mark removed field numbers as reserved so nobody accidentally recycles them. Break these rules and you get silent data corruption, not a loud error.
Where it is used in production
Created protobuf and uses it for nearly all internal service communication, serializing billions of messages a day across its data centers.
gRPC
Uses Protocol Buffers as its default interface definition language and wire format, generating both client stubs and server interfaces from the same .proto file.
Apache Kafka
Commonly stores protobuf-encoded events, with the Confluent Schema Registry tracking .proto versions so producers and consumers stay compatible.
Netflix
Uses protobuf over gRPC for high-throughput internal microservice calls where JSON parsing overhead and payload size would hurt at scale.
Frequently asked questions
- Is Protocol Buffers faster than JSON?
- Usually yes, for both size and CPU. Protobuf payloads are typically 3 to 10 times smaller because field names are not sent, and decoding binary varints is cheaper than parsing and validating text. The gap is largest for numeric and structured data; for short string-heavy messages the advantage shrinks.
- What is a .proto file?
- It is the schema. You declare your messages, their fields, each field's type, and a unique field number. The protoc compiler reads this file and generates typed encode and decode code in your chosen languages, so every service shares one source of truth for the data shape.
- Why does every field need a number?
- The number is the field's permanent identity on the wire. The encoded bytes carry field numbers, not names, which is what keeps payloads small and lets old and new code interoperate. Because of this you must never change or reuse a number once it has been deployed.
- What is the difference between Protocol Buffers and gRPC?
- Protocol Buffers is the data format: how a message is described and serialized to bytes. gRPC is the RPC framework that moves those messages between services over HTTP/2. gRPC uses protobuf by default, but protobuf can be used on its own to serialize data without gRPC.
- Can I add or remove fields later?
- Yes, if you follow the rules. Add new fields with brand new numbers and they will be ignored by old readers. When you remove a field, mark its number as reserved so it is never reused. Never renumber an existing field, because old stored data still references the old number.
Learn Protocol Buffers hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Protocol Buffers as part of a larger topic.
See also
Related glossary terms you might want to look up next.
gRPC
A high-performance RPC framework by Google using Protocol Buffers and HTTP/2. Much faster than REST for service-to-service communication.
JSON
JavaScript Object Notation: a lightweight text format for data interchange using key-value pairs and arrays. The lingua franca of web APIs.
REST API
An architectural style for building APIs using standard HTTP methods (GET, POST, PUT, DELETE). Resources are identified by URLs.
WebSocket
A protocol for full-duplex communication over a single TCP connection. Unlike HTTP, the server can push data to the client without being asked.
SSE
Server-Sent Events: a one-way channel where the server pushes updates to the client over HTTP. Simpler than WebSockets when you only need server-to-client streaming.
Rate Limiting
Controlling how many requests a client can make in a given time window. Protects your API from abuse and ensures fair usage.