We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A centralized service that stores and validates schemas for event-driven systems. Confluent Schema Registry ensures Kafka producers and consumers agree on data formats.
In short
A schema registry is a central service that stores the data formats (schemas) used by messages in an event-driven system, assigns each one a version and an ID, and checks that any new schema is compatible with older ones before producers are allowed to use it. It lets Kafka producers and consumers agree on the shape of every record without shipping the full schema inside each message.
In short
A schema registry is a central service that stores the data formats (schemas) used by messages in an event-driven system, assigns each one a version and an ID, and checks that any new schema is compatible with older ones before producers are allowed to use it. It lets Kafka producers and consumers agree on the shape of every record without shipping the full schema inside each message.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Schema Registry across a run of lessons, not one page. These 4 alone are about 72 minutes of step-by-step reading, every one with a quiz.
A central repository for API schemas, enforce compatibility, enable evolution, prevent breaking changes.
Decouple producers from consumers with a buffer that holds messages until they're processed.
Changing database schemas and data formats in production without breaking running applications.
Transfer complete log files from primary to standby, the oldest and simplest replication method.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Schema Registrypay once,
yours for life
When systems pass messages through a broker like Kafka, the producer and the consumer are written by different teams, deployed at different times, and updated on different schedules. Nothing in the broker forces the bytes in a message to match what the reader expects. A schema registry is the agreement layer that fixes this. It is a standalone service that holds the canonical definition of every message format, usually written in Avro, Protobuf, or JSON Schema.
Each schema is registered under a subject, which is typically tied to a Kafka topic. The registry stores every version of that subject and gives each unique schema a globally unique integer ID. Producers register or look up a schema, get back the ID, and write only that small ID into the message bytes. Consumers read the ID, fetch the matching schema from the registry, and decode the payload correctly.
The point is decoupling. Producers and consumers never have to share code or coordinate a release. They only have to agree, through the registry, on what a valid version of the data looks like.
Take the Confluent Schema Registry, the most common implementation. It runs as a REST service and stores all schemas in a special compacted Kafka topic called _schemas, so the registry itself has no separate database and survives restarts by replaying that topic.
On the produce side, the serializer first asks the registry: is this exact schema already registered for this subject? If yes, it gets the cached ID back. If no, the registry runs a compatibility check against existing versions, and if it passes, stores the new version and returns a fresh ID. The serializer then writes a 5 byte header (a magic byte plus the 4 byte schema ID) followed by the binary-encoded payload. The full schema text is never in the message.
On the consume side, the deserializer reads those first 5 bytes, extracts the ID, and fetches the schema from the registry over HTTP, caching it locally so the lookup happens once per ID, not once per message. With Avro it can decode using the writer schema while resolving fields against the reader schema, which is how a consumer on an old version still reads records written with a newer one.
Compatibility is the core safety feature. You set a mode per subject. BACKWARD (the default) means new consumers can read old data, so you may add optional fields or remove fields. FORWARD means old consumers can read new data. FULL requires both. NONE turns checking off. The registry rejects any registration that violates the rule, which stops a bad schema from ever reaching production.
Reach for a schema registry once more than one team or service reads a stream, or once you expect the data format to change over time, which is almost always. It pays off most in large event-driven and microservice systems where a single careless field rename could silently break dozens of downstream consumers.
The benefits are concrete: messages get smaller because the schema lives in the registry instead of in every record, breaking changes are caught at deploy time instead of at 3 AM, and new consumers can join a topic and immediately know how to parse it.
The costs are real too. The registry becomes shared infrastructure that needs to be highly available, because if producers cannot reach it to register a new schema, they may stall. Most setups run it in a cluster and rely on aggressive client-side caching so a brief outage does not stop reads. You also take on operational discipline: someone has to own compatibility policy, and JSON without a registry is simpler if you only have one producer and one consumer that ship together.
Confluent / Apache Kafka
The Confluent Schema Registry is the reference implementation, storing Avro, Protobuf, and JSON schemas in a compacted _schemas Kafka topic and exposing them over REST.
AWS Glue Schema Registry
Managed registry that integrates with Amazon MSK and Kinesis, validating and versioning schemas so streaming producers and consumers stay compatible.
Apicurio Registry
Open source registry from Red Hat that supports Avro, Protobuf, JSON Schema, and OpenAPI, used as a Confluent-compatible alternative in Kafka and event-driven stacks.
Pioneered large-scale Avro schema management for Kafka internally, which directly shaped the design of the schema registry pattern now in wide use.
No. That is the whole point. The producer writes a tiny header containing a 4 byte schema ID, and the consumer uses that ID to fetch the actual schema from the registry once and cache it. This keeps each message small.
Compatibility rules decide whether a new schema version is allowed. BACKWARD compatibility, the common default, lets new consumers read data written with old schemas, which permits adding optional fields or dropping fields. The registry rejects changes that break the rule, so an incompatible schema never reaches production.
Avro is the most established choice with Kafka and has rich schema evolution rules. Protobuf is a good fit if your services already use gRPC and Protobuf elsewhere. JSON Schema is the easiest to read and debug but produces larger messages. All three are supported by Confluent and most registries.
It is shared infrastructure, so it is run in a cluster for high availability. Clients heavily cache schemas locally, so a short outage does not stop consumers from decoding messages. It mainly affects producers that need to register a brand new schema during the outage.
No. If you have a single producer and a single consumer that are deployed together and the format rarely changes, plain JSON is simpler. A registry earns its keep once multiple independent teams read a stream or the data format is expected to evolve over time.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Protocol Buffers
Google's language-neutral, binary serialization format. Smaller and faster than JSON. Defines schemas in .proto files that generate typed code for any language.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
This page explains the idea. The "Schema Registry" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Schema Registry lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Schema Registry in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
