Apache Avro

Protocol Buffers require you to compile .proto files into language-specific code. Every time the schema changes, you recompile and redeploy. For tightly-coupled microservices, this is fine. For big data pipelines where data flows through dozens of systems, some of which you do not control, it becomes a burden.

Apache Avro takes a different approach. The schema is defined in JSON, stored alongside the data (or in a schema registry), and used at runtime. No code generation required. A reader can decode data using any compatible schema, even one that differs slightly from the writer's schema. This "schema resolution" makes Avro ideal for systems where producers and consumers evolve independently.

Avro is the default serialization format for Apache Kafka, Hadoop, and the broader data engineering ecosystem. When LinkedIn processes over 7 trillion messages per day through Kafka, those messages are Avro-encoded. Confluent's Schema Registry, which manages Avro schemas for Kafka, handles over 100 billion schema lookups per day across their cloud customers.

Doug Cutting, creator of both Hadoop and Avro, designed Avro specifically to address Protobuf's limitations in Hadoop. The key insight: in batch processing, the schema should travel with the data, not require a separate compilation step.

Schema (JSON):

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

Key differences from Protobuf:

• Schema is JSON, not a custom language. Any JSON parser can read it.
• No field numbers. Field matching uses names, resolved at runtime.
• Schema resolution: the reader's schema can differ from the writer's schema.
• No code generation required (though optional code generation is available for performance).
• Schema stored alongside the data (in the file header) or in a schema registry (for Kafka).

Schema evolution rules:

• Add field with default: safe (reader uses default for missing field)
• Remove field with default: safe (reader ignores extra field from writer)
• Rename field: use aliases to maintain compatibility
• Change type: limited (int to long is safe, int to string is not)

Schema resolution in detail. Writer wrote with schema v1 (fields: id, name). Reader has schema v2 (fields: id, name, email with default null). Avro resolution: map id to id, name to name, fill email with null. No recompilation. No coordination between teams. The reader adapts to whatever the writer produced, as long as the schemas are compatible.