Schema Registry for Kafka: Avro, Protobuf, JSON Schema

Why Kafka needs a schema registry, how subjects, versions and compatibility modes work, and how to evolve Avro schemas safely. €4.99 a month.

Adrian Silaghi

10 min read

10 views

kafkaschema-registryavroprotobufjson-schemakarapaceschema-evolution

Kafka moves bytes. It has no idea what is inside a record, and that is the source of a familiar failure: a producer team renames a field, deploys on a Tuesday, and three consumer teams find out from their error logs. A schema registry is the usual fix. It stores the schema of every record format, gives each one an id, and refuses a new schema that would break existing readers.

This guide explains how a registry works, what the compatibility modes mean, and how to change a schema without breaking anyone, with worked Avro examples. It also covers the Schema Registry add-on for DanubeData's managed Kafka: €4.99 a month, billed hourly while it is enabled, and available on request. Ask support to turn it on for your account. If you have not set up a cluster yet, start with Managed Apache Kafka in the EU.

Why schemas matter in Kafka

A topic is a log that outlives its producers. A consumer may read a record written today, last week or, after an offset reset, last year. Three things follow:

  • Producers and consumers deploy on their own schedules. At any moment there are old and new versions of both in production.
  • Old records stay in the log. A new consumer must still understand them.
  • Nothing in Kafka checks the payload. A record that no consumer can parse is accepted like any other.

A registry turns the format into an explicit contract with rules for changing it. Putting the full schema in every record would waste space, so the record carries only a small id, and the registry maps the id to the schema.

Avro, Protobuf and JSON Schema

Our registry stores all three formats that Confluent-compatible tools use:

Format On the wire Worth knowing
Avro Compact binary, no field names in the data A reader needs the writer's schema to decode, which is exactly what the registry supplies. The usual choice in Kafka, with defaults and aliases for evolution.
Protobuf Compact binary, numbered fields Good fit if you already use Protobuf for gRPC. Schemas can import other schemas, which the registry stores as references.
JSON Schema Plain JSON Readable with any tool, larger on the wire. Gives JSON payloads a checked contract.

If you have no strong preference, start with Avro. The examples below use it, and the compatibility modes work the same for the others, with format-specific rules for what counts as a breaking change.

Subjects and versions

The registry organizes schemas into subjects. A subject is a name under which versions of a schema are kept, and by convention it is the topic name followed by -value or -key: the value schema of the topic orders.created lives in the subject orders.created-value.

Each time you register a schema that differs from the latest one, the subject gets the next version: 1, 2, 3. Registering a schema the subject already has changes nothing and answers the existing id and version, so repeated registration from every deploy is harmless. Every schema also has an id, the number that travels with the records.

The Confluent wire format

Confluent's serializers, and the tools built on the same convention, prefix every record's key or value with five bytes:

byte 0       magic byte, always 0
bytes 1-4    schema id, a big-endian 32-bit integer
bytes 5...   the payload, encoded with that schema

A deserializer reads the id, fetches the schema from the registry once, caches it, and decodes. This is also how a browser tool such as Kafka Studio shows readable data: when it sees the magic byte and an id it can resolve, it decodes the payload with the registered schema. Records written without the prefix are plain bytes to it, shown as JSON, text or base64.

Compatibility modes

Before a new version is accepted, the registry compares it with the existing ones under the subject's compatibility rule. Seven values exist:

Mode The new schema must Upgrade first
BACKWARD be able to read data written with the previous version Consumers
BACKWARD_TRANSITIVE be able to read data written with every earlier version Consumers
FORWARD produce data the previous version can read Producers
FORWARD_TRANSITIVE produce data every earlier version can read Producers
FULL satisfy both rules against the previous version Any order
FULL_TRANSITIVE satisfy both rules against every earlier version Any order
NONE nothing: no check is made You are on your own

Two things trip people up.

The direction is about the reader. BACKWARD means the new schema reads old data. With BACKWARD_TRANSITIVE it is the mode to pick when you want to replay a topic from the start, because consumers on the newest schema can then read everything in the log. FORWARD means old consumers can read data written with the new schema, which lets you upgrade producers while old consumers still run.

Without TRANSITIVE, only the latest version is checked. Version 3 can be compatible with version 2 and version 2 with version 1, and still be incompatible with version 1. If your log holds records of all three versions and consumers may read them all, use a transitive mode.

You can set a rule for the whole registry and override it per subject. A subject without its own rule follows the registry's global level; Kafka Studio shows that as Registry default.

Evolving a schema safely: a worked example

Start with an Order record, registered as version 1 of orders.created-value:

{
  "type": "record",
  "name": "Order",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "amount_cents", "type": "long"}
  ]
}

A safe change: add a field with a default

You need the currency. Add it with a default:

{
  "type": "record",
  "name": "Order",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency", "type": "string", "default": "EUR"}
  ]
}

This is compatible in both directions. A new reader that meets an old record, which has no currency, fills in the default (BACKWARD). An old reader that meets a new record ignores the field it does not know (FORWARD). It passes under FULL, and it is the pattern to reach for first.

A breaking change: add a field without a default

{
  "type": "record",
  "name": "Order",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency", "type": "string", "default": "EUR"},
    {"name": "customer_id", "type": "string"}
  ]
}

A new reader that meets an old record has no customer_id and no default to fall back on, so it cannot decode the record. Under BACKWARD the registry rejects this version.

Check before you register

You do not have to find out by deploying. A candidate schema can be tested against a subject's rule without registering it. In the console, the schema form of Kafka Studio has a Check compatibility button, and the same check is in the API. The schema goes in as a string:

curl -X POST "https://danubedata.ro/api/v1/queue/$QUEUE_ID/kafka/schemas/subjects/orders.created-value/compatibility" \
  -H "Authorization: Bearer $DANUBE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"schema_type": "AVRO", "schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"com.example.orders\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount_cents\",\"type\":\"long\"},{\"name\":\"customer_id\",\"type\":\"string\"}]}"}'

The answer is {"data": {"is_compatible": false, "messages": [...]}}, and the messages say what breaks. Registering that schema would be refused with a 409. Put the check in your CI so a breaking schema fails the pull request, not the deploy.

The rules of thumb for Avro

  • Add fields with a default. Compatible in both directions.
  • Remove only fields that have a default. Removing a field without one breaks FORWARD, because an old reader still expects it.
  • Never rename in place. A rename is a removal plus an addition. Add the new name with the old one as an aliases entry, and plan the migration.
  • Widen numeric types, never narrow them. Avro lets a reader promote int to long, so that change is backward compatible.
  • Make a field you want to remove optional first. Give it a default in one version, remove it in a later one.

Rolling out a compatible change

The mode tells you the order. Under BACKWARD, deploy consumers with the new schema first, then producers: consumers can already read both generations. Under FORWARD, deploy producers first, because old consumers can read the new data. Under FULL the order does not matter, which is why it is the comfortable choice for teams that do not control all their consumers.

Using the registry on DanubeData

The registry is Karapace, an open-source registry that implements Confluent's REST API, so Confluent's serializers and most Kafka tools talk to it unchanged. It stores its schemas in the cluster's own _schemas topic.

Enable it

In the console, open the cluster and turn on Schema Registry. With the CLI, danube queue schema-registry enable events asks you to confirm the monthly price first. The API call is POST /api/v1/queue/{id}/schema-registry. The cluster must be running, and the registry is ready within a few minutes. It gets its own HTTPS endpoint with a publicly trusted certificate:

https://schema-<name>-<project>.queues.danubedata.ro

Clients sign in with HTTP Basic authentication. The username is registry, and the password is the registry's own, separate from the Kafka admin password, shown on the cluster page to anyone who may read the cluster's credentials. The registry follows the cluster's firewall: the addresses the firewall lets reach Kafka may reach the registry, and the others are refused. The password is what protects it from everything else, so keep it secret and rotate it if it leaks.

Produce Avro from Python

import os
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import MessageField, SerializationContext

registry = SchemaRegistryClient({
    "url": "https://schema-events-<project>.queues.danubedata.ro",
    "basic.auth.user.info": "registry:" + os.environ["SCHEMA_REGISTRY_PASSWORD"],
})
schema = '{"type": "record", "name": "Order", "fields": [{"name": "id", "type": "string"}]}'
serialize = AvroSerializer(registry, schema)

producer = Producer({
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "SCRAM-SHA-512",
    "sasl.username": os.environ["KAFKA_USER"],
    "sasl.password": os.environ["KAFKA_PASSWORD"],
})
producer.produce("orders", value=serialize({"id": "42"}, SerializationContext("orders", MessageField.VALUE)))
producer.flush()

Java

Add these to the producer or consumer properties, next to the SASL settings, and use Confluent's serializer such as io.confluent.kafka.serializers.KafkaAvroSerializer:

schema.registry.url=https://schema-events-<project>.queues.danubedata.ro
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=registry:<password>

The client.properties file you can download from the cluster's Connectivity tab already contains these three lines once the registry is on.

Turn off auto-registration in production

Confluent's serializers register the schema they are given the first time they use it, unless you set auto.register.schemas=false. That is convenient in development. In production, prefer to register schemas from your CI after a compatibility check and let the application only look them up, so that a code change cannot alter a contract by accident.

Good to know

  • Do not write to or delete _schemas. That topic is the registry's data.
  • Disabling the registry stops billing and removes the endpoint, but the _schemas topic stays. Enable it again and your subjects and schemas are back, with the same password.
  • Deleting a cluster deletes its registry and schemas. Snapshots of the cluster include the schemas, since they live in the cluster's topic.
  • Deletes are soft first. Deleting a subject or version keeps the schema readable by id, and registering the same schema again restores it. A permanent delete removes it for good, ids included, so records that carry that id can no longer be decoded.
  • Rate limit: the registry's endpoint allows 50 requests per second per client address. Serializers cache schemas, so normal use stays far below it.
  • One broker. The registry runs next to a single-broker cluster and keeps its data in that broker's _schemas topic.

Frequently asked questions

Is it compatible with Confluent Schema Registry?

It implements Confluent's REST API, and Confluent's serializers and most Kafka tools work against it. It is Karapace, not Confluent's own registry, so a feature that is specific to Confluent Platform is not part of it.

Which schema types are supported?

Avro, Protobuf and JSON Schema. A schema can reference other registered schemas by name, subject and version.

What does it cost?

€4.99 a month, billed by the hour while it is enabled, on top of the cluster. See the pricing page for current figures.

Do I need a registry at all?

If one team owns both ends of a topic and the data is short-lived, you may not. As soon as several teams read a topic, or old records stay in the log, a registry is cheaper than the first incident it prevents.

Where do I browse and manage schemas?

In the console, Kafka Studio lists subjects and versions, compares two versions, checks compatibility and registers new versions. The same operations are in the API and the MCP server. The reference is in the Managed Kafka documentation.

See pricing or contact our team to get the Schema Registry enabled.

Browse all articles
Queues9 min read

Introducing Kafka Studio for DanubeData Kafka

Kafka Studio is a console workspace for topics, messages, consumer groups and schemas. It is also in the API and the MCP server. Available on request.

Adrian Silaghi11 views
Queues11 min read

Managed Apache Kafka in the EU, on DanubeData

Managed Apache Kafka 4.3 in KRaft mode, hosted in Germany: TLS, SCRAM users with ACLs, snapshots and lag metrics. Plans from €19.99 a month.

Adrian Silaghi14 views

Ready to get started?

Deploy your infrastructure in minutes with DanubeData's managed services.