DanubeData now runs Apache Kafka as a managed service. You pick a plan, we run the broker: Kafka 4.3 in KRaft mode, a publicly trusted TLS endpoint, SASL/SCRAM-SHA-512 users with ACLs, snapshots, metrics and consumer lag, all in our Falkenstein, Germany location. Any standard Kafka client connects to it.
Kafka queues are available on request. They are not switched on for every account yet, so ask support to enable them for yours. This post explains what Kafka is, when it is the right tool next to RabbitMQ (which we also run as a managed queue), what exactly you get, and how to connect.
What Kafka is, in five minutes
Kafka is a distributed, append-only log. Producers write records to a topic. A topic is split into partitions, and every record in a partition gets an increasing number, its offset. Records with the same key go to the same partition, so their order is kept.
Consumers read the log by position. A consumer group is a set of consumers that share a topic: each partition is read by one member of the group, and the group stores its committed offset per partition. The difference between the end of a partition and the group's committed offset is the consumer lag, the number you watch to know whether your consumers keep up.
Two properties set Kafka apart from a classic queue:
- Reading does not remove a record. Records stay until the topic's retention (time or size) removes them, whoever has read them. A new consumer group can start from the beginning, and a group can be moved back to replay history.
- Many independent readers. Billing, search indexing and analytics can each read the same topic in their own group, at their own pace, without the producer knowing about them.
Kafka or RabbitMQ?
We offer both from the same Queues area of the console, API and CLI, so the choice is about your workload, not about what we happen to run.
| Question | RabbitMQ | Kafka |
|---|---|---|
| What is a message? | A task or request to hand to one worker, removed once acknowledged | A fact in a log, kept until retention removes it |
| Replay old data | Not its model | Yes, by moving a consumer group's offsets |
| Routing | Exchanges, bindings and routing keys | Topics and partitions, routed by key |
| Several consumers of the same data | One queue per consumer, fed by a fan-out | One topic, one consumer group per consumer |
| Protocols | AMQP, plus MQTT and STOMP on DanubeData | The Kafka protocol |
Pick RabbitMQ for background jobs, work queues, request/reply and anything with MQTT devices. Pick Kafka when the data is an event stream that several systems read, when you want to replay it, or when throughput per topic is the main concern. Many teams use both: RabbitMQ for the jobs, Kafka for the events. For a longer comparison that includes Amazon SQS, read RabbitMQ vs Kafka vs Amazon SQS.
What you get
- Kafka 4.3 in KRaft mode. No ZooKeeper. The cluster is managed by the Strimzi operator, and a cluster is one node that acts as both controller and broker.
- A public TLS endpoint with a publicly trusted certificate. Clients verify the server with the system trust store, so you do not need a CA file. The address looks like
kafka-<name>-<project>.queues.danubedata.ro, with a port shown in the console. - SASL/SCRAM-SHA-512 authentication with an admin user and up to 20 more users, each limited by ACLs.
- Persistent storage sized by the plan. Size and price are in the table below.
- Snapshots, taken by hand or every day, and a restore from the console.
- Metrics and consumer lag in the console and the API.
- A firewall you can narrow to your own addresses.
- Optional extras, also on request: a Schema Registry add-on and Kafka Studio, the console workspace for topics, messages and consumer groups.
One broker per cluster
Be clear about what this is. Version one is a single broker. There is no high availability and no replication across brokers, so topics use a replication factor of 1. If the broker restarts, clients are disconnected until it is back. That fits development, staging, pipelines whose source data you can replay, and event streams that tolerate a short pause. It does not fit a workload that must keep accepting writes through a broker failure. Multiple brokers are not available yet.
Plans and pricing
Three plans, billed hourly like every other DanubeData resource. A cluster is billed until you delete it.
| Plan | Console name | vCPU | RAM | Storage | Per month |
|---|---|---|---|---|---|
kafka-small |
DD Isac | 1 | 2 GB | 25 GB | €19.99 |
kafka-medium |
DD Golovita | 2 | 4 GB | 75 GB | €39.99 |
kafka-large |
DD Merhei | 4 | 8 GB | 200 GB | €79.99 |
The optional Schema Registry is a separate add-on at €4.99 a month, billed hourly while it is enabled. The pricing page always has the current figures. A plan above your account's limit shows Upgrade required with a link to request a limit increase.
Create a cluster
In the console, open Messaging, Apache Kafka and create a cluster: choose a plan, give it a name (lowercase letters, numbers and hyphens, 2 to 24 characters, fixed for the life of the cluster) and create it. Provisioning takes a few minutes while the public endpoint and its certificate are set up. When the cluster is ready we email you the bootstrap address, the SASL settings and the username. The email never contains the password.
From the API:
curl -X POST https://danubedata.ro/api/v1/queue \
-H "Authorization: Bearer $DANUBE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "events", "provider": "kafka", "resource_profile": "kafka-small", "version": "4.3"}'
Or with the danube CLI:
danube queue create --name events --provider kafka --profile kafka-small --version=4.3
danube queue connection-info events
Write the version as --version=4.3, with the equals sign.
Users and ACLs
Every request is checked against ACLs. The admin user, kafka-<name>-admin, is a super user and is never limited, so treat it like a root credential and keep it out of application code. Give each application its own user with only the access it needs. A user's name on the wire is kafka-<cluster>.<name>, and the password is generated and shown once.
A rule names a resource (topic, group, transactional_id or cluster), a name matched literally or by prefix, and the operations it allows. Everything not allowed is denied. A producer needs write and describe on its topics. A consumer needs read and describe on its topics and read on its consumer group. With the CLI:
danube queue users create events orders-app \
--acl topic:orders.:prefix:read,write,describe \
--acl group:billing:literal:read
That user can read and write every topic whose name starts with orders. and join the billing group, and nothing else. Today every rule is an allow rule that applies from any address: deny rules and rules limited to a client address are not available yet. Passwords can be rotated, and a rotation reaches new connections as soon as the user shows as active again.
Connect your client
The fastest start is the cluster's Connectivity tab. It shows the bootstrap server and settings, reveals the password on demand, and has a Download client.properties button. The file holds bootstrap.servers, security.protocol, sasl.mechanism and a sasl.jaas.config line, plus the Schema Registry settings when the registry is on. It signs in as the admin user, so use it for your own tools and give applications their own users. The same file comes from danube queue client-properties events -o events.properties and from the API.
The examples below read the address and credentials from environment variables:
export KAFKA_BOOTSTRAP='kafka-events-myproject.queues.danubedata.ro:29092'
export KAFKA_USER='kafka-events.orders-app'
export KAFKA_PASSWORD='<password>'
The address and port are examples. Your real bootstrap address is in the console and in connection-info.
Java
bootstrap.servers=<bootstrap address>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-events.orders-app" password="<password>";
No truststore is needed: the JVM's default trust store already trusts the certificate. If the password contains a backslash or a double quote, escape it for the JAAS line and again for the properties file.
Python (confluent-kafka)
import os
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
"security.protocol": "SASL_SSL",
"sasl.mechanisms": "SCRAM-SHA-512",
"sasl.username": os.environ["KAFKA_USER"],
"sasl.password": os.environ["KAFKA_PASSWORD"],
})
producer.produce("orders.created", b"hello")
producer.flush()
Node.js (KafkaJS)
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
brokers: [process.env.KAFKA_BOOTSTRAP],
ssl: true,
sasl: {
mechanism: 'scram-sha-512',
username: process.env.KAFKA_USER,
password: process.env.KAFKA_PASSWORD,
},
});
const producer = kafka.producer();
await producer.connect();
await producer.send({ topic: 'orders.created', messages: [{ value: 'hello' }] });
await producer.disconnect();
Go (franz-go)
package main
import (
"context"
"crypto/tls"
"log"
"net"
"os"
"time"
"github.com/twmb/franz-go/pkg/kgo"
"github.com/twmb/franz-go/pkg/sasl/scram"
)
func main() {
dialer := &tls.Dialer{NetDialer: &net.Dialer{Timeout: 10 * time.Second}}
client, err := kgo.NewClient(
kgo.SeedBrokers(os.Getenv("KAFKA_BOOTSTRAP")),
kgo.Dialer(dialer.DialContext),
kgo.SASL(scram.Auth{
User: os.Getenv("KAFKA_USER"),
Pass: os.Getenv("KAFKA_PASSWORD"),
}.AsSha512Mechanism()),
)
if err != nil {
log.Fatal(err)
}
defer client.Close()
record := &kgo.Record{Topic: "orders.created", Value: []byte("hello")}
if err := client.ProduceSync(context.Background(), record).FirstErr(); err != nil {
log.Fatal(err)
}
}
kcat
kcat -L -b "$KAFKA_BOOTSTRAP" \
-X security.protocol=SASL_SSL \
-X sasl.mechanisms=SCRAM-SHA-512 \
-X sasl.username="$KAFKA_USER" \
-X sasl.password="$KAFKA_PASSWORD"
Desktop tools such as Kafkio, Offset Explorer and Conduktor work with the same settings. Most of them default to SASL_PLAINTEXT with the PLAIN mechanism, which the cluster refuses, so change the security protocol to SASL_SSL and the mechanism to SCRAM-SHA-512, and leave the truststore empty.
Create topics
You can create a topic from your application, from Kafka's own tools, or from Kafka Studio in the console. With the Kafka 4.x command-line tools and the downloaded properties file:
kafka-topics.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
--command-config client.properties \
--create --topic orders.created --partitions 3 --replication-factor 1
Use --replication-factor 1. The cluster has one broker, so a higher factor is rejected.
Operating a cluster
Metrics and consumer lag
The cluster's Metrics tab shows its health with the reason for any warning, then CPU, memory, disk used against the plan's storage, messages in per second, bytes in and out per second, partition count and consumer lag. Charts cover 1 hour to 1 week, and a table lists the 50 consumer groups with the most lag. The same figures are in GET /api/v1/queue/{id}/metrics.
Snapshots and restore
A snapshot is a copy of the broker's volume at one moment: every topic and message, consumer group offsets and, with the registry, its schemas. Take one from the Backups tab while the cluster keeps running, or turn on automated daily backups: one is taken at 02:00 UTC and each automated snapshot is kept for 3 days. Snapshots you take yourself stay until you delete them.
Know what a snapshot is. It is taken the way a sudden power cut would leave the disk, so writes from the last seconds that were not yet on disk may be missing. Restoring replaces all data with the snapshot's, resets consumer groups to the offsets they had then, and disconnects every client while the broker restarts. Snapshots live on the same server as the cluster: they protect against mistakes such as a deleted topic or a bad deploy, not against losing the server. If you need a copy elsewhere, consume your topics into your own storage. Taking and restoring snapshots through the API and CLI is not available yet, only in the console.
Firewall
The default firewall opens only the Kafka port, to any address, and every connection must authenticate. If your clients have fixed addresses, edit the rule on the cluster's Security tab and allow only those, IPv4 and IPv6. Once a cluster is limited, a workload inside DanubeData must connect from its public address, which you allow like any other. See Firewalls.
What is not available yet
- Multiple brokers and replication across brokers
- Stop and start, plan changes (resizing) and changes to broker settings
- Deny rules and rules limited to a client address
- Taking or restoring snapshots from the API or CLI
The API answers "not available for Kafka yet" to stop, start, plan and settings changes, rather than failing silently.
Frequently asked questions
Is the cluster highly available?
No. A cluster is one broker. A restart disconnects clients until it is back, and data is not replicated to another broker. Use snapshots for protection against mistakes and keep your source of truth somewhere you can replay from.
Which Kafka version do I get?
Kafka 4.3, running in KRaft mode. There is no ZooKeeper to manage or pay for.
Do I need a CA certificate file?
No. The endpoint uses a publicly trusted certificate, so the default trust store of your language runtime accepts it.
Where is the data stored?
In Falkenstein, Germany, on the cluster's persistent volume. Deleting a cluster deletes its topics and data, and its snapshots can no longer be restored, so export what you need first.
Can I use Confluent's serializers?
Yes, with the optional Schema Registry add-on, which speaks the Confluent API. See our guide to the Schema Registry.
How do I get access?
Kafka queues are available on request. Ask support to enable them for your account, then create a cluster as described above. The full reference is in the Managed Kafka documentation.
See pricing or contact our team to get Kafka enabled.