Apache Kafka vs RabbitMQ - comparison and implementation in C#
What they are - a fundamental difference in model
RabbitMQ is a classic message broker - it implements a queue model: a message goes into a queue, waits, is delivered and deleted. It speaks the AMQP protocol (which we discussed in detail with Rebus and RabbitMQ) - a binary protocol with a push model, where the broker itself pushes the message to a ready consumer.
Kafka is a distributed event log (a distributed event streaming platform) - messages are durably stored in partitions for a configured time (retention), regardless of whether anyone has consumed them. It is not a "queue" in the classic sense, but an append-only log which can be read many times, independently.
Comparison table
| RabbitMQ | Kafka | |
|---|---|---|
| Model | A queue (message broker) | A distributed log (event streaming) |
| Protocol | AMQP (binary, push to the consumer) | Its own binary TCP protocol |
| Durability after being received | The message is deleted after the ACK | The message stays (time-based retention, e.g. 7 days) |
| History replay | ❌ No (once received, it's gone) | ✅ Yes (a new consumer can replay from the beginning) |
| Throughput | High | Very high (millions of messages/s) |
| Routing | Rich (exchange: direct/topic/fanout/headers) | Simpler (topic + partitions + key) |
| Ordering | Per queue | Per partition (not global within a topic) |
| Typical use | Task queuing, RPC, commands between services | Event sourcing, log streaming, large volumes of events, analytics |
| Learning curve | Lower | Higher (partitions, consumer groups, offsets) |
In the shortest terms: with RabbitMQ you ask "do this task" (a command, to be executed once), with Kafka you say "this happened" (an event, which many can read, many times, later).
Implementation in C# - RabbitMQ (the RabbitMQ.Client library)
Producer
using RabbitMQ.Client;
using System.Text;
using System.Text.Json;
public class RabbitMqProducer : IDisposable
{
private readonly IConnection _connection;
private readonly IModel _channel;
public RabbitMqProducer(string hostName = "localhost")
{
var factory = new ConnectionFactory { HostName = hostName };
_connection = factory.CreateConnection();
_channel = _connection.CreateModel();
// Queue declaration - idempotent, safe to call multiple times
_channel.QueueDeclare(queue: "orders-queue", durable: true, exclusive: false, autoDelete: false);
}
public void Publish(OrderPlacedEvent orderEvent)
{
var json = JsonSerializer.Serialize(orderEvent);
var body = Encoding.UTF8.GetBytes(json);
var properties = _channel.CreateBasicProperties();
properties.Persistent = true; // the message survives a broker restart
_channel.BasicPublish(exchange: "", routingKey: "orders-queue", basicProperties: properties, body: body);
}
public void Dispose()
{
_channel?.Dispose();
_connection?.Dispose();
}
}
Consumer
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.Text;
using System.Text.Json;
public class RabbitMqConsumer : IDisposable
{
private readonly IConnection _connection;
private readonly IModel _channel;
public RabbitMqConsumer(string hostName = "localhost")
{
var factory = new ConnectionFactory { HostName = hostName };
_connection = factory.CreateConnection();
_channel = _connection.CreateModel();
_channel.QueueDeclare(queue: "orders-queue", durable: true, exclusive: false, autoDelete: false);
}
public void StartConsuming()
{
var consumer = new EventingBasicConsumer(_channel);
// PUSH - the broker ITSELF invokes this handler when a message arrives
consumer.Received += (model, ea) =>
{
var json = Encoding.UTF8.GetString(ea.Body.ToArray());
var orderEvent = JsonSerializer.Deserialize<OrderPlacedEvent>(json);
Console.WriteLine($"Received: order {orderEvent.OrderId}");
_channel.BasicAck(ea.DeliveryTag, multiple: false); // potwierdzenie odbioru
};
_channel.BasicConsume(queue: "orders-queue", autoAck: false, consumer: consumer);
}
public void Dispose()
{
_channel?.Dispose();
_connection?.Dispose();
}
}
Key elements: autoAck: false + a manual BasicAck - until you acknowledge, RabbitMQ considers the message "undelivered" and will hand it out again (e.g. if the consumer crashes during processing) - this is the same "at-least-once delivery" mechanism we discussed with the Outbox Pattern.
Implementation in C# - Kafka (the Confluent.Kafka library)
Producer
using Confluent.Kafka;
public class KafkaProducer : IDisposable
{
private readonly IProducer<string, string> _producer;
public KafkaProducer(string bootstrapServers = "localhost:9092")
{
var config = new ProducerConfig { BootstrapServers = bootstrapServers };
_producer = new ProducerBuilder<string, string>(config).Build();
}
public async Task PublishAsync(OrderPlacedEvent orderEvent)
{
var json = System.Text.Json.JsonSerializer.Serialize(orderEvent);
await _producer.ProduceAsync("order-events", new Message<string, string>
{
Key = orderEvent.OrderId.ToString(), // decides WHICH partition it goes to (ordering per key)
Value = json
});
}
public void Dispose() => _producer?.Dispose();
}
Consumer
using Confluent.Kafka;
public class KafkaConsumer
{
private readonly IConsumer<string, string> _consumer;
public KafkaConsumer(string bootstrapServers = "localhost:9092", string groupId = "order-service-group")
{
var config = new ConsumerConfig
{
BootstrapServers = bootstrapServers,
GroupId = groupId, // consumer group - shares the partitions among the instances of the same group
AutoOffsetReset = AutoOffsetReset.Earliest
};
_consumer = new ConsumerBuilder<string, string>(config).Build();
}
public void Consume(CancellationToken cancellationToken)
{
_consumer.Subscribe("order-events");
while (!cancellationToken.IsCancellationRequested)
{
var result = _consumer.Consume(cancellationToken); // blocks until a message arrives
var orderEvent = System.Text.Json.JsonSerializer.Deserialize<OrderPlacedEvent>(result.Message.Value);
Console.WriteLine($"Received from partition {result.Partition}: order {orderEvent.OrderId}");
_consumer.Commit(result); // saves the offset - "I've read up to here"
}
}
}
The key difference visible in the code: Kafka doesn't have a BasicAck for a single message - instead Commit() saves the offset (a position in the partition). This reflects the fundamental difference in the model: RabbitMQ acknowledges a specific message, Kafka acknowledges a point in the log up to which you have read.
A working example - run RabbitMQ and Kafka and see the difference
Below is a complete, minimal console project (.NET 8) that you can run: one application with four modes (rabbit-send, rabbit-receive, kafka-send, kafka-receive) and the brokers in Docker. Just copy the files below into one MessagingDemo folder.
1. The brokers in Docker - docker-compose.yml
services:
rabbitmq:
image: rabbitmq:3-management
ports:
- "5672:5672" # AMQP (used by the app)
- "15672:15672" # management UI: http://localhost:15672 (guest / guest)
kafka:
image: apache/kafka:3.8.0 # single node, KRaft mode (no ZooKeeper)
ports:
- "9092:9092"
docker compose up -d # poczekaj ok. 20 s na start Kafki
2. The project - MessagingDemo.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="RabbitMQ.Client" Version="7.0.0" />
<PackageReference Include="Confluent.Kafka" Version="2.6.0" />
</ItemGroup>
</Project>
Creating it from scratch: dotnet new console -n MessagingDemo, then dotnet add package RabbitMQ.Client and dotnet add package Confluent.Kafka. The example uses the asynchronous RabbitMQ.Client 7.x API (CreateConnectionAsync, BasicPublishAsync, AsyncEventingBasicConsumer).
3. The message - OrderMessage.cs
namespace MessagingDemo;
// The message sent through both brokers (serialized as JSON).
public record OrderMessage(int Id, string Customer, decimal Amount);
4. RabbitMQ - RabbitDemo.cs
using System.Text.Json;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
namespace MessagingDemo;
public static class RabbitDemo
{
private const string QueueName = "orders";
private static ConnectionFactory CreateFactory() => new()
{
HostName = "localhost",
Port = 5672,
UserName = "guest",
Password = "guest"
};
// PRODUCER: publishes messages to a queue (default exchange routes by queue name).
public static async Task SendAsync(int count)
{
await using var connection = await CreateFactory().CreateConnectionAsync();
await using var channel = await connection.CreateChannelAsync();
// Idempotent: creates the queue if it does not exist. Durable = survives broker restart.
await channel.QueueDeclareAsync(queue: QueueName, durable: true, exclusive: false, autoDelete: false);
for (var i = 1; i <= count; i++)
{
var order = new OrderMessage(i, $"Customer {i}", 10m * i);
var body = JsonSerializer.SerializeToUtf8Bytes(order);
var props = new BasicProperties { Persistent = true, ContentType = "application/json" };
await channel.BasicPublishAsync(
exchange: string.Empty, // default exchange: routingKey == queue name
routingKey: QueueName,
mandatory: false,
basicProperties: props,
body: body);
Console.WriteLine($"[RabbitMQ] sent: {order}");
}
}
// CONSUMER: the broker PUSHES messages; we must ACK each one, otherwise it is redelivered.
public static async Task ReceiveAsync(CancellationToken ct)
{
await using var connection = await CreateFactory().CreateConnectionAsync();
await using var channel = await connection.CreateChannelAsync();
await channel.QueueDeclareAsync(queue: QueueName, durable: true, exclusive: false, autoDelete: false);
// Fair dispatch: at most 1 unacknowledged message per consumer.
await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 1, global: false);
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.ReceivedAsync += async (_, ea) =>
{
var order = JsonSerializer.Deserialize<OrderMessage>(ea.Body.Span);
Console.WriteLine($"[RabbitMQ] received: {order}");
await Task.Delay(300); // simulate work
// Confirm THIS message. After the ACK the broker deletes it from the queue.
await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
};
await channel.BasicConsumeAsync(queue: QueueName, autoAck: false, consumer: consumer);
Console.WriteLine("[RabbitMQ] waiting for messages... (Ctrl+C to quit)");
try { await Task.Delay(Timeout.Infinite, ct); }
catch (OperationCanceledException) { }
}
}
5. Kafka - KafkaDemo.cs
using System.Text.Json;
using Confluent.Kafka;
using Confluent.Kafka.Admin;
namespace MessagingDemo;
public static class KafkaDemo
{
private const string Topic = "orders";
private const string Bootstrap = "localhost:9092";
// Creates the topic with 3 partitions (does nothing if it already exists).
private static async Task EnsureTopicAsync()
{
using var admin = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = Bootstrap }).Build();
try
{
await admin.CreateTopicsAsync(new[]
{
new TopicSpecification { Name = Topic, NumPartitions = 3, ReplicationFactor = 1 }
});
}
catch (CreateTopicsException e) when (e.Results.All(r => r.Error.Code == ErrorCode.TopicAlreadyExists))
{
// already there - fine
}
}
// PRODUCER: appends messages to the log. The KEY decides the partition
// (same key => same partition => ordering is preserved per key).
public static async Task SendAsync(int count)
{
await EnsureTopicAsync();
var config = new ProducerConfig { BootstrapServers = Bootstrap, Acks = Acks.All };
using var producer = new ProducerBuilder<string, string>(config).Build();
for (var i = 1; i <= count; i++)
{
var order = new OrderMessage(i, $"Customer {i}", 10m * i);
var message = new Message<string, string>
{
Key = $"customer-{i % 3}",
Value = JsonSerializer.Serialize(order)
};
var result = await producer.ProduceAsync(Topic, message);
Console.WriteLine($"[Kafka] sent: {order} -> partition {result.Partition.Value}, offset {result.Offset.Value}");
}
}
// CONSUMER: PULLS messages (Consume) and remembers its position (offset) per consumer GROUP.
// Messages are NOT deleted - another group (or a reset offset) can read them again.
public static async Task ReceiveAsync(string groupId, CancellationToken ct)
{
await EnsureTopicAsync();
var config = new ConsumerConfig
{
BootstrapServers = Bootstrap,
GroupId = groupId,
AutoOffsetReset = AutoOffsetReset.Earliest, // new group starts from the beginning of the log
EnableAutoCommit = false // we commit the offset manually
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(Topic);
Console.WriteLine($"[Kafka] group '{groupId}' waiting for messages... (Ctrl+C to quit)");
try
{
while (!ct.IsCancellationRequested)
{
var cr = consumer.Consume(ct);
var order = JsonSerializer.Deserialize<OrderMessage>(cr.Message.Value);
Console.WriteLine($"[Kafka] received: {order} (key {cr.Message.Key}, partition {cr.Partition.Value}, offset {cr.Offset.Value})");
await Task.Delay(300); // simulate work
// Save the position in the log - NOT an ACK of a single message.
consumer.Commit(cr);
}
}
catch (OperationCanceledException) { }
finally
{
consumer.Close(); // leave the group cleanly
}
}
}
6. The entry point - Program.cs
using MessagingDemo;
// Usage: dotnet run -- <mode> [argument]
// rabbit-send [count] publish messages to RabbitMQ
// rabbit-receive consume messages from RabbitMQ
// kafka-send [count] publish messages to Kafka
// kafka-receive [groupId] consume messages from Kafka (default group: demo-group)
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
var mode = args.Length > 0 ? args[0] : string.Empty;
var arg = args.Length > 1 ? args[1] : null;
var count = int.TryParse(arg, out var n) ? n : 5;
try
{
switch (mode)
{
case "rabbit-send": await RabbitDemo.SendAsync(count); break;
case "rabbit-receive": await RabbitDemo.ReceiveAsync(cts.Token); break;
case "kafka-send": await KafkaDemo.SendAsync(count); break;
case "kafka-receive": await KafkaDemo.ReceiveAsync(arg ?? "demo-group", cts.Token); break;
default:
Console.WriteLine("Usage: dotnet run -- <rabbit-send|rabbit-receive|kafka-send|kafka-receive> [argument]");
break;
}
}
catch (Exception ex)
{
Console.WriteLine($"Error: {ex.Message}");
Console.WriteLine("Is the broker running? Start it with: docker compose up -d");
Environment.ExitCode = 1;
}
7. Scenarios to try
RabbitMQ - the message disappears after being received:
dotnet run -- rabbit-send 5 # 5 messages go to the "orders" queue
dotnet run -- rabbit-receive # receives 5 messages (Ctrl+C to quit)
dotnet run -- rabbit-receive # again: empty, because after the ACK the messages were deleted
Run two rabbit-receive instances at once and send more messages - they will be distributed between the consumers (competing consumers). The management panel: http://localhost:15672 (guest / guest).
Kafka - the log can be read many times:
dotnet run -- kafka-send 6 # the keys customer-0/1/2 go to different partitions
dotnet run -- kafka-receive # the group "demo-group" reads everything from the beginning
dotnet run -- kafka-receive # ta sama grupa: nic nowego (offset zapisany)
dotnet run -- kafka-receive another-group # a NEW group reads the whole history again
What this example shows
- The same message, a different fate: in RabbitMQ after
BasicAckthe message disappears from the queue; in Kafka afterCommitthe consumer merely remembers a position in the log, and the messages stay and can be read by another group. - Push vs pull: RabbitMQ pushes messages to the consumer (
ReceivedAsync), while a Kafka consumer pulls them itself (Consume). - Key and partitions: in Kafka the
Keydecides the partition, so messages with the same key keep their order; in the output you can seepartitionandoffset. - Competing consumers vs consumer groups: many consumers of one queue share the messages; in Kafka many members of one group share them (by partitions), and every further group gets the full set.
- Cleaning up: after the tests
docker compose down -vremoves the brokers together with their data.
When to use which
RabbitMQ - when you have commands/tasks that should be executed once, with rich routing (fanout to many queues, routing by patterns) and a lower learning curve. Kafka - when you have business events that many independent systems want to read (sometimes at different paces), you need history replay, or you are scaling to very large volumes (logs, telemetry, event sourcing).
The relationship to the other topics on this site
This comparison brings together in one place threads scattered across the other topics of the "Distributed" category: Rebus (and MassTransit/NServiceBus) are libraries that abstract over a specific transport - they can use RabbitMQ or (less often) Kafka "under the hood", without changing the handler code. The Saga Pattern and the Outbox Pattern apply to both technologies equally - whether you publish through RabbitMQ.Client or Confluent.Kafka, you need the same guarantee of atomicity between the write and the publishing of the event.