Apache Kafka - the basics
What Kafka is - a distributed event log
Kafka is an event streaming platform that works as a distributed, durable log (an append-only log) - unlike a traditional message broker (such as RabbitMQ), where a message is deleted once it is received, Kafka retains messages for a configured time (retention), regardless of whether they have already been consumed.
This is exactly the technology you would naturally use to implement IEventBus from the Saga/Outbox examples we discussed earlier.
Key concepts
Topic - the channel you publish messages to
Topic: "order-events"
Analogous to a table in a database or a queue in RabbitMQ - a logical category of events of a similar nature (e.g. all events concerning orders).
Partitions - how Kafka scales throughput
Each topic is divided into partitions - independent, ordered logs which can be placed on different servers (brokers) in the cluster:
Topic: "order-events" (3 partycje)
├── Partition 0: [msg1, msg4, msg7, ...]
├── Partition 1: [msg2, msg5, msg8, ...]
└── Partition 2: [msg3, msg6, msg9, ...]
The key property - ordering is preserved ONLY within a single partition. Kafka guarantees the order of messages within one partition, but not across different partitions of the same topic.
// The partitioning key decides WHICH partition the message goes to
await producer.ProduceAsync("order-events", new Message<string, string>
{
Key = order.Id.ToString(), // all the events for THE SAME orderId go to THE SAME partition
Value = JsonSerializer.Serialize(orderEvent)
});
This is a practical consequence for a Saga - if you want the events concerning one order to be processed in order (e.g. OrderPlaced before PaymentCompleted), you have to use orderId as the partitioning key - Kafka will guarantee that all the events with the same key go to the same partition, and so are processed in the order they were published.
Consumer Groups - how many instances share the work
var config = new ConsumerConfig
{
GroupId = "inventory-service-group", // ALL the InventoryService instances belong to the same group
BootstrapServers = "localhost:9092"
};
The key rule: within one consumer group, each partition is assigned to ONLY ONE consumer at a given moment - this is the natural load balancing mechanism between multiple instances of the same service.
Topic "order-events" (3 partycje), Consumer Group "inventory-service-group" (3 instancje InventoryService)
Partition 0 → Consumer Instance A
Partition 1 → Consumer Instance B
Partition 2 → Consumer Instance C
If you run more InventoryService instances than there are partitions, the extra instances stay idle - this is a practical limitation to remember when planning scaling: the number of partitions limits the maximum parallel throughput within one consumer group.
Many different consumer groups can read the same topic independently - this is useful when both InventoryService and AnalyticsService want to process the same order-events events, but independently of each other:
Topic "order-events"
├── Consumer Group "inventory-service-group" - czyta i przetwarza dla logiki magazynowej
└── Consumer Group "analytics-service-group" - reads the SAME events, independently, for reporting
Producer - publishing messages
public class KafkaEventBus : IEventBus
{
private readonly IProducer<string, string> _producer;
public async Task PublishAsync<T>(T @event) where T : class
{
var topic = typeof(T).Name.ToLower(); // np. "orderplacedevent"
var payload = JsonSerializer.Serialize(@event);
await _producer.ProduceAsync(topic, new Message<string, string>
{
Key = Guid.NewGuid().ToString(),
Value = payload
});
}
}
Consumer - the listening loop
public class InventoryEventConsumer : BackgroundService
{
private readonly IConsumer<string, string> _consumer;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_consumer.Subscribe("order-events");
while (!stoppingToken.IsCancellationRequested)
{
var result = _consumer.Consume(stoppingToken); // BLOCKS until a message arrives
try
{
var orderEvent = JsonSerializer.Deserialize<OrderPlacedEvent>(result.Message.Value);
await ProcessEventAsync(orderEvent); // idempotent logic, as we discussed with the Saga/Outbox
_consumer.Commit(result); // confirms processing - the "offset" is moved forward
}
catch (Exception ex)
{
_logger.LogError(ex, "Error processing the message");
// NO Commit() - the same message will be delivered AGAIN
}
}
}
}
Offset - a "bookmark" saying where the consumer finished reading
Partition 0: [msg1, msg2, msg3, msg4, msg5]
↑
Consumer offset = 3 (it processed msg1, msg2, msg3)
Kafka remembers what offset (position) a given consumer group has reached in each partition - if the consumer crashes and comes back, it continues from the remembered offset, not from the beginning. This is the mechanism that gives "at-least-once delivery" (exactly the same guarantee we discussed with the Outbox Pattern) - if the consumer processed a message but crashed before committing (Commit) the new offset, the same message will be delivered again after the restart.
Retention - why Kafka is more than just a "queue"
Konfiguracja topicu: retention.ms = 604800000 (7 dni)
Unlike RabbitMQ (where a message is deleted after it is received by the consumer), Kafka retains messages for a configured period regardless of whether they were consumed. This makes possible: - Replay - a new consumer (or a new consumer group) can "replay" the history of events from the beginning - Many independent consumers reading the same data at different paces
Kafka vs a traditional message broker (RabbitMQ) - which one when
| Kafka | RabbitMQ | |
|---|---|---|
| Model | A distributed log (time-based retention) | A queue (deleted after being received) |
| Throughput | Very high (millions of messages/s) | High, but lower than Kafka |
| History replay | ✅ Yes (data kept for the retention period) | ❌ No (the message is deleted after being received) |
| Routing complexity | Simpler (topic + partitions) | Richer (exchanges, routing keys, bindings) |
| Typical use | Event streaming, event sourcing, large volumes of logs/events | Classic task queuing, RPC between services |
Summary - the relationship to the Saga/Outbox
Kafka is the natural technology for implementing
IEventBusin a Choreography Saga architecture - partitioning by key (e.g.orderId) guarantees the ordering of events concerning the same order, consumer groups provide load balancing between multiple instances of the same service, and the mechanism of offsets + no automatic deletion of messages supports exactly the kind of delivery guarantee ("at-least-once") we required from the Outbox Pattern - with the necessity that every consumer be idempotent, because the same message may be delivered more than once.