C# / .NET Compendium PLEN
.NET / C# / Distributed architecture

Apache Kafka - the basics

What Kafka is - a distributed event log

Kafka is an event streaming platform that works as a distributed, durable log (an append-only log) - unlike a traditional message broker (such as RabbitMQ), where a message is deleted once it is received, Kafka retains messages for a configured time (retention), regardless of whether they have already been consumed.

This is exactly the technology you would naturally use to implement IEventBus from the Saga/Outbox examples we discussed earlier.

Key concepts

Topic - the channel you publish messages to

Topic: "order-events"

Analogous to a table in a database or a queue in RabbitMQ - a logical category of events of a similar nature (e.g. all events concerning orders).

Partitions - how Kafka scales throughput

Each topic is divided into partitions - independent, ordered logs which can be placed on different servers (brokers) in the cluster:

Topic: "order-events" (3 partycje)
├── Partition 0: [msg1, msg4, msg7, ...]
├── Partition 1: [msg2, msg5, msg8, ...]
└── Partition 2: [msg3, msg6, msg9, ...]

The key property - ordering is preserved ONLY within a single partition. Kafka guarantees the order of messages within one partition, but not across different partitions of the same topic.

// The partitioning key decides WHICH partition the message goes to
await producer.ProduceAsync("order-events", new Message<string, string>
{
    Key = order.Id.ToString(), // all the events for THE SAME orderId go to THE SAME partition
    Value = JsonSerializer.Serialize(orderEvent)
});

This is a practical consequence for a Saga - if you want the events concerning one order to be processed in order (e.g. OrderPlaced before PaymentCompleted), you have to use orderId as the partitioning key - Kafka will guarantee that all the events with the same key go to the same partition, and so are processed in the order they were published.

Consumer Groups - how many instances share the work

var config = new ConsumerConfig
{
    GroupId = "inventory-service-group", // ALL the InventoryService instances belong to the same group
    BootstrapServers = "localhost:9092"
};

The key rule: within one consumer group, each partition is assigned to ONLY ONE consumer at a given moment - this is the natural load balancing mechanism between multiple instances of the same service.

Topic "order-events" (3 partycje), Consumer Group "inventory-service-group" (3 instancje InventoryService)

Partition 0 → Consumer Instance A
Partition 1 → Consumer Instance B
Partition 2 → Consumer Instance C

If you run more InventoryService instances than there are partitions, the extra instances stay idle - this is a practical limitation to remember when planning scaling: the number of partitions limits the maximum parallel throughput within one consumer group.

Many different consumer groups can read the same topic independently - this is useful when both InventoryService and AnalyticsService want to process the same order-events events, but independently of each other:

Topic "order-events"
├── Consumer Group "inventory-service-group" - czyta i przetwarza dla logiki magazynowej
└── Consumer Group "analytics-service-group" - reads the SAME events, independently, for reporting

Producer - publishing messages

public class KafkaEventBus : IEventBus
{
    private readonly IProducer<string, string> _producer;

    public async Task PublishAsync<T>(T @event) where T : class
    {
        var topic = typeof(T).Name.ToLower(); // np. "orderplacedevent"
        var payload = JsonSerializer.Serialize(@event);

        await _producer.ProduceAsync(topic, new Message<string, string>
        {
            Key = Guid.NewGuid().ToString(),
            Value = payload
        });
    }
}

Consumer - the listening loop

public class InventoryEventConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _consumer.Subscribe("order-events");

        while (!stoppingToken.IsCancellationRequested)
        {
            var result = _consumer.Consume(stoppingToken); // BLOCKS until a message arrives

            try
            {
                var orderEvent = JsonSerializer.Deserialize<OrderPlacedEvent>(result.Message.Value);
                await ProcessEventAsync(orderEvent); // idempotent logic, as we discussed with the Saga/Outbox

                _consumer.Commit(result); // confirms processing - the "offset" is moved forward
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing the message");
                // NO Commit() - the same message will be delivered AGAIN
            }
        }
    }
}

Offset - a "bookmark" saying where the consumer finished reading

Partition 0: [msg1, msg2, msg3, msg4, msg5]
                              ↑
                    Consumer offset = 3 (it processed msg1, msg2, msg3)

Kafka remembers what offset (position) a given consumer group has reached in each partition - if the consumer crashes and comes back, it continues from the remembered offset, not from the beginning. This is the mechanism that gives "at-least-once delivery" (exactly the same guarantee we discussed with the Outbox Pattern) - if the consumer processed a message but crashed before committing (Commit) the new offset, the same message will be delivered again after the restart.

Retention - why Kafka is more than just a "queue"

Konfiguracja topicu: retention.ms = 604800000 (7 dni)

Unlike RabbitMQ (where a message is deleted after it is received by the consumer), Kafka retains messages for a configured period regardless of whether they were consumed. This makes possible: - Replay - a new consumer (or a new consumer group) can "replay" the history of events from the beginning - Many independent consumers reading the same data at different paces

Kafka vs a traditional message broker (RabbitMQ) - which one when

Kafka RabbitMQ
Model A distributed log (time-based retention) A queue (deleted after being received)
Throughput Very high (millions of messages/s) High, but lower than Kafka
History replay ✅ Yes (data kept for the retention period) ❌ No (the message is deleted after being received)
Routing complexity Simpler (topic + partitions) Richer (exchanges, routing keys, bindings)
Typical use Event streaming, event sourcing, large volumes of logs/events Classic task queuing, RPC between services

Summary - the relationship to the Saga/Outbox

Kafka is the natural technology for implementing IEventBus in a Choreography Saga architecture - partitioning by key (e.g. orderId) guarantees the ordering of events concerning the same order, consumer groups provide load balancing between multiple instances of the same service, and the mechanism of offsets + no automatic deletion of messages supports exactly the kind of delivery guarantee ("at-least-once") we required from the Outbox Pattern - with the necessity that every consumer be idempotent, because the same message may be delivered more than once.

Buy me a coffee