The Outbox Pattern - guaranteeing consistency between the database and event publishing
The problem Outbox solves - the "dual write problem"
This is a direct follow-up to the Saga (Choreography) we have just discussed - for a Saga to work, services have to publish events after saving changes to their database. But what if the database write succeeds and publishing the event fails (e.g. the message broker is temporarily unavailable)?
// ❌ DANGEROUS - two independent systems, two independent points of failure
public async Task PlaceOrderAsync(Order order)
{
_context.Orders.Add(order);
await _context.SaveChangesAsync(); // ✅ zapisano do bazy danych
// What if the SERVER CRASHES right HERE, between these two lines?
await _eventBus.PublishAsync(new OrderPlacedEvent(order.Id)); // ❌ zdarzenie NIGDY nie zostanie opublikowane!
}
This is the dual write problem - you write to two different systems (a database + a message broker) which don't have a shared transaction. You can't wrap it in BeginTransactionAsync(), because the message broker (Kafka, RabbitMQ) doesn't participate in SQL Server/PostgreSQL transactions.
The consequence in the context of a Saga: if OrderPlacedEvent is never published, the whole Saga stops - InventoryService will never reserve the goods, even though the order exists in the OrderService database. This is an inconsistency between the actual state and what the rest of the system "knows".
The solution - write the event TO THE SAME database, in the same transaction
The Outbox Pattern consists of this: instead of publishing the event directly to the broker, save it as a row in an "outbox" table in THE SAME database, in the same transaction as your actual business data.
public class OutboxMessage
{
public Guid Id { get; set; }
public string EventType { get; set; }
public string Payload { get; set; } // JSON containing the event data
public DateTime CreatedAt { get; set; }
public DateTime? ProcessedAt { get; set; } // null = jeszcze nieopublikowane
}
// ✅ BEZPIECZNE - JEDNA transakcja SQL obejmuje OBA zapisy
public async Task PlaceOrderAsync(Order order)
{
_context.Orders.Add(order);
_context.OutboxMessages.Add(new OutboxMessage
{
Id = Guid.NewGuid(),
EventType = nameof(OrderPlacedEvent),
Payload = JsonSerializer.Serialize(new OrderPlacedEvent(order.Id, order.Items)),
CreatedAt = DateTime.UtcNow
});
await _context.SaveChangesAsync(); // ONE transaction - EITHER both changes are saved, OR neither
}
Now SaveChangesAsync() (as we discussed with EF Core transactions) automatically wraps both writes (Order + OutboxMessage) in a single SQL transaction - it is impossible to save an order without saving its corresponding row in the outbox (and vice versa).
Step 2 - a separate process publishing events from the outbox
This is the second part of the pattern - a background worker (exactly the BackgroundService we talked about with IServiceScopeFactory) which regularly checks the outbox table and publishes the messages that haven't been sent yet:
public class OutboxPublisherService : BackgroundService
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly IEventBus _eventBus;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
using var scope = _scopeFactory.CreateScope(); // A Scoped DbContext, as we discussed earlier
var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();
var pendingMessages = await context.OutboxMessages
.Where(m => m.ProcessedAt == null)
.OrderBy(m => m.CreatedAt)
.Take(20)
.ToListAsync(stoppingToken);
foreach (var message in pendingMessages)
{
try
{
await _eventBus.PublishAsync(message.EventType, message.Payload);
message.ProcessedAt = DateTime.UtcNow; // mark as sent
}
catch (Exception)
{
// DON'T mark as processed - the next iteration of the loop will try AGAIN
break;
}
}
await context.SaveChangesAsync(stoppingToken);
await Task.Delay(TimeSpan.FromSeconds(2), stoppingToken);
}
}
}
The key guarantee: if publishing to the broker fails, the row stays in the outbox table with ProcessedAt = null - the worker's next iteration will try again. Nothing gets "lost", even if the whole application process crashes in the meantime - after a restart the worker simply continues from where the outbox table points to unsent messages.
"At-least-once delivery" - a consequence you have to handle
Outbox guarantees that the event will be published at least once - but it may be published more than once (e.g. the worker published the event, but crashed before saving ProcessedAt, so the next iteration publishes it again).
This is a direct link to the idempotency I mentioned with the Saga - the event consumer (e.g. InventoryService from the previous example) must be resilient to duplicates:
public async Task Handle(OrderPlacedEvent orderPlaced)
{
// Idempotency - check whether THIS SPECIFIC event has already been processed
var alreadyProcessed = await _context.ProcessedEvents.AnyAsync(e => e.EventId == orderPlaced.EventId);
if (alreadyProcessed) return;
// ... the actual logic ...
_context.ProcessedEvents.Add(new ProcessedEvent { EventId = orderPlaced.EventId });
await _context.SaveChangesAsync();
}
An alternative - Change Data Capture (CDC), a more advanced version
In some architectures, instead of a dedicated worker polling the outbox table (polling), CDC is used (e.g. Debezium with Kafka Connect) - a tool which listens directly to the database's transaction log and automatically publishes changes to Kafka, without the need to write your own BackgroundService. This solves the same problem (dual write), but moves the responsibility for "publishing" to the infrastructure, not to the application code.
Summary - the Outbox ↔ Saga relationship
The Outbox Pattern solves a fundamental engineering problem underlying every Choreography-style Saga implementation: how to atomically save a state change and publish the corresponding event, when these two operations concern different systems (a database vs a message broker). The solution: save the event as a row in a table in the same database, in the same transaction as the business data (
SaveChangesAsync()guarantees atomicity), and a separate background process (BackgroundService) asynchronously publishes these events to the broker, with a retry on failure. This is one of the fundamental "building blocks" needed for a Choreography Saga to work safely in practice.