Making your Application stable with Outbox Pattern

Whenever we send asynchronous messages and modify state in database within same action, we put ourselves at risk. The risk come from the possibility that state will be persisted and message will fail or vice versa.

When we send asynchronous messages to external broker like RabbitMQ or SQS this may fail, when we store something in the database this may fail. Operations over network are not 100% reliable, and in order to keep our applications stable, we need to build software that considers that.

So prevent backfire when we need to send asynchronous messages and modify the state, we may implement Outbox pattern, which ensures that no message or state will be lost.

Examples in the article will be using Ecotone, PHP Framework.

Implementing Outbox Pattern

Let’s consider below example:

In the example we store Order in database, however we may fail to send our OrderWasPlaced Event Message.

General idea of Outbox Pattern is to be sure that no messages or state is lost along the way, if this is achieved then we’ve a success.
So how do we achieve this? Instead of having two different storages in same action, we need to have one.

In our example above we could achieve that, by replacing EventBus implementation with something that will instead of sending to Message Broker, will store in the database. Then we would wrap whole Command Handler in transaction. This ensures that no state or messages are lost, as messages would be persisted together with state.
The final step is to fetch messages from database and send them to Message Broker, after that we can discard message. Then message consumer will pick it up and handle.

Most of the Message Brokers are at-least once delivery. This means that you are guaranteed to receive the message at least once, however you may actually receive it two or more times. This may happen because consumer will fail to acknowledge the message to the Broker, yet message was handled successful.

That’s why it’s worth to implement Message Deduplication. So even, if we receive the message for second time, we will know that it was successful and then we can skip it.
Ecotone provides message deduplication by default.

Consuming Message from Database

The solution that we are aiming at, is to not lose messages or state, and there are more ways to achieve that.
The previous example is most common implementation of Outbox Pattern, however we can achieve the same differently.

In above example we made Event Handler asynchronous. If the transport channel will be database transport, then we will send related Messages to the same database in which we store the state. So instead of fetching and sending them to broker, we can consume and handle them directly without any Message Broker involved.

Using Message Channels and consuming messages directly from Database does not require custom implementation for Outbox pattern. As database channel is just another implementation, that we can use of.

What we need to be sure of, is that sending the message and storing the state are wrapped in same database transaction.

Scaling with Outbox pattern

It’s possible to scale your database channel consumers, like any other consumers. If you want to keep the order and still scale, you may use different asynchronous channels to split the workload and then running single consumer per channel.

However you may decide to not scale your database consumers, as they put more workload on your database, especially when there is no auto-scaling possible. In that case we can make use of the database channel just for the Outbox pattern to pass through message to different channel that will be responsible for handling it:

In Ecotone you may pass Messages from one channel to another.
In above example, we defined Event Handler which will receive Message first on Database Channel, and after fetching it from there it will passed to RabbitMQ Channel.
This way database consumers are only responsible for passing Message to next channels, and actual work is done RabbitMQ Consumers.

In Ecotone you may pass message through multiple channels. This allow for example to move messages to consumers that are easily scalable and use other channels for keeping consistency (outbox pattern).

Summary

Whatever solution you choose to use, the most important is to store messages and the state within same transaction. You may actually use outbox pattern for critical parts of your system and in places where losing message is acceptable or publishing messages is the only operation, you can publish messages directly to RabbitMQ channels.

Ecotone aims for building PHP applications that are stable and solid, that’s why Outbox pattern and Message deduplication is available out of the box.