My Database is not a Message Broker!

Whenever we send Messages to Message Broker and store changes in our database, one of those actions may fail. If this happen we will end up in inconsistent state of the system.

To solve this we may use Outbox Pattern which stores the Messages in the database, which then can be re-published to the Message Broker.
Yet not everyone want to use database as Message Broker, this may be true because:

• Our database does not support transactions
• We want to avoid constantly polling our database
• We send really high amount of Messages and we want to avoid scaling database Message Consumers

Besides the above, the Outbox pattern may be considered unnecessary complexity, or simply developers may prefer to avoid using Database as Message Broker.
I am not going to dive into, if using Database as Message Broker is valid approach, as it depend on your own context. What I will do however is to show you the other ways to increase the consistency, using the patterns I’ve applied to Ecotone Framework.

Sending Retries

This one is easiest to achieve. Whenever we fail to send Message to a Message Broker we will retry the call.

It’s important to provide a delay between each attempt, this way we give Message Broker a chance to become available, so sending Message can be self-healed.

Ecotone provides by default 2 retry attempts, first after 10ms and second after 400ms, this is configurable.

Ghost Buster

Let’s consider scenario with Customer registration:

Suppose that whole register method is under database transaction.
And after storing Customer and publishing “Customer Registered” Event Message (to a Message Broker) we fail on storing Address. This as a result will rollback database transaction, however we already sent the Event Message to a Broker.

Messages that have been sent where related data has been rolled back are Ghost Messages. When Message will be consumed there will be nothing to reference too, which will make your Message Handlers fail.

To solve this we should delay sending asynchronous Messages after Message Handler (RegistrationService) have finished processing.
In case of Ecotone sending asynchronous Messages happens just before the transaction is committed or simply after Command Handler have finished execution. This way in case Message Handler fails for any reason, Messages won’t be sent to the Message Broker.

Error Channel

Let’s consider fatal scenario, our Message Broker becomes unavailable completely. This means that retries won’t really help us to recover. To recover from this situation, we need to send the Message to other place, where we can store it and replay later.

With Ecotone we may set up so called Error Channel, where we define how we will handle given Error Message, for example we may use inbuilt Dead Letter with Database.
If we choose Database Dead Letter, then if Message will fail during sending, we will store it in database under our current transaction.
This means we will be able to safely commit the transaction and preserve Error Messages which we can later replay using CLI or Ecotone Pulse Dashboard.

Storing Messages that failed on sending in Database Dead Letter help us recover from failure and allows for safe commit of the data without losing any information. After Error Message is stored in the database, we can safely review and replay it.

Yet to make it more reliable we need to consider one more thing…

Message Serialization Failure

Suppose we send three Messages and fourth has broken serialization. We can’t really rollback now as we will end up with Ghost Messages, yet we can’t store Failed Message in Database Dead Letter, because we can’t serialize it. To solve this we need to make step back.

To make sending reliable on architecture level, we will need to split serialization from sending and serialize all Messages before we will start sending them. This way we ensure that from application perspective all Messages are fine and what is left to be done is to send Messages to the Message Broker.

In case of Ecotone whenever we send multiple Messages, they are first serialized and then collected for later sending, this way we ensure that Messages are serializable. If error happens, then no Messages will be sent.

If sending to Message Broker fails it has to be due to integration with a Message Broker, so most likely automatic retries will solve the problem. If not then Message is already serialized, so we can send it to Error Channel and store in Database Dead Letter.

Critical Messages

If we are super unlucky, there is still single scenario where we may fail even so we applied above patterns. This may happen when all messages were sent to the Broker and database transaction commit will fail.

To solve this we would need to introduce Outbox pattern and commit Messages and the data together. Yet we tried to avoid using Database as a Broker, in those scenarios we would need to accept this risk. However it’s worth to consider different approach where we will join those two solutions together.

So in our system we may have critical flows and non-critical flows. Sending an email will non-critical, yet storing an payment will critical.
In case of Ecotone we can steer and apply Outbox pattern to critical flows and the other pattern we described above to non critical flows.
By doing so, we ensure that business critical components will be always consistent, yet we avoid the pitfalls of Outbox pattern in places where a bit more risk is just fine.

Summary

There are multiple patterns which we can apply to increase the assurance of our Data and Messages being consistent. It’s important to build those pattern it into underlying architecture, so the resiliency becomes the default way of working.

Ecotone provides those patterns out of the box, you may read more about them in following chapter about “sending resiliency”.
If you would like to introduce Ecotone with Laravel or Symfony, then Ecotone provides packages for that. If you use your own framework or any other (Magneto, Zend, Laminas etc) then you may use Ecotone Lite, which is standalone solution that can be used in any environment.

Whatever you choose to do, stay resilient and stay safe! :)