Building your own Message-Driven Framework — Foundation

Building Message-Driven Framework — Foundation

Most of the materials will say not to build your own Messaging Framework, yet somehow we’ve a lot of them, better or worse, internal or open sourced. Great amount of them starts as a tiny integration with RabbitMQ, Kafka or SQS, and after awhile starts to grow.

It starts to grow because it has to, if we want to have reliable architecture then Message Broker by it’s own is not enough. We need a framework within the language that will support Message based communication.
In this article we will dive into how to build such Framework, as I will be revealing the concepts that have been introduced behind Ecotone Framework coming from Enterprise Integration Patterns book.

Communicating with Message Broker directly

We may decide to go with direct integration with Message Broker, as consuming and sending messages is not big deal, right?
It mostly works fine till the moment it lands on the production. Now different scale, long running processes and scenarios that we have not thought before, start to happen.

After some time on production we will reveal that there is more things, that we need to cover besides consuming and sending Messages, like:

1. Handling infrastructure failures — Message Consumer will be now running for hours or days, so we will need to introduce reconnecting strategies, when connection will be broken. Besides we may need to handle cases when connection to Broker goes into zombie mode and just hangs.
2. Receiving duplicated Messages — We will have to deal with double consumption of the Message, either when handling Message simply failed or was failed be acknowledged.
3. Handling transient and unrecoverable errors — We will need to handle application level issues with grace (networks issues, concurrency, simply bugs in the code) to recover from them to avoid manual fixing or interventions (Instant and delayed retries, DLQ).

There is of course more, like maintaining integration code, bugs that can be only solved by battle testing for longer period of time and great deal of knowledge which we will need to acquire.

So if we went path of direct integration with Message Broker then we will have to fix production errors and add missing features. This is mostly the point where we unconsciously start building our own Messaging Framework. However building Messaging Framework should start as conscious decision, not as a side effect of bug fixing and production errors, as only then we can invest time in proper abstraction.

Proper Messaging Abstraction

So considering we made a conscious decision about building Messaging Framework, this means we can invest time into building and learning new things. What is most important is that good Messaging Framework begins in the language. While building Ecotone I have not touched integration with Message Broker for the first year, as I was fully focusing on the foundation.

The foundation lies in the Messaging patterns. Luckily patterns have been already described and battle tested via implementation in various languages and frameworks like Spring Integration in Java, Ecotone Framework in PHP and NServiceBus in C#.
Messaging patterns I will be mentioning here are part of Enterprise Integration Patterns book.

Enterprise Integration Patterns

EIP book provides as with Domain Model for Message based Systems, which introduces patterns for decoupled communication. The foundation patterns on which everything is built are Messages and Message Channels.

Message is data record, which contains of payload and headers. Payload can be anything, it can XML, JSON or PHP class, headers are key-value metadata.

Often abstractions put equal sign between Message Payload and Message, which means it lack of possibility to pass Message Headers.
This as a result makes it hard to pass anything extra. It complicates the messaging framework and application level code, as now we need to carry meta-data information like message id, timestamp, executor id within the message’s payload and possibly introduce framework level interfaces into application level code.

Message Channel is abstraction for communicating between Message Endpoints (Message Handlers) using Messages. In Message based architecture there is no direct reference between components as communication goes via Channels.

Shipping Service consumes Event Message from “asynchronous_messages” Queue

Message Channels can be synchronous and asynchronous depending on the implementation, yet the underlying abstraction for them stay the same. We send Message to a channel and other party consumes it on the other end.

This is the core and foundation model for Message based communication. Wrong abstractions on this fundamental level, creates a lot of extra complexity, so it’s important to avoid this. So before we will go further into EIP abstraction, let’s discuss one most common abstraction that have deviated from it - Message Bus.

Message Bus Abstraction

In EIP we will find Message Bus pattern, yet it’s not the same as common implementation of Message Bus, which is not based on Message Channel communication.
So the common Message Bus implementation works like below:

Connecting Message Endpoints

Message Bus is actually a trigger for the Message flow and in concept of EIP it’s called Messaging Gateway. It takes the input, prepares the Message and sends it to the Message Channel. Yet in Message Bus abstraction we don’t have channels, so we trigger Message Endpoints right away.

Each Message Endpoint is configured via Message Bus to route the Message properly. As the routing exists on the Message Bus level, it’s required to use the Message Bus to push the Message forward. This put burden on application level code as now we need to use Message Bus in each of the Message Endpoints in order to push the flow forward.

In case of Message Flows combined from multiple Message Endpoints, we will need to retrigger Message Bus in each of them to push the flow forward.

Asynchronous Handling

Message Bus has its roots in synchronous communication, where each Message Endpoint is executed directly. There is no core abstraction that asynchronous processing is part of.

To make asynchronous possible, whenever Message is sent via Message Bus, it checks if given Message is defined as asynchronous, and if so pushes the Message to given transport (Message Broker).
When Message is consumed from the transport it executes Message Bus again with information “I am in asynchronous mode, let’s handle the Message”. The other known solution is introducing Asynchronous Message Bus, which always sends to asynchronous transport and when Message is consumed, executes Synchronous Message Bus.

Message Bus treats asynchronous processing as something external, an feature that have to be built on top of the abstraction, not as part of it.

Message Endpoint Isolation

After Message is consumed from given transport, it executes all the related Message Endpoints. This leaves no space for isolation. If one of the Handlers fails, then when Message is retried it will retry succesfully Endpoints too.

To achieve failure isolation with Message Bus, it’s often required to create multiple transports and introduce complexity in infrastructure layer.

Message Channel Abstraction

Enterprise Integration Pattern’s abstraction is based on communication using Message Channels:

Connecting Message Endpoints

This abstraction connects Message Endpoints using Message Channels without introducing mediator in between like it’s with Message Bus.
To target specific Message Endpoint, we are targeting Message Channel to which Endpoint is connected.

To kick off the flow we are using Messaging Gateway, just like with Message Bus, yet it can be triggered once for given flow, as the Message will flow using connected Channels.

We can actually pass the same Message through the flow, because the Message itself is not an routing, the channel is. This creates flexibility of connecting, intercepting and modifying the flows without affecting application level code much.

Asynchronous Handling

This abstraction fits naturally with asynchronous communication.
Message Channels can be asynchronous and synchronous, therefore in order to make communication asynchronous, it’s matter of changing the implementation of the channel.

This means that Messaging Gateway send the Message to asynchronous channel from which the Message is consumed and given Message Endpoint executed.

Yet this requires having Asynchronous Message Channel per Message Endpoint, which may not be ideal when the volume of Message Endpoints is large. However as adding and modifying flows in Message Channel abstraction is easy, this can be solved by using routing slip:

With routing slip we can make use of single Asynchronous Message Channel which we will be sending Messages to, yet within the Message we provide routing slip header. It contains of the channel name to which it should be routed after being consumed from the Asynchronous Channel.

With Message Channel abstraction we pass the Message directly to Message Endpoint after consuming it from Message Broker.

Message Endpoint Isolation

With Message Channel abstraction we can achieve isolation on the architecture level.

In case of asynchronous processing, as we send Message to each of the channels, the flows becomes naturally isolated. This means that in case of failure only given Message Endpoint that failed will be retried.

With routing slip solution where there is only one Asynchronous Message Channel in play, the solution works pretty much the same. We would be sending two copies of the Message, each with it’s own routing header that would target specific Message Endpoint:

The isolation of Message Endpoints becomes natural part of the development when using Message Channel abstraction. Therefore retries of failed Messages becomes safe, as it does not produce unexpected side effects.

Summary

The cost of wrong abstraction is huge, it’s either becoming problem for the framework or for the end users. Some wrong abstractions are so common, that we may start to think it has to work this way, and accept the limitations and workarounds as a normal part of development. Yet good abstractions does not need workarounds, they create environment where things just work and “click” together.

There will be two more follow up articles, on which I will be touching subjects like:
1. Messaging architecture in practice — In this article we will focus on how those patterns works in practice, using real life code examples.
2. Messaging architecture optimizations — We will see where Messaging Framework can be optimized to make it work smooth and fast.

If you want to explore more about Messaging Architecture, check my previous article on YOLO Message-Driven architecture.