YOLO Message-Driven Architecture

At the root of Messaging is communication using Messages, where each of the Message is fully identifiable and handled in complete isolation. The promise of investing into this is, resiliency, self-healing capabilities and scalability.
Yet at some point Messaging has deviated from those basic rules into something I call YOLO Message-Driven Architecture.

The rules of YOLO Messaging

The YOLO became a standard way of doing Messaging and is pretty straight forward to follow. To implement YOLO Messaging, we just need to counter the very basics of Messaging and follow below rules:

1. We use Messages only to offload tasks — Instead of communicating between components, modules and services using Messages, we downgrade it usability only to offload some task/job like sending an email. As a follow up we put an equal sign between Messaging and Job Queue. At the end if it’s all about offloading a task, then it has to be the same, right?
2. Messages are not handled in isolation — We publish an Message and execute multiple Subscribers within same Message consumption process. If one of them fails, flows stops, or if we do have retries, we retrigger successful handlers too. At the end who would care that same action get triggered twice or more, if it happens then YOLO, right?
3. If it fails, then it fails, it’s an edge case anyway — We skip supporting patterns like Dead Letter, Retries, Outbox pattern. This as a result make us lose messages or require to manually recover from errors and look through stack of logs to find the cause. If we are lucky and we lost message during the weekend, then we can ignore, as who would care to go through weekend’s log, right?

Of course the above is written as parody, yet this parody is unfortunately more often that not, real. That’s why in my articles, workshops or discussions I put a prefix “resilient” Message-Driven architecture, as I want to differentiate from what is commonly known as Messaging.

The YOLO pillars are standing the ground well and it’s hard demolish them. To do it we need to shake their foundations by returning to the very basics of programming, return to what Object Oriented Programming is about.

Message Oriented Programming

In “Object Oriented Programming” term the first class citizen is “Object”, which clearly states what we should focus on: objects, their structures and data. The naming is clear, so we know what to do, we can finish the article :)

Yet before we do that, it’s worth to consider if “OOP” is actually the thing we think it is. Everyone who ever programmed can share a story of the code base, where the naming was misleading and the behaviour was doing something totally different than the name stated. And unfortunately OOP is no different, and it was described by Alan Kay nicely:

“Object oriented programming, gets people to focus on the lesser idea.
The big idea is Messaging.” — Alan Kay

Alan Kay is one of the godfathers of computer science, who coined the term “Object Oriented Programming”, yet what he meant by that was different from what we mostly consider OOP now. OOP was meant to promote communication via Messages, where objects are consumers and publishers of Messages. Alan Kay stated that using Object in the term was unfortunate, as it shifted the focus in wrong direction.

Knowing that OOP is about Messaging, we can rediscuss our YOLO rules, to throw new light on them.

Resilient Message-Driven Architecture

Let’s redefine what Messaging architecture should provide to us, in context of Objects and Messages:

• Each Message is a data record, which is fully identifiable
• Object expose an API by consuming particular Messages, and may publish new Messages
• Object handle given Message in full isolation and has enough internal information to perform it’s action

When we have those three basic rules, we can reiterate the false assumptions of YOLO Messaging:

“We use Messages only to offload tasks”

One of the advantages is possibility to offload tasks using Messaging, yet this is one of the features, not the only feature it enables. Considering this as the main aim is pushing Messaging on the edges of the architecture. When we want to consider Messaging as part of OOP, then it has to become central way of developing things, not something on the edges.
With Messaging we communicate using Messages between Modules, Services and even Classes, so it’s much more than putting an job in Beanstalkd.

“Messages are not handled in isolation”

One of the keys to enable Resilient Messaging is to be able to let each Message be handled in full isolation. This means that for same Message when we have multiple Handlers, which happens in case of Event Handlers and Event Messages, there should be a way to enable isolation.
We do that in publish-subscribe way, by delivering a copy of the Message to each of the Handlers separately.
This way we guarantee safe retries and that no side effects will be done when retries happens.

“If it fails, then it fails, it’s an edge case anyway”

Failing is an edge case till someone lose the money, then it becomes an business problem. If our architecture does not support resiliency we can’t put trust in it. It becomes hard to recognize when bug in our code base made the data disappear or when it was “Rabbit, being Rabbit”. Actually it’s never RabbitMQ fault that we lose the data, it’s our architecture that does not support resiliency.

There are a lot patterns that support resiliency like Delayed and Instant Retries, Dead Letter Storage, Outbox pattern. Together with message handling isolation, they create a real combo architecture that support resiliency out of the box.

Summary

Some Message Brokers support patterns and solutions that were described along the way. Yet what is important, to push Messaging to the level of OOP, it has to become part of the development tools we use on the daily basics. For this we need to have inbuilt support in our programming languages for working with Messages.
Messaging Frameworks that provide such inbuilt support are C#’s NServiceBus from Udi Dahan, Java’s Spring Integration which is foundation for Netflix’s Spring Cloud and Ecotone for PHP which I am founder of.

Having Messaging as part of our daily development, does not mean we will need to deal with Messaging concepts. We may actually work from higher level code, where we won’t be dealing with low level Messaging API. This way we can join clean business code with powerful Messaging concepts that guarantee resiliency and recoverability, which at the end will help us to get to the roots of what OOP was about.