Building Reactive — Message Driven Systems in PHP

Building Reactive - Message Driven Systems in PHP

I believe applications in 2023 and beyond should be able to self-heal, isolate failures so they don’t cascade on other components, and provide us with help to get back on track when unrecoverable error happens.
They should help the developer on the design level when adding new features without breaking old ones, and be easily tested and maintainable in the long term.
Besides that, they should scale, not only through the infrastructure it runs on, but also through the application’s architecture and design.
The Reactive Manifesto provides guidelines to achieve this.

The goal of this article is to show not only why, but how, we can build applications in PHP that are resilient, scalable and amenable to change.
This article applies to business oriented applications, where business logic, processes and workflows can be found.
It’s a summary of my experience gathered by years of working in business oriented software, and while building the Ecotone messaging framework.

Reactive Systems

Reactive systems solve everyday problems before they manifest themselves. They focus on making explicit what may go wrong, and preparing the system to deal with these issues. They are flexible, loosely-coupled and scalable.

Systems that are Responsive, Resilient, Elastic and Message Driven, are called Reactive Systems.
The Reactive Manifesto describes each of these four principles.

Word “reactive” may be understood as Reactive Programming, this is not the same as Reactive Systems. Reactive Systems enforce design principles when building and integrating applications, reactive programming, on the other hand, is a programming paradigm. If you want to know more, you may read article by Jonas Bonér, the Reactive Manifesto’s author.

Let’s dive into each of the principles.

• Responsive is being able to respond in a timely manner and provide the users with a good experience, which in result encourage them to keep on using the application. This can be achieved on numerous ways, to name a few: dedicated read models, caching, scaling etc.
• Elastic is being able to scale the system to meet its needs.
  When the load rises, and responsiveness drops, we scale more web servers or consumers. On the other hand, when the load drops, we want to scale down to free the resources and decrease generated costs.
• Resilient is being prepared for failures. When we rely on anything external to our application — and this can even be another application over a local network — it may fail. Being resilient means understanding that failures will happen, anticipate this and implement solutions into the application’s design that will help us recover.
• Message Driven means that components, modules, services communicate via messages, not direct calls. Messages are pieces of data that can be transferred over the network. They are identifiable and carry an intention. When a Message is sent, other components may consume it at some point in the future. Yet, there is no direct dependency between components and they don’t rely on each other’s availability. By avoiding direct dependency between components, they become naturally isolated and remain unaffected by failures of one and the other.

In this article we will focus mainly on two principles: Message-Driven and Resilient. However as the four principles are deeply interconnect, focusing on those two we will, in the end, be touching on all of them.
For example, Message-Driven systems make it possible to scale the consumers and throttle the messages. This way, the systems can be scaled and remain responsive.
In Resilient Systems, failures are isolated, recovery can be automated and responsiveness can be preserved.

Building Reactive Systems in PHP

We went through a bit of theory, which was needed to define the article’s goal. We now know that Reactive System principles help in building scalable, extensible and resilient software.

Now as we understand the theory, time to get our hands dirty and refactor a PHP implementation that does not make use of a Message-Driven architecture.

Suppose we run an e-commerce business. Customers come to our website to buy home accessories and furniture. We are so popular that we sell our products in multiple countries.
Our core business grows around providing fast product matching, placing orders and providing great promotions and discounts which encourage customers to visit our website.
In order to deliver products to customers we integrate with an external Shipping Service

We will focus the order flow:

• Storing Order in database
• Sending confirmation e-mail to the customer with the order’s summary
• Starting delivery process by calling a Shipping Service over HTTP API

Phase 1 — Understand the problem

Let’s consider following implementation for the above scenario:

There is a lot going on to place an order.
We deal with data (storing order) and with side effects (sending email and calling the Shipping Service).
If one of the components fails (for instance: sending the email) it may affect the other parts (shipping or storing order).
Besides there is absolutely no way to recover from an error. If something fails, it will probably propagate to the end user, without clear details about what really happened.

If we try to resolve this within the current design, we will start leaking infrastructure code into our business code.
And with every new feature we will try to implement, the problem will come back as a boomerang, as we will not have dealt with its root cause.

There is a different architectural approach where this kind of problems are solved on the design level: Message-Driven systems.

Phase 2 — Making it Message Driven, making it Resilient

One of the characteristics of a Reactive System is being Message-Driven.
It means that instead of direct calls between components, module or services, information transits using Messages.
Message-Driven architecture needs abstraction built on top of the language. The NServiceBus Framework provides implementation of messaging architecture in C#, the Spring Integration foundation project for Spring Cloud introduce messaging in Java, Ecotone Framework implements Message-Driven architecture in PHP

The term “Ecotone”, in ecology means transition area between ecosystems, such as forest and grassland.
The Ecotone Framework functions as transition area between your components, modules and services. It glues things together, yet respects the boundaries of each area.

Ecotone’s Message contains a Payload and a Headers:

Payload can be anything, it can be a json/xml string, object instance, an array etc.
Headers are message’s metadata, contain framework specific information and custom information that is not part of the payload (e.g. timestamp, executor of the message, executor’s roles).

You will not have to interact directly with Ecotone’s Messages, this will be hidden from your daily development. In fact Ecotone does not force you to extend or implement any framework specific classes or interfaces.
Ecotone helps in keeping the business code clean and focused on the business problems, not the infrastructure.

Let’s move forward and define our Command Message and Event Message which will help us to decouple side effects.

We will change our OrderService so it can receive Commands and publish Events.

Sometimes people discard the solution when they see Events and Commands, because all they want is to send an Asynchronous Message. Commands or Events are Messages.
They are actually higher level concepts that make the intention of the Message explicit. In Ecotone they are POPO (Plain Old PHP Objects).
Ecotone takes care of the serialization and the deserialization, of the payload and headers, when they are emitted.

Readability of this class has greatly improved. It becomes naturally oriented towards a single responsibility: Placing an order. Yet what is most important is that we have separated data storage from the side effects. We will see below how side effects are called now.

Ecotone uses a declarative way to configure messaging. This means that you will be writing business code and tag it with attributes to connect it to the messaging system.
By using the #[CommandHandler] attribute we indicate that this method is responsible to handle the PlaceOrder Command. Ecotone infers this from the type of the first typed property.

And this is how we call this Command Handler from the Controller now using a Command Bus:

If you’re using Symfony or Laravel, Ecotone will automatically register Command and Event Bus interfaces in your Dependency Container.

Let’s define the Event Handlers that subscribe to the OrderWasPlaced event and trigger the side effects:

We have defined Event Handlers that subscribe to the OrderWasPlaced event.

Ecotone makes use of concept called Message Channels.
Message Channels are like pipes in which messages transit. In the example above, the channel is called “asynchronous_channel”.

When we use Command or Event Bus, we use a Messaging Gateway.
It takes our data and converts it into Ecotone’s Message. This is how we connect to messaging through simple interface.
When the Command is sent via the Command Bus, it goes through a channel to the Command Handler. The channel can be asynchronous or synchronous.
From there, in our case, we publish Event Message using the Event Bus. The Event will be deliver to each of the subscribing Event Handlers.
As we have defined Event Handlers to be asynchronous, with the
#[Asynchronous(“asynchronous_channel”)] attribute, events will be handled asynchronously.

Depending on your needs, multiple implementations of the Message Channel exist: RabbitMQ, SQS, Redis etc.
Yet, whatever the picked solution is, the business code remains unchanged.

You can read more about about Commands handling in Ecotone’s documentation.
If you want to read more about Events handling read this section Ecotone’s of documentation.

Protecting data and side effects from failures

By making our Event Handlers asynchronous, we have separated side effects from data persistence.
Right now, our Event Handlers will be executed asynchronously, after receiving the Message from a Message Broker. Failure of the summary email will not affect the placing of the order, as the order is already placed and stored in the database.

In resilient applications we have to consider that the side effects will fail sooner or later. By decoupling side effects from data persistence, we prepare our system for this.

Event Handlers need to be isolated as well. If we were to handle them together, failure in one would affect the other. This would result in the same problems as coupling data persistence and side effects.

When we publish an Event, Ecotone delivers a copy of the message to each of the Event Handlers separately. This means, that each Asynchronous Handler handles his own Message in isolation.

You can imagine this solution as a Publish-Subscribe implementation, where each Event Handler is treated as separate subscription.
This provides isolation of each Message Handler and safe retries become possible.

Read more about asynchronous event handlers in Ecotone’s blog post.
Read more about publish-subscribe Event Handlers in Ecotone’s documentation.

Securing your Messages from being lost

We have not yet fully secured placing of the order.
After storing the Order, we are publishing events to an external Message Broker. This means that there is more than one storage engine involved (Database and Message Broker).
What does this mean for us? Well, the delivery of the Event to the Message Broker could fail and, depending on the implementation, this might cause a rollback of the order or its persistence without triggering the side effects. As a result, the customer won’t receive his Order.

The Outbox Pattern is a solution for such use cases and Ecotone implements this using Message Channels.

By default, Ecotone wraps the Command Handlers and all subsequent actions in a transaction. This means that if we use Database based Message Channel, the Messages will be committed along with the Order.

Ecotone wraps the Command Handler and all subsequent actions in a transaction. So all the synchronous Event Handlers are part of the same transaction and changes remain atomic.
It’s especially useful when the application design requires the modification of multiple Entities or when synchronous event projections are required.
Non-atomic actions such as side effects can run asynchronously.

With a ServiceContext, one can add extra configuration to Ecotone.
Above configuration switches Message Channel of name “asynchronous_channel” to make Database Channel.
As this is database channel now, messages will be committed together with data change.

Read more about using Outbox Pattern and scaling the solution in this Ecotone’s blog post.

Handling Message Deduplications

In most of the Message Broker setups, Message can be emitted more than once. This happens, because our asynchronous Message Handler may successfully handle the message but fail to confirm this to the Message Broker.

In case of failure, the Message will be emitted again by the Broker, but we have to prevent handling the Message a second time.

All Ecotone Messages contain a unique MessageId and can therefor be identified and tracked. Based on this, Ecotone provides deduplication mechanism by implementing idempotent consumer.
When message is handled successfully, Ecotone stores the MessageId. This way duplicated messages can be tracked and, when required, discarded.

Deduplication can be customized for the Message Handlers. This may be useful when handling external events (e.g. webhooks) with custom identifiers that we would like to track in order to recognize duplicates

In above example we’ve sent a Command with Metadata. All metadata (e.g. executorId) is accessible in the Message Handlers, via the Header attribute.
If you want to know more about how Handlers are executed read this part of Ecotone’s documentation.

If we were to generate the orderId on the frontend side, we could actually deduplicate orders this way. This ensures that a given order will only be submitted once, even if we get two or more same requests.

Recovering From Synchronous Errors

One of the characteristics of a Reactive System is being Resilient.
Resilience means we’ve prepared our system for failure and that it can gracefully recover without any intervention.

Placing orders uses synchronous calls through a Command Bus. Imagine losing the database connection. The customer will not be able to finalize his order and if there is no automatic retry mechanism in place, the order will be lost.

As we now have isolated the side effects from the Place Order Command Handler, it becomes possible to retry it.
Ecotone lets us setup a retry strategy for retrying synchronous Command Handlers.

With the configuration in the example above, Commands will be retried automatically when the set exceptions are thrown. Only after three failed retries will these exceptions will be thrown outside of Command Bus.

We isolated Command Handler from the side effects and it only stores the Order now. The number of possible errors has been reduced.
The most common problems are database connection issues or optimistic locking exceptions when concurrent access occur. Recovering from these issues becomes trivial.

Recovering from Asynchronous Errors

In case of our Asynchronous Event Handlers, we may retry them too.
However in this case we have a bit more options, how we want to do it.

Instant Retries

The same mechanism as for synchronous Command Handlers can be used for asynchronously handlers. In this case, it works for both Command and Events (remember our Event Handlers are isolated, so we can safely retry them).

Delayed Retries

Sometimes, instant retries will not solve our problem. The Shipping Service could be unavailable for a couple of minutes for instance. In that case we may retry the message with delay.

With Ecotone you can set up delayed retries. If Shipping Service is down, we give it some time to recover and try again.
This mechanism deals with two problems: we unblock the customer as the message is delayed, and we give the system a chance to self-heal.

In this configuration we’ve set up a retry strategy with three attempts. Initial delay is 1 second (1000ms) and will be multiplied by 10 with each attempt. It works out of the box when your Message Channel supports delays (SQS, RabbitMQ, Dbal, Redis).

When the Message cannot be handled and the number of delayed retries is exceeded, the error is unrecoverable and the Message will be moved to the Error Channel.

Everything that goes over network is not reliable and resilient systems are built with this in mind.
Resilient systems, when things go wrong, recover and continue working.
With good foundations, the system self-heals and you don’t need to worry about external service being down for a couple of minutes. Eventually, everything will be fine.

Handling Unrecoverable Errors

After multiple delayed retries, if the problem is not resolved, more retries will probably not change anything.
The error is unrecoverable and needs to be dealt with differently.

Ecotone solves this by implementing Error Channel.
After retries are exhausted the Message is moved to Error Channel.

The Error Channel will work as publish-subscribe (if not defined otherwise). This means as many as needed Event Handlers can be connected in order to provide custom logic, like sending Slack notification or Email, when Message fails.

Ecotone comes out of the box with solutions that connects to Error Channel in order to store Error Messages in a database and a provide set of functionalities to review, replay or delete them.

The Message is moved to Error Channel, as we enabled Dbal Dead Letter (Ecotone’s out of the box solutions), it’s stored in the database.
Then we review what happened, it could be a new bug introduced in our notification template. We then fix the issue, release the change and replay the Message.
The message will return to the original message channel it came from, so it can be re-consumed and handled by Notification Subscriber.

Read more about handling Error Messages in the section of the Ecotone’s documentation.

Handling multiple Applications

The default solution that Dbal Dead Letter provides is to manage Error Message via CLI.

Symfony: bin/console ecotone:deadletter:replay {messageId} 
Laravel: artisan ecotone:deadletter:replay {messageId} 
Ecotone Lite: $messagingSystem->runConsoleCommand("ecotone:deadletter:replay", ["messageId" => $messageId]);

This is not ideal as we need to access production servers to replay the message and, when we have more than one service, we may need to jump between the servers.

Ecotone, provides Ecotone Pulse which is an application that aggregates Error Messages from your different services and provides a UI for reviewing, replaying and deleting them.

Read more about Ecotone Pulse in the Ecotone’s documentation.

Summary

As we’ve seen in the examples above, some small changes were enough to decouple the components from each other by introducing Message-Driven architecture. The system has become much more stable and reliable. This is the power of messaging architecture: it enables us to go beyond limited programming of synchronous calls and build systems that are robust.

It took me over five years to build Ecotone into it’s current shape, and I believe that, right now, it opens new ways to integrate and build applications in PHP and use well known and robust reactive principles.

You really don’t need to start new project from scratch, go through a major rewrite or change your main framework to start building applications following message driven architecture.
Ecotone can be used with Laravel and Symfony and integration is trivial.
In case you don’t use these frameworks, then you can use Ecotone Lite.
This article summarizes how to implement Message-Driven architecture but there is much more information in the Ecotone documentation.

Start small, refactor one functionality at a time, play with it and test it out. Once you feel more confident with this design pattern take on another module. Iterate and with each increment, you’re application will become more message-driven and reliable.

Demo project of the integration is available in this repository.