Message Processing in PHP — Symfony Messenger, Laravel Queues and Ecotone

Message Processing in PHP — Symfony Messenger, Laravel Queues and Ecotone

Message processing becoming more and more popular in PHP.
Putting all the logic inside simple Request — Response model becomes not enough for our current system needs.
Our applications do more and more. It becomes standard to send an email after registration, call external Services or perform some intensive tasks.

Messaging platforms (RabbitMQ, Kafka, SQS, etc) helps in solving those problems by increasing the amount of load system can take, introducing background processing and handling failure retries.

Symfony Messenger, Laravel Queues and Ecotone Framework provides higher level abstraction for message processing that hides integration with given messaging platforms and introducing additional features.

Differences between Task Queues and Message Brokers

There are different types of platforms that may be used to solve asynchronous processing in PHP, some of them are Task Queues and other are Message Brokers.

Task Queues like Beanstalkd provides abstraction over Task (Message).
Task or Job is describing what needs to be done and asynchronous Worker process handles it.
Task Queues provides full lifecycle support, so we can track status of given Task (ready to process, being processed, completed).
They provide Point to Point Channels, which means we send Task (Message) and it will be delivered to exactly one Worker (Consumer).

Task Queues are helpful with offloading the system workload

On other hand Message Brokers like RabbitMQ provides more flexible way of working with messages.
It allows for Message Routing based on different criteria or delivering copy of a message to multiple Consumers (Workers).
For Message Brokers, Message is just an information that is passed from one place to another. The application makes actual meaning out of it.

Thanks to routing capabilities, Message Brokers can deliver the Message in Point To Point fashion or by Publish And Subscribe.
In Publish and Subscribe, Message is published and any party that is interested can subscribe to it. Thanks to that Publisher can be fully decoupled from the subscriber (consumer).

Message Brokers can offload your workload, but also are great for building message flows and cross service communication, as by it nature they handle message distribution and decoupled communication.

Symfony Messenger, Laravel Queues and Ecotone provide own abstraction to work with sending and handling messages.
This means, that they treat platforms as transport layers providing enough integration to cover their own functionality.
This helps in providing similar experience over different platforms and lowers the entrypoint by hiding the complexity.

It’s worth to know what functionalities your framework provides.
As long as it will be covering your use case, you will not need to deal with Message Broker or Task Queue integration directly.

What is Messaging?

Description bellow comes from great book Enterprise Integration Patterns, which describes usage and implementation of messaging patterns.

Messaging or Messaging Architecture is a way to implement communication using Messages.

Just like a letter, Message contains of Payload and Headers.
The payload contains of data that we want to transfer.
Headers on other hand are used to route the message and enrich it with extra information, like identifier of the message, who was the executor or when it occurred.

Messages are sent and received from Channels using Endpoints.
Channels can be Point-to-point where only one consumer can receive message sent to the channel, or Publish-subscribe which broadcast the message to all subscribing endpoints.
Endpoints are places where we process, transform or filter out the Messages, this is the place which we mostly call Message Handlers.

In general, an application should not be aware that it is using Messaging.
Most of the application’s code should be written without messaging in mind.
This frees the application from integration concerns and to let it solves business problems.

So how can Message Handler (Endpoint) stay unaware of Messaging and still receive and send Messages?
It’s wrapped by Consumer, which depending on the need can be Event-Driven (executed synchronously) or Polling (asynchronous, polling the messages).
To send messages to the channel, we are using Messaging Gateway. This is abstraction, that hides complexity of constructing Messages and connecting to external platforms.

High Level Framework Abstraction

Let’s take a look on the abstraction that each of the Framework provides. I will be following Messaging terms, however I will map those terms to implementation details of each of the frameworks.

Symfony Messenger

Messenger introduces messaging solution in Symfony Framework.
It provides a message bus with the ability to send messages and then handle them immediately in your application or send them through transports (e.g. queues) to be handled later.

Symfony Messenger has more similarities with Message Broker than with Task Queue.

What is Message?

Message (called Envelope) is a class, which contains of Payload (called Message) and Headers (called Stamps).

In Symfony Messenger Payload must be PHP Class.

As long as you don’t want to extend basic functionality, you will not need to work with Symfony implementation of Message directly.

How Messages are send to the Message Handler?

The Message Gateway is called Bus.

$bus->dispatch(new PlaceOrder('milk'));

Dispatch method takes class and based on class name looks for Message Handler that can handle that.

Symfony Messenger does not provide Point to Point channels by default, as you may subscribe for single Message from multiple Message Handlers.
However Messenger allows for defining own Buses that can be set up to achieve that.

How messages are routed by routing keys?

There is no possibility to route Message using routing keys.
This comes from the design, as Message expects an actual class as payload, and based on the given class we know where it’s supposed to be routed.

How the behaviour can be enriched?

Symfony Messenger comes with implementation of Middlewares.
Middlewares are connected to given bus and intercept message flow providing a possibility to add extra behaviour or changing the Message.

How Messages are handled synchronously?

Whenever we send Message using Message Gateway (Bus), it’s handled synchronously by default.

How Messages are handled asynchronously?

When we are sending Message using Message Gateway, Symfony adds special middleware SendMessageMiddleware.
This Middleware stop the flow and sends the Message to external Platform (Database, RabbitMQ, etc).
Then there is a Consumer that consumes the Message and run Message Gateway once more, however this time with ReceivedStamp which tell the Middleware to let the flow continue.

How to pass Message Headers?

To construct Message with Headers we need to provide class for each header, that implement StampInterface.

You may access headers in Middleware, however there is no possibility to access headers inside Message Handler at this moment.

How to integrate different Applications?

There is not out of the box solution for this.
However you may share the configuration between two applications, where one application will publish the Message and the second will be running Consumer.

Payload in Messenger is a Class, so it requires this class to be available on both ends in order to deserialize it correctly.

Ecotone Framework

Ecotone from the ground is built around messaging concepts.
It provides implementation of Enterprise Integration Patterns and provides easy to work API that hides messaging details from application code.
Ecotone can be used with Symfony, Laravel or no extra framework at all.

Ecotone has more similarities with Message Broker than with Task Queue.

What is Message?

Message is a class, which contains of Payload and Headers.

In Ecotone Payload can be anything PHP class or array, XML, JSON, simple string.

Ecotone follows principle that you should not be forced to use framework specific classes. Probably, you will never need to work with Ecotone’s implementation of Message directly.

How Messages are send to the Message Handler?

Ecotone provides Message Gateway implementation. By default delivers 3 implementations: CommandBus, EventBus and QueryBus.

$commandBus->send(new PlaceOrder('milk'));

Send method takes class and based on class name looks for Message Handler that can handle that.

CommandBus and QueryBus are backed by Point to Point channels, which means that they aim single Message Handler.
EventBus on other hand provides publish subscribe solution, where multiple handlers can subscribe to one given Message.

How messages are routed by routing keys?

You may send your Message using routing.

Your Command Handler will be connected to given Bus using routing key provided in Attribute.

Ecotone will take Payload of the Message and based on type hint of Message Handler deserialize it to given class.
You can also invoke Command Handler without any parameters just by using routing.

How the behaviour can be enriched?

Ecotone provides concept of Interceptors.
It allows to enrich the behaviour for given Message Handler or group of them, at chosen moment of time.
You may intercept Message Flow before/after Handler execution or before Message is sent to asynchronous channel.

You may also define your own Attributes to annotate your Message Handlers then you may target them for intercepting.

How Messages are handled synchronously?

Whenever we send Message using Message Gateway (Bus), it’s handled synchronously by default.

How Messages are handled asynchronously?

Ecotone allows replacing default synchronous channel that Message Handler is connected to with asynchronous one.

Message Handler decides on it’s own, if it want to handle given Message in synchronous or asynchronous manner.

Instead of marking whole message to be handled asynchronously, Ecotone allows for marking specific Message Handler to run asynchronously.
This makes difference when there is more than one Message Handler subscribing to given Message, as it allows to decide which parts of the system should run asynchronously and which not.

Each Message Handler receives it’s own copy of Message. This is game changer when we are running them asynchronously, as each Handler is handling his own message in isolation, so one failing Handler will not affect another one.

How to pass Message Headers?

Ecotone provides possibility to inject Message Headers into your Message Handler and your Interceptors.

How to integrate different Applications?

Ecotone provides possibility to integrate different applications together using RabbitMQ integration.

You may send Commands to achieve Point to Point integration by targeting given Application.
Besides that you may publish Events to achieve Publish and Subscribe, where multiple applications can subscribe to given Message.

As Ecotone works with Symfony and Laravel and No Framework at all, you can use it to integrate any PHP Application.

Laravel Queues

Laravel Queues allows you to easily create queued jobs that may be processed in the background to offload from your web requests intensive tasks.

Laravel Queues has more similarities with Task Queue than with Message Broker.

What is Message?

Laravel Queues introduces Job as a Message.
Message is a PHP Class, that is also a Message Handler.
Concept of headers does not exists.

How Messages are send to the Message Handler?

Message is a Message Handler, so it executes itself in order do the work.

PlaceOrder::dispatch($order);

How messages are routed by routing keys?

The design expects having access to given class in order to execute it, so there is no concept of routing by keys.

How the behaviour can be enriched?

Laravel Queues provides possibility to register your Middlewares inside your Message.
Middleware must conform to given interface.

How Messages are handled synchronously?

To dispatch in synchronous manner make use of dispatchSync method.

PlaceOrder::dispatchSync($order);

How Messages are handled asynchronously?

Messages are handled asynchronously by default

You may omit onQueue method, if you want to use default queue.

How to pass Message Headers?

There is no concept of Message Headers.

How to integrate different Applications?

Laravel Queues is a Task Queue like implementation, it was not designed for cross application communication.

Summary

Symfony Messenger and Ecotone are built towards general message handling, that can be used to decouple the system, build message flows and offload the work.
Laravel Queues are mostly about offloading the work.

The most important is that each of the frameworks promotes working with messages and asynchronous processing. This pushes our PHP community forward :)

Learn more about Symfony Messenger here.
Learn more about Ecotone Framework here.
Learn more about Laravel Queues here.