Ecotone Enterprise: Kafka, Distributed Bus, Dynamic Channels and more…

Ecotone comes now with new Landing Page at “ecotone.tech” and Enterprise plan which provides more advanced functionalities and features.

On new landing page we will be able to find information on:

• What Ecotone is about, what key features it provides, and how can it be used
• What’s the difference between Free and Enterprise Ecotone
• What development philosophy and architecture is behind Ecotone

Therefore feel free to visit new Ecotone’s page, especially if you have not worked yet with Ecotone to get the feeling on how things are done using Ecotone and how your project can benefit from it. And now, in this article, we will focus on deeper look on new Enterprise features.

Ecotone Enterprise

From now besides Free features, Ecotone will also provide Enterprise ones. Enterprise features are more advanced functionalities which aims to help building larger scale systems, optimize system costs, and to speed up daily development even more.

Ecotone Enterprise is available via subscription. After subscription is started (which can be done at the main page), we will receive licence key, which will grant us with access to new Enterprise features.
There few key Enterprise features, which we will now go through, to see how do they looks like and what we can expect from them.

Kafka Integration

One of the key Enterprise features is integration with Kafka.
Ecotone provides two main ways of integrating using Kafka:

• Custom Message Publishers and Consumers
• Kafka Message Channels

Custom Message Publishers and Consumers

Custom Message Publisher and Consumers are meant for making it possible to integrate Ecotone Applications into existing Kafka infrastructure with ease.
Publisher and Consumers are higher level abstractions, so we don’t need to deal with low level complexity of the Broker.

Let’s take a look, how we would define Message Consumer:

To start subscribing to given topic, we will use KafkaConsumer attribute, which will register new Message Consumer for us:

Above code will register Message Consumer named “shipping-order”, which will consume from topic “orders”. By default it will use endpointId as our Group Id, however that can be customized.
There is no need for any more configuration, we are basically ready to run this Message Consumer (Worker):

# Symfony 
bin/console ecotone:run shipping-order -vvv 
 
# Laravel 
artisan ecotone:run shipping-order -vvv 
 
# Ecotone Lite 
$messagingSystem->run("shipping-order");

Under the hood Ecotone use fast and stable integration with “rdkafka”, and adds ability to customize all rdkafka options for Publishers and Consumer if we wish too. However in most of the scenarios, no extra configuration will be needed, as Ecotone provides sensible defaults.

By default Kafka is not configured for resiliency, things like ensured Message order, delivery guarantee are disabled. Ecotone changes that default configuration, therefore all safety configurations are enabled by default.

Let’s take a look, how we would define Message Publisher:

Message Publisher are configured using Ecotone’s Service Context which returns configuration objects.

By providing referenceName we provide name under which Message Publisher will be registered in our Dependency Container.
This configuration is enough to already be able to inject our Publisher in our Application level code and start sending Messages.

After Publisher is executed, it will convert given Object into serializable format (depending on configuration), and then send it to topic with name “orders”. This way with minimal amount of code, we can build Publishers for different topics.

There are different ways we can send our Messages using Message Publishers. We could send simple string based data and provide Content Type, or like in above example specific Object and let Ecotone do the serialization. Alongside with data, we could also pass Metadata.

Let’s now take a look on Kafka Message Channels.

Kafka Message Channels

Message Channels are one of the main Ecotone’s abstraction for communication. It provides seamless way to communicate asynchronously using Messages, and integrates nicely with higher level features of Ecotone.

If you want to find out about Ecotone’s architecture and how all Messaging parts play well together, read documentation page at “docs.ecotone.tech”.

As our example we will take simple scenario of sending notification after User was registered. On high level it would look like this:

Let’s start by defining our Kafka Message Channel using Service Context:

We could then have some Command Handler which is publishing an User Was Registered Event using Event Bus.

and then we would have Event Handler subscribing to this Event, which would be triggered asynchronously after this Message is consumed from our Kafka Topic:

As we can see this is really simple. We can reuse our Kafka Message Channel for different Asynchronous Message Handlers, and Ecotone will take care of routing to the correct Message Handler. This makes the development using Kafka really smooth and quick for Developers, moving technical the focus on the application level side of code.

Aggregates with Kafka Channels

Suppose we are build ticketing service, and we do have Ticket Aggregate (model/entity), which provides an action “close”. This action is asynchronous, and In Ecotone world could be modeled like this:

Now what will when “async” is Kafka Message Channel, is that Ecotone will use aggregate id instance as partition key for Kafka Topic. This way all Commands that will be send to given Ticket instance, will be handled in order as they will all end up in same partition. This way out of the box we get ordering in our systems.

Ecotone will provide context to each Event Message recorded in Aggregate, about it’s origins — Aggregate Id from which Event originated. This Aggregate Id will be used as partition key for Asynchronous Event Handlers, ensuring that even if system is under high load, there will be no conflicts in processing. Therefore automatic partitioning works for both Async Command Handlers and Event Handlers.

You can read more about Kafka integration in the documentation page.

Dynamic Message Channels

The next big feature of Ecotone Enterprise are Dynamic Message Channels.
Dynamic Message Channels are meant to provide flexible way of changing how messages are routed, how they are consumed and to allow for customization for even more sophisticated business requirements.

Distribution per Client

There may be situations when we would like to introduce Message Channel per Client. This is often an case in Multi-Tenant environments, where some Client would pay extra for additional processing power.
In Ecotone we can keep our code agnostic of Multiple Channels, and yet provide this ability to end users in a simple way.

Taking as an example Order Process:

This code is fully agnostic to the details of Multi-Tenant environment. It does use Message Channel “orders” to process the Command. We can however make the “orders” an Dynamic Channel, which will actually distribute to multiple Channels. To do this we will introduce distribution based on the Metadata from our Command.

Now to actually route a Message we will provide metadata while sending Command:

Of course we can have “standard” Clients which will running under shared Message Channel (Queue), therefore they won’t have their own processing pipeline:

Then we would run Message Consumption (Workers) for each of the channels:

# Symfony 
bin/console ecotone:run tenant_a_channel -vvv 
bin/console ecotone:run tenant_b_channel -vvv 
bin/console ecotone:run shared_channel -vvv 
 
# Laravel 
artisan ecotone:run tenant_a_channel -vvv 
artisan ecotone:run tenant_b_channel -vvv 
artisan ecotone:run shared_channel -vvv 
 
# Ecotone Lite 
$messagingSystem->run("tenant_a_channel"); 
$messagingSystem->run("tenant_b_channel"); 
$messagingSystem->run("shared_channel");

Of course we could also scale up Message Consumption process for each of the Channel separately.

We don’t even need to run shared_channel directly, we could run our Dynamic Message Channel instead. Which would pick up Messages from all related sub-channels in round robin manner:

# Symfony 
bin/console ecotone:run orders -vvv 
 
# Laravel 
artisan ecotone:run orders -vvv 
 
# Ecotone Lite 
$messagingSystem->run("orders");

Dynamic Message Channels can also be used to simplify deployment strategy. We may simply combine multiple channels being used in Application under single Dynamic Message Channel and run it single process that will consume from all of them. This may be especially useful if the volume of Messages in our System is low.

Throttling Strategy

Let’s take as an example of Multi-Tenant environment where each of our Clients has set limit of 5 orders to be processed within 24 hours. This limit is known to the Client and he may buy extra processing unit to increase his daily capacity.

So let’s start by defining our Dynamic Message Channel with throttling strategy:

In here we are using “requestChannelName” this is router to a Internal Handler that will make the decision about the consumption. Internal Handler is fully under our control and should return true/false for given channel name:

Our Internal Handler can call database for example to verify how many orders have been placed for given Tenant (Client). If it reached the limit we would simply return false and skip the consumption from that Channel.
This way we take over the process of consumption at run time, which allows us to business based decisions based on current situation.

Often used solution to skip processing/throttle is to reschedule Messages with a delay and recheck after some time. This solution however will waste resources and block processing of other Messages, as we consume something that is not meant to be handled only to reschedule. Therefore Ecotone’s throttling strategy provides alternative, which skips the consumption completely, so we can avoid wasting resources on polling or rescheduling Messages, as we simply don’t consume them at all.

Dynamic Message Channels does also provides ability to define any customized Sending or Receiving Strategy. This way we can fully take over the process and customize it to our needs.
To find out more, read related documentation page.

Distributed Bus with Service Map

On more big feature which we will mention within this article is Distributed Bus with Service Map. Distributed Bus is meant to solve complexities that cross application integration brings, by making integration easy to follow, change and understand no matter of Developers experience level.

Service Map is a map of integrated Services (Applications), and points to specific Message Channels to which Messages for given Service should be sent:

In this approach Message Channels (Pipes) are simple transport layer, and the routing is done on the Application (Endpoint) level using Service Map to make the decision.

When given Message is sent it will be routed to related Channels. This way Message will land in Channels owned by given Service, and can be consumed from there.

Ecotone provides multiple implementations of Message Channels e.g. RabbitMQ, Kafka, Redis, SQS, Dbal, or even Symfony Messenger Transport or Laravel Queues. This means that all of those can be used for cross-service integration with ease.

In the code configuration for Service Map could look like this:

It’s good practice to share Service Map between Applications. This way it becomes one source of truth on how integration works. Making it easy for everyone to understand the topology of the System.

Command Distribution

Let’s suppose User Service wants to create Ticket by sending Command to Ticket Service.

In Ticket Service we will explicitly state that we allow given Command Handler to be executed in Distributed way. This makes it clear for everyone that we can’t simply delete this Command Handler, as other Services may rely on this integration:

On the side of User Service we would send this Command to Ticket Service:

targetServiceName is the name of the Service which we target, and will be used to find out Channel Name to which we need to send using Service Map, routingKey is the name of Command Handler which should be executed.

By providing target Service Name we explicitly state where this Command should go. This way we secure the integration, as in case targetted Command Handler would not exist, we would still deliver the Command, which would fail on other side, and land in Dead Letter, therefore it gives ability to fix the integration without losing Message.

Event Distribution

Event distribution is a bit different from Command distribution. In case of Command we do have single Service that will receive the Message, in case of Events however there may be multiple of them.

Let’s expand our previous example to include Order Service, and our scenario is that whenever new User is registered in User Service, we will publish this event to both Ticket and Order Services.

On the consumption part we will be marking our Event Handlers with Distributed:

On the publishing side we will use send Message as Event:

By default Event will be published to all Services in the Service Map, with exception of originating Service that publish this Event, this one will be skipped (to avoid publishing to itself). Therefore the default behaviour broadcast the Event to all Services defined in Service Map.

Filtered Event Publishing

The default behaviour speeds up development, as with minimal configuration we can integrate multiple Services. However for large scale volume of Events, we may want to avoid publishing Events to non-interested Services. For this we can use filtered publishing.

In our Service Map the configuration for filtered publishing would could look like this:

Using above Service Map, Event will be published to given Service only if it matches subscription key. Therefore we can avoid publishing Events to non-interested parties.

When Service Map is defined as separate shared library. It becomes explicit what Events is given Service interested in. This also makes the process of subscribing to new Event visible for everyone, therefore we avoid hidden coupling that could lead to broken integration.

There are of course more features around Distributed Bus, which allows us for example to create multiple Service Maps or to introduce multiple Message Channels for targeted Service (e.g. Event or Command Channels).
To find out more, read related documentation page.

Summary

Ecotone Enterprise does provides more advanced set of features, which greatly enhance development process, by providing tools which help to deliver business features quicker, and also to solve product related challenges like optimization of resources, visibility of how things are integrated, and long-term maintainability by keeping things simple.

For first new Users, Ecotone Enterprise comes with 40% discount code for first subscription payment. Go to https://ecotone.tech, proceed with chosen subscription, provide promo code “resilientmessaging”, and discount will be applied.
Offer will be available up to 10th of April with limited number of uses.

Features mentioned in this article are covered on high level, and are not all features provided by Ecotone Enterprise. There to find out to find out more, visit documentation page.