With the ever increasing amount of data that modern applications have to handle, comes the need for new and more sophisticated system architectures.
One of the shiniest gems of recent years are event-driven architectures.
This type of setup has helped companies like Facebook, Google, LinkedIn and other giants to overcome their unique set of problems, but it has also helped small fast growing companies to keep up their momentum and become successful organisations.
So… What’s the deal?
Event-Driven Architectures (or EDA) is a software design style backed by the idea that communication between services must be done by sending and reacting to “events”.
These events are usually defined as changes of state somewhere in the system, which are put out in the open for other services that may be interested parties, rather than communicating them directly.
Take the case of a simple customer order on any online store.
In a request-driven architecture (fancy name for the simple request/response setup), an online purchase may look like the following diagram:
Here we have a web server that submits the user’s order to the orders service, then
- The Orders Service sends a
shipOrder()request to the Shipping Service. - The Shipping Service needs the customer’s information, so it requests it from the Customer Service.
- Then when the information is back, the Shipping Service can finish processing.
This sounds like a logical first set of steps, however these types of services can get really complicated as the system grows, especially when adding more interactions.
We may want to add an order’s analytics component, we may want to have a real time fraud detection component, or a separate inventory management system, we may need to deal with customer’s requests, but that’s not possible until certain information is past a stage, like the shipping service. All sorts of complexities can arise.
In an Event-Driven Architecture, the previous transaction looks like this:
There’s one minimal change in this over-simplistic example, the way in which the Orders Service decides to communicate the creation of an order.
Here, instead of sending a request to the Shipping Service directly, it sends an “Order Created” event basically to anyone who’s listening, notifying them of what just happened.
That tiny difference is a crucial advantage of EDA. You no longer need to interact or even know of the existence of services outside your own subdomain, and that opens up a lot of possibilities.
Introducing new services
Consider the case of adding a new service that needs to know when orders were created in order to do some additional processing, like analysing order’s data in real time to display it on a product team dashboard.
In the request/response world, you’d need to modify the Orders Service to also send the shipOrder() call to the new service, however in the EDA world, the new service just needs to subscribe to the Order Created event and nobody else needs to know or change anything.
Cool, uh?
I’ve introduced quite a few concepts in this last example, so in the following sections there’s a breakdown of components, considerations, patterns and deciding whether EDA is for you.
What components are essential in an Event-Driven Architecture?
Services
To be able to design an EDA, your system needs to be structured in loosely coupled services. Either you start building them right now, or start migrating from a monolith slowly. It could be a SOA or a Microservices architecture. The point is there’s a need for communication across different services.
Communication
The communication medium is also important, you need a message system that can carry event data, preferably a message queue system like RabbitMQ or a streaming platform like Kafka.
These are the only 2 major components. After that, the needs of your system will vary quite considerably and you may or may not need other components like data stores, analytics, APMs etc.
That being said, in a system this complex, you likely want good monitoring and a well designed data mesh. If you’re taking the trouble of going this route, it’s definitely worth it, if not an actual requirement.
What do you need to consider when designing an EDA?
If you find yourself designing an event-driven architecture, you’re probably not sitting there drawing before you build it all from scratch. Chances are you will (or have been) evolving it during a period of time.
The design considerations are more of an ongoing set of principles and patterns you need to keep in mind, and will vary depending on your growth and desired scale.
The point in time in which you set it up is also important. If you’re just getting started, there’s no point in designing something to handle Facebook’s type of traffic.
At the beginning, keep it simple but allow room for growth. Decide which patterns you’d like to follow and build your services following those guidelines.
That being said, there are a few common patterns you might wanna check out
Domain Driven Design
The idea, as the name might indicate, is that software development is centered around a particular domain, with specific rules and processes for that domain. There are clear boundaries and communication with other domains happen via events triggered to “the outside”. This allows for other domains to listen and update based on what’s going on somewhere else, without linking together decoupled parts of the system.
Event Sourcing
This approach stores the state of a business entity (or object) as it goes through changes. Whenever there’s a change event, we store it as part of the transaction. This allows us to reconstruct the state of an object based on the events that happened to it.
Setting up a streaming platform like Kafka, where events can be permanently stored, makes it easier to perform these operations atomically.
An event sourcing example could be our customer order’s scenario. We could store the order created, validated, processed, shipped and completed after every state change.
Command Query Responsibility Segregation (CQRS)
This is probably the ideal pattern to follow when designing event-driven architectures. Read and write responsibilities are separated, allowing services to read particular data stores (or build their own local caches), while data owners are in charge of writing the data. This pattern can be complex and should only be used if you really need it, but the idea of having models to read and others to write creates a world of problem solving opportunities.
Is EDA for you? Complexity vs Scale
We’ve probably all heard of the people that designed a complex Microservices or Serverless architecture, with a multitude of programming languages and frameworks when they only needed a website with a couple of services.
Like any other pattern or tool in Software Engineering, EDA is not meant for every scenario. You need to measure what you are getting out of it, and whether you really need it.
The natural path of a small Software system that becomes a big system under this paradigm is actually getting started with a single (or a small collection) of monolith apps that do just the basics.
As more functionality gets added, you can start building services (or microservices) to tend those needs. At some point you’ll reach the threshold where communication is a constraint and you may need to redesign your services to communicate via events. This is the point when you want to start thinking about complex patterns, but wait until you see the need.
You can even use lightweight solutions like message queues to pass messages around before committing to a full blown streaming service.
The point is, you need to keep a balance between complexity and scale. You want to be prepared for scale, but not to make your engineers’ lives miserable too soon without good reason.
What tools are out there to help you
Two of my favourite tools for implementing communication in Event-Driven architectures out there are RabbitMQ (for lightweight message queues) and Kafka (for heavy distributed streaming).
They both have their own advantages and disadvantages. As general guidance, I’d use RabbitMQ for message passing to replace some RPC scenarios, or to implement a lightweight EDA. If you set it up in a correct way, RabbitMQ can carry you a long way before you need a heavier solution.
If your needs are more complex and elaborated, Kafka is probably the way to go. Nowadays you can use cloud services like Confluent or Cloudkarafka that make it easier for you to set up a Kafka cluster with a few clicks and start testing.
The most important part here is to select something that matches your needs. Anything else will be a bother rather than helpful.
So, what does it look like in the end?
EDAs can be as complex as you want it, but the principle remains the same. You communicate passing events about what happened and let other services decide whether they’re interested or not.
Here’s how our example of placing an online order could look like in the background. The below figure shows how the different services interact with the events placed on the streaming platform (in this case Kafka).
Notice how the orders service does three things,
- It receives orders and sends an order creation event.
- It listens to an order validated event.
- It shows the status of an order via GET once it updates its own data store.
Nothing else is important for it. There’s a fraud component, an inventory component, an order validation component, and email service… All of these are working independently to fulfil their duty and when they do it, they just follow the pattern and send the appropriate event.
An additional benefit is that all these services can be scaled independently too. If some services are busier than others, more power can go to them. At the same time, they are just dealing with one item of the stream at a time (figuratively speaking, more than likely you’ve parallelised it), so the processing is really efficient as they’re constantly performing tasks like an assembly line.
Conclusion
Software Engineering is full of pretty cool challenges and innovation is always around the corner. It seems like every day people are creating really cool stuff and coming up with new ways to solve problems.
The hard thing is to take a closer look and decide what’s the best tool for the job at hand. From the best programming language(s), to the best cloud hosting, framework or system architecture.
When opting to go the complex route towards Event-Driven architectures, there’s a lot of fun to be had, but also lots of challenges to overcome. We must keep an eye on the fundamentals and learn the software patterns to avoid common mistakes.
At this level, bad design decisions are going to cost you down the line, and are probably harder to undo. Doing the heavy thinking upfront will help you understand better what you’re trying to achieve and determine how you want to build your components.
Keep an eye on what the industry is doing, ask lots of questions and get together with people that have already solved some of the problems you’re facing. That’s probably the best approach to designing complex systems.
Happy designing!
