Software requirements tend to become more complex as more features are needed to serve existing or new users. To deliver some of these features or cater for the ever-increasing users base, an application might need to transition from a monolith to one with a microservice architecture. Depending on how large the entire system is, it is common to find services depending on the other services or long-running processes capable of reducing the overall performance of the system. In this article, I will explain what message queues are, how to use them in solving the issues highlighted earlier, the pros and cons of message queues.
Table of Contents:
What are Message Queues?
A queue is a line of items waiting for their turn to be attended to or processed in the order they appeared. Using computer science terminologies, we can say a queue is a First-In-First-Out (FIFO) data structure. A message is the data transferred from the sender (producer) to the receiver (consumer). An example of such a message could be an instruction to start processing a task, e.g. parsing uploaded resumes. It could be a plain message or contain more information needed for the processing.
A message queue is a solution that allows for application-to-application communications in microservice and serverless architectures. Messages are sent and received using an asynchronous communication protocol. The messages are created by the producer, temporarily stored in the message queue and then processed by a single consumer.
Message Queue
Components of a Message Queuing System
The major components of a message queuing systems are the producer, consumers, message, message queue and the message broker. Depending on the message broker selected, you might have other components unique to it. As an example, we will use the simple E-commerce application outlined in the following image.
E-commerce Application Using Microservice Architecture
Producer
The producer is an application or service that creates and publishes the message to the message using an asynchronous communication protocol. In the E-commerce example above, the producer in this case is the Order Service. Once an order is made, it publishes a message to the designated message queue.
Consumer
Consumers are applications or services that connect to a specific message queue to read and process messages in that queue. Messages in the queue are meant to be short-lived. Hence, the consumer is expected to send an acknowledgement so the message can be marked for deletion. Following the E-commerce example, the Shipping Service is a consumer. It consumes messages sent to the queue by the Order Service producer.
Message
A message is the data transferred from the producer to the consumer. This data could typically be a plain message or contain more information along with some headers needed by the message broker for the processing. A simple message sent by the Order Service might look this:
{
"order_id": "78934134422",
"product_sku": "8120725247",
"product_size": "XL",
"delivery_address": {
"city": "Lagos",
"country": "NG",
"postcode": "105102"
}
}
Sample message
Message Queue
A message queue is essentially a container using a queue data structure to hold messages for eventual consumption. A message queue can have one or more producers and/or consumers connected to it.
Message Broker
Message brokers are software that allows applications, services and systems to communicate with each other using an asynchronous communication protocol. The asynchronous nature of this operation makes it possible for applications and services to continue functioning without sacrificing performance while waiting for incoming messages.
RabbitMQ, Apache Kafta, Redis, Amazon SQS, IBM MQ and IronMQ are some of the popular message brokers. There are several open-source message brokers, with RabbitMQ being the most widely deployed open-source message broker. In subsequent articles, we will demonstrate how to integrate some of these message brokers into a project.
Pros and Cons of Message Queues
Pros
- Improved Performance: With message queues, requests to carry out long-running tasks can be added to the queue to be later processed by the consumer. The producers can simply add the requests to the queue and then move on to other tasks, while can the consumers can always process new incoming messages. Performance is improved since these tasks can run in the background and doesn't block the system.
- Low Coupling & High Cohesion: Message queues reduce dependencies between different components of the system. With the dependency bottleneck resolved, components can focus solely on their responsibility and business logic.
- Increased Data Reliability: Most message brokers offer message persistency. As soon as messages reach the queue, they are written to the disk apart from being kept in memory. In the event of a failure, there is the guarantee that the messages are still available for the consumers to work with.
Cons
- Increased Complexity: Introduction of message queues to a system architecture increases the overall complexity of the system. Ensuring data consistency and managing the network between the components For simple applications with a limited number of users, message queues might be an overkill.
- Difficulty in Debugging: When the need to debug problem arises, it is difficult to track the program flow. Fortunately, debugging tools, plugins and loggers have been created for most message brokers.
Conclusion
So far we have seen that message queues can greatly improve the performance of software built using microservice or serverless architecture. Once we get over the initial challenge of setting up and integrating message brokers into the software architecture, we stand to gain tremendously from the several benefits of message queues.
In subsequent articles, we will demonstrate a practical implementation of some of these message brokers. I am sure you wouldn't want to miss that.
If you are enjoying this article please share with a friend or drop us a comment below if you have any feedback on this article. Kindly subscribe to our newsletter not to miss future articles.