In this post we discuss on what are message queues. Message queues are that part of the messaging infrastructure that allow our applications to be decoupled. Whenever there is a need for communication between two disconnected platforms, message queues are one of the primary considerations. They are a feature which supports FIFO (First In First Out), reliable processing etc. Messaging also can be used when either of the communicating parties is offline.
Different messaging technologies
The below are some of the messaging techniques that you might have seen around
Windows MSMQ: Also known as Microsoft Message Queue, this comes as a part of the standard Windows OS offering from Microsoft. This can be enabled / disabled from the Programs and Features dialog box as shown:
IBM WebSphere: IBM has a suite of products and one of them is WebSphere which is also a messaging platform. You have one MQ Server where you define the Queue, Manager, Channel, Protocol, etc. Then using client libraries you can connect and insert messages to this MQ server. More details can be found below:
Kafka: Kafka is also a messaging platform from the Apache foundation which is open-source unlike the above 2 which are proprietary. In Kafka too you have a Kafka server queue to which messages are written and then can be consumed by any client. More details can be found below:
Microsoft Azure Service Bus: Azure the cloud offering from Microsoft also has a messaging capability called ‘Service Bus Queues’. Here you can manage, view all messaging related configuration from the Azure portal in your cloud subscription. More details can be found below:
Basic concepts of messaging
The below are some of the most basic concepts of any messaging platform –
Messaging is termed as a reliable end to end communication medium that provides some guarantee that the content is / will be delivered. This guarantee sometimes refers to a number of attempts made at sending / retrieving messages.
One of the most distinguishing factors of any messaging platform is the message life. Messaging only enables communication between 2 platforms and hence it should not be considered as a permanent storage medium. Therefore the messages have a specific shelf life. Most of the messaging tools have methods where one can specify if they just want to browse the queue and just peek at a message or do a full retrieval of the message which also removes it from the queue.
Topic / Subscriber / Publisher
You might have subscribed to some technical content like newsletters, magazines etc. Hence in this scenario you are the subscriber and the source generating the content becomes the publisher. In a similar way, most messaging platforms allow the calling program to subscribe to a specific topic which is sometimes also referred to as ‘filters’. This enables the source to publish content to subscribers who are interested in one particular topic. For e.g. A messaging platform can create topics and send content to those channels. And the subscribers will see their related content filtered by topics of their choice.
Broker is simply a middle-man program which takes the content from the publisher and sends it to the subscriber in a format agreed by both of the parties. This is referred to as brokered messaging.
Sometimes the content of the message becomes unreadable and when this occurs it is referred to as message poisoning. Such types of messages usually end up in a dead letter queue.
Dead Letter Message Queues
Dead letter queues are the place where all messages who fit the below criteria are kept:
Failed to deliver
Failed to read
Maximum message size reached
Maximum threshold of queue reached
The below are some of the technical concepts of any messaging platform
Messaging Queue Server:
This is a server where the main messaging platform is installed and all server related configuration is done here.
Most of the messaging is done over TCP / IP and hence any MQ server will have at least one channel defined.
Queues and Queue Manager:
Apart from channels, the main communication unit is the queue where the actual messages will be pushed by the MQ server and then pulled or read by the client. This queues have specific names by which the client libraries are able to locate them using a queue manager.
Message Queue Clients:
These are language / framework related libraries or classes that allow one to create a messaging queue client using the above combination of ServerName-IP / ChannelName / QueueName. Clients allow us to establish a connection and once it is established, you can retrieve the messages from the queue which you have specified. While reading the message the client library also gives you many options for connecting to the queue. It can be using a ‘BROWSE’ mode where you can read the message but it won’t be removed from the queue. You can also use a ‘GET’ method which will read but also remove the message from the client. Hence you need to first decide on how you intend to use the messaging queue.
Note: In most cases, you will need some kind of port openings, TLS or SSL level configurations to be in place first before you try to create an MQ client using these libraries.
Since the messaging happens at a channel level, you won’t be getting actual run-time exceptions just like in WCF. Hence most of the messaging exceptions are thrown as a Code-Description combination where it will be something like 2001-No message found kind of format. Hence you will have to handle message exception accordingly in your calling code in .Net.
Whenever there is a need for communication between 2 independent applications / systems, you can consider messaging or a SOA based design for implementation. Please factor in the point that most of the messaging components are proprietary hence the license costs are to be beared in mind.