What is rabbitmq queue?

According to your use case, there are several RabbitMQ Queue Types that allow several services in your application to interact with each other without worrying about message loss while ensuring the quality of service (QoS) requirements. It also enables better control and efficient message routing facilitating extensive decoupling of applications.

In this article, you will learn about various RabbitMQ Queue Types in detail.

RabbitMQ is a popular open-source Message Queue software that allows several applications to interact and scale up. Acting as a messaging broker, RabbitMQ provides you with a common platform that supports Asynchronous messaging. Here, applications can securely send and receive messages and RabbitMQ ensures a safe environment for your messages to live until consumed.

RabbitMQ is designed using Erlang for building a stable, reliable, fault-tolerant, and highly scalable system that effectively handles multiple operations in parallel. It also supports different types of protocols(method of data transportation between devices). Introduced with support for Advanced Message Queuing Protocol (AMQP), it now also provides complete support for Message Queuing Telemetry Transport (MQTT), Streaming Text Oriented Messaging Protocol (STOMP), and several other common protocols.

Internally, the architecture of all the RabbitMQ Queue types is simple. There is a client application called a producer that creates a message and sends it to a broker (message queue). Other applications, called consumers, link to the Message queues and subscribe to the messages that are processed.  All of the Queued messages are stored until the consumer extracts the message. The RabbitMQ software can act as a message producer or consumer, or both a consumer and a producer.

Using RabbitMQ Message Queues, you can avoid unnecessary response delays as it allows web servers to respond to requests quickly rather than being forced to perform resource-intensive tasks on the spot. Employing any of the RabbitMQ Queue Types is often a good choice when you need to distribute a message among multiple consumers(message receivers) or even for balancing the load between workers. Owing to the flexibility of all the RabbitMQ Queue Types, you can send a request in one programming language and handle it in some other programming language. You can use RabbitMQ in the following scenarios:

RabbitMQ Queues is an ordered data structure that allows you to add(enqueue) at the tail and retrieve(dequeue) from the head. Many messaging protocols and platforms consider publishers and consumers to interact using a queue-like storage method. It is to be noted that all the RabbitMQ Queue Types are based on the FIFO (“first-in, first-out”) principle. To understand the RabbitMQ Queues, you can go through the following aspects:

All the Rabbit Queues Types have to be properly named allowing the applications to reference them easily. These applications can either choose queue names or let the broker generate a name for them. While naming the queue, the following guidelines are followed:

Queue Properties determine how a queue behaves. You can note down the following set of mandatory and optional properties:

When queues like auto-delete or exclusive use a known (static) name and the client disconnects and reconnects immediately, the RabbitMQ node will delete such queues and then try to reconnect them. This leads to a natural race condition of recovering the client. This can lead to errors and exceptions during client-side reconnection, causing unnecessary confusion and impacting application availability.

You must declare a queue before you can use it. When you declare a queue, it is created if the queue does not already exist. In case a queue already exists and its attributes match the attributes in the declaration, the declaration has no impact. If the existing queue attribute does not match the attribute of the declaration, a channel-level exception with code 406 (PRECONDITION_FAILED) will be thrown.

You can either choose durable or transient RabbitMQ Queues. Metadata of a durable queue is stored on disk, while metadata of a transient queue is stored in memory when possible. You can assume a similar difference for messages at publishing time in some protocols, e.g. AMQP 0-9-1 and MQTT.

For cases where durability is essential, use durable queues and ensure that publish mark published messages as persisted. All the transient queues are deleted when the node boots and therefore, by design, it does not survive a node reboot. This also results in discarding all the Messages in transient queues On the other hand, Durable queues will be recovered on node boot along with the messages in them published as persistent. It is to be noted that even if the messages are stored in a durable queue, they will be deleted during the recovery if they were published as transient.

You can define Queues RabbitMQ as sequenced collections of messages. Messages are enqueued and dequeued (delivered to consumers) in the FIFO manner. However, you may not always get the FIFO order, for instance, for priority and sharded RabbitMQ Queue Types.

Check out the following RabbitMQ Queue Types:

You will often encounter that with some workloads, their respective queues don’t last that much. You also may choose to manually delete the queues you have declared before disconnection, though this is not an effective method. Moreover, the client connections often fail, thereby leading to unused resources(queues) being left behind.

Thus, you can consider Temporary Queues as another of the RabbitMQ Queue Types. There are potentially three ways to make the queue deleted automatically:

An auto-delete queue will be deleted when its last consumer is canceled (e.g. using the basic.cancel in AMQP 0-9-1) or exits (closed channel or connection, or lost TCP connection with the server).

There will be cases where a queue never had any consumers. For example, when all consumption happens using the basic.get method (the “pull” API), it won’t be automatically deleted. For such cases, you have to use exclusive queues or queue TTL.

In AMQP 0-9-1, your application can ask the broker to generate a unique queue name. You can employ this feature by sending an empty string as the queue name argument. You can get the same generated name through various methods in the same channel by using the empty string where a queue name is required. This concept is applicable as the channel remembers the last server-generated queue name.

Server-named queues are ideal for a state that is transient and applicable to a particular consumer (application instance). This allows the applications to send such names in message metadata to let other applications respond to them. Or else, you should know the names of server-named queues and utilized them only by the declaring application instance. The instance is also responsible for properly setting up appropriate bindings (routing) for the queue so that publishers can use well-known exchanges rather than using the server-generated queue name directly.

While dealing with various RabbitMQ Queue Types, you need to consider the following aspects:

For all the RabbitMQ Queue Types, you can easily limit their length. Also, both Queues and messages can have a TTL(Time-to-Live). You can use both of these functions for data expiration as well as a method to limit the resources (RAM, disk space) a queue can use at most. For example, when consumers go offline or their throughput falls behind publishers.

RabbitMQ records several parameters about all the RabbitMQ Queue Types. You can easily access them via RabbitMQ HTTP API and management UI, which is designed for monitoring. These metrics include queue length, ingress and egress rates, number of consumers, number of messages in various states (for instance, ready for delivery or unacknowledged), number of messages in RAM vs. on disk, etc.

You can also use the rabbitmqctl to list queues and some basic metrics. By using the rabbitmq-top plugin, you can easily monitor runtime metrics like VM scheduler usage, queue (Erlang) process GC activity, amount of RAM used by the queue process, queue process mailbox length, etc. You can also view the individual queue pages in the management UI.

Whether your queue replica is a leader or follower, it is limited to a single CPU core on the hot code path. Therefore, by design, it assumes that most practical systems use several queues. On a general note, a single queue is usually assumed as an anti-pattern (not just because of resource usage).  If there is a need to replace the order of messages with parallelism (improved CPU core utilization), rabbitmqsharding provides a great way for clients to do that transparently.

Messages can be used by registering a consumer (subscription). That is, RabbitMQ either sends the message to the client or retrieves it separately for a protocol that supports the message (such as the basic.get AMQP 091 method), as with HTTP GET. Delivered messages are explicitly or automatically verified by the consumer as soon as the delivery is written to the connection socket.

Auto acknowledgment mode typically provides higher throughput and uses less network bandwidth. However, with regard to failure, the warranty is the least. As a good practice, you should first use the manual verification mode.

In this article, you have learned about various RabbitMQ Queue Types. RabbitMQ is a widely used messaging broker that provides a safe environment for your applications to send or receive messages securely. This open-source tool provides you with several RabbitMQ Queue Types that can be used according to your business needs. With auto-delete queues, TTL & exclusive queues you can easily delete queues automatically when required. All the RabbitMQ Queue Types can be configured and monitored based on various parameters and metrics respectively.

As you collect and manage your data in RabbitMQ and across several applications and databases in your business, it is important to consolidate it for a complete performance analysis of your business. However, it is a time-consuming and resource-intensive task to continuously monitor the Data Connectors. To achieve this efficiently, you need to assign a portion of your engineering bandwidth to Integrate data from all sources, Clean & Transform it, and finally, Load it to a Cloud Data Warehouse, BI Tool, or a destination of your choice for further Business Analytics. All of these challenges can be comfortably solved by a Cloud-based ETL tool such as Hevo Data.

Hevo Data, a No-code Data Pipeline can transfer data in Real-Time from a vast sea of 100+ sources to a Data Warehouse, BI Tool, or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!

If you are using Messaging systems like Kafka and searching for a no-fuss alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo, with its strong integration with 100+ sources and BI tools(Including 40+ Free Sources), allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.

There are a number of tools in this field, but RabbitMQ and Apache Kafka are two of the most popular. While both are robust and reliable, they have unique features and use cases that make them distinct.

RabbitMQ is an open-source message broker. It's recognized for its flexibility and support for various messaging protocols. Apache Kafka is rapidly gaining popularity and is known for its ability to handle real-time data feeds with low latency.

In this tutorial, I'll focus on RabbitMQ, its core features, and how you can use it to effectively build scalable, loosely coupled applications.

Stay with me as we explore the world of RabbitMQ, its unique capabilities, and how it sets itself apart in the ever-evolving landscape of message queue technologies.

RabbitMQ is an open-source message broker software (also called a message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). It provides a common platform for sending and receiving messages.

RabbitMQ supports multiple messaging protocols and can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

Consider an e-commerce website (like Amazon) where users can place orders that need to be processed.

The order processing system might involve several steps, such as inventory checks, payment processing, shipping, and so on, each of which can potentially take some time and are ideally handled asynchronously.

In this example, I'll be showing you how to use Docker to run RabbitMQ. But if you prefer, you can install and run it manually on your system. The official documentation provides a detailed guide on how to do this.

I find Docker to be a convenient tool for running RabbitMQ because it simplifies the setup and management processes. If you're new to Docker, I recommend reading my previous tutorials related to Docker for a thorough understanding.

To get started, you'll need to pull the Docker image from Docker Hub.

Before running the image, you need to map the two port numbers (15672 and 5672).

I'm running RabbitMQ on the Docker by adding the above-mentioned ports. Refer the below screenshot for additional reference.

Our RabbitMQ Server is up and running in Docker.

I'll using amqplib, which is a popular NodeJS library that provides an API for interacting with RabbitMQ. It supports all the features of RabbitMQ's AMQP 0-9-1 model, including things like confirm channels, exchanges, queues, bindings, and message properties.

I have been using the term AMQ Protocol in this tutorial and I feel like this is the right time to give a quick introduction to it.

AMQP stands for Advanced Message Queuing Protocol. It is an open-standard protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability, and security.

AMQP has the following components:

The AMQP protocol enables standardized communication between different applications, making it a good choice for a messaging system in a microservices architecture. This protocol can ensure that a message is delivered not just to the messaging system, but all the way to the correct consumer.

You'll remember the example that I described at the beginning of this article about the high level implementation of message queues in a E-commerce site.

Let's go through the same but a bit deeper in the RabbitMQ context.

When a customer places an order on the e-commerce website, the order service produces a message to a RabbitMQ exchange. The message contains information about the product ID and the quantity ordered.

An inventory service is set up as a consumer to receive messages from a queue bound to the exchange. Once it receives a message, it reduces the inventory for the specified product by the ordered quantity. If the inventory is insufficient, it can send a message back to the order service to indicate the problem.

Once the inventory service successfully updates the inventory, it'll send a message to the order service. The order service, set up as a consumer for this exchange, can then update the order status and notify the customer.

Let's assume your inventory service is down for some time. Then the messages in the RabbitMQ queue will stay there and won't be lost. Once the inventory service is back online, it'll continue processing the messages from where it left off.

During high traffic periods, more instances of the inventory service can be launched, all consuming messages from the same queue. This enables load balancing and ensures that the system can handle the increased load.

Let's come back to our implementation. In this example, we'll send a message from sender to our receiver. On the receiver end, we print the message on the console.

This is the component or part of our application that creates and sends messages to the messaging queue.

The sender does not send messages directly to the consumer. Instead, it sends the messages to an exchange in RabbitMQ. The exchange then routes the messages to the appropriate queue based on certain criteria.

Here we're creating a queue called product_inventory. Alternatively, you can clone my repo from here.

We can send only byte arrays in the message, So I convert the message object to a string and send it to the queue. From the above code, you can understand that we're creating a channel and sending a message through it.

This is the component or part of our application that receives and processes the messages from the queue. A consumer can continuously poll the queue for new messages or be set up to automatically trigger when a new message is added to the queue.

Here we're listening for messages. Alternatively, you can clone my repo from here.

In the above code, we're listening for messages (consume) and print them on the console once we receive it.

In the context of our specific use case, the file containing the message sender (or producer) should typically be located in the root directory of our e-commerce site project. This is where we generate and send messages based on user actions, such as placing an order.

On the other hand, the file containing the message receiver (or consumer) should ideally be located in the inventory management service. This is because the inventory management service is responsible for processing these messages, such as updating the inventory when an order is placed.

Let's run our receiver service first:

The initial output of the consumer service looks like this:

Once we run our sender, a message will be sent to the consumer. Run the same command yarn start on the sender repo.

Here's the output of it:

Hurray! We received a message sent from the RabbitMQ producer in our consumer service.

In this article, we've explored the basics of RabbitMQ, a robust and efficient message broker, and demonstrated its application in a NodeJS environment.

Using a simple e-commerce scenario, we showcased how to set up a sender (producer) and a consumer to handle asynchronous messages between different components of our application. But in real-world applications, you will likely encounter more complex scenarios that require advanced integrations and the usage of RabbitMQ.

To navigate these complexities, it's crucial to have a solid understanding of RabbitMQ's underlying concepts and its AMQP protocol. As you delve deeper into RabbitMQ, you'll find it to be an incredibly versatile tool, capable of handling a wide range of messaging needs, and ultimately helping you build scalable, decoupled, and resilient applications.

Check out the source code of the project on GitHub: rabbit-sender, rabbit-receiver.

Now we will see how to create a queues in rabbitmq using web management portal. To create a queue in rabbitmq, open web management portal and enter a default credentials to login and then choose Queues tab.

After navigate to Queues tab, you will see “Add a new queue” just click on that panel to expand like as shown below.

After clicking on Add a new queue option, a new panel will open and that will contain a different properties to create a new queue like as shown below.

If you observe above picture, to create a new queue in rabbitmq we need to enter a value for multiple parameters. Now we will see what is the necessity of each parameter in detail to create a queue in rabbitmq.

Following are the different type of properties which we need to enter to create a queue in rabbitmq.

It’s a name of the queue which we can use it to refer in our application. The name must be unique and it must not be any system define queue name.

Durability is a property of queue which tells that a message can survive even after server restarts (broker restart) or not.

There are 2 types of Durability options.

There are 2 options in auto delete, those are

If queue is exclusive, then the durability attribute has no effect because the queue will be deleted as soon as client disconnects (or its connection is lost). The auto-deleted queues are deleted when the last consumer is canceled (or its channel is closed, or its connection is lost).

In rabbitmq, arguments are an optional and these are used by plugins and broker-specific features such as message TTL, queue length limit, etc.

Following are the different type of arguments which we can add based on our requirements.

By using time-to-live (ttl) argument, we can set a timespan to queue and it will be discarded if it reaches defined timespan. Here, we can set time in milliseconds.

By using auto expire argument, we can set an expiry time to queue and it is used to define how long a queue can be unused before it is automatically deleted.

Here unused means the queue has no consumers, the queue has not been re-declared, and basic.get method has not been invoked for a duration of at least the expiration period.

By using max length argument, we can define how many (ready) messages a queue can contain before it starts to drop them from its head.

A maximum number of messages can be set by supplying the x-max-length queue declaration argument with a non-negative integer value.

E.g. If you set value x-max-length = 2 and if you publish 3 messages in the queue, then only 2 messages will be there and the oldest will be deleted from the queue.

By using max length bytes argument, we can set total body size for ready messages a queue can contain before it starts to drop them from its head.

We can set Maximum length in bytes by supplying x-max-length-bytes queue declaration argument with a non-negative integer value.

E.g. If you set value x-max-length-bytes = 1000000 (1MB) and if you publish messages in queue and the queue size increase more than 1 MB then the oldest will be deleted from the queue (drop them from its head).

By using overflow behaviour argument, we can set the queue overflow behavior. This determines what happens to messages when the maximum length of a queue is reached. Valid values are drop-head or reject-publish.

By using dead letter exchange argument, we can set an optional name for exchange to which messages will be republished if they are rejected or expire.

Optional replacement routing key will be used when a message is dead-lettered. If this is not set, then the message's original routing key will be used.

For example, if you publish a message to an exchange with routing key foo, and that message is dead-lettered, then it will be published to its dead letter exchange with routing key foo. In case, if we declared a queue message with x-dead-letter-routing-key, then the message will be published to its dead letter exchange with routing key bar.

By using this argument, we can set a maximum number of priority levels for the queue to support. In case, if we didn’t set ("x-max-priority") argument, then the queue will not support message priorities.

To declare a priority queue, we need to use x-max-priority optional queue argument. This argument should be a positive integer between 1 and 255, indicating the maximum priority the queue should support.

By using lazy mode ("x-queue-mode") argument, we can set the queue into a lazy mode, keeping as many messages as possible on disk to reduce the RAM usage. In case, if it not set, then the queue will keep an in-memory cache to deliver messages as fast as possible.

By using master locator ("x-queue-master-locator") argument, we can set the queue into master location mode to determine the rule by which the queue master is located when declared on a cluster of nodes.

Finally, we are done with all the properties of queue. Now we will create a new queue for that enter queue name as “demoqueue” after that choose durability as a Durable and final option we are going set is auto delete to No and click on Add queue button to create a queue.

After creating a queue, you can view queue which you have recently added, it is located just above the add queue panel like as shown below.

RabbitMQ is a message broker, a tool for implementing a messaging architecture. Some parts of your application publish messages, others consume them, and RabbitMQ routes them between producers and consumers. The broker is well suited for loosely coupled microservices. If no service or part of the application can handle a given message, RabbitMQ keeps the message in a queue until it can be delivered. RabbitMQ leaves it to your application to define the details of routing and queuing, which depend on the relationships of objects in the broker: exchanges, queues, and bindings.

If your application is built around RabbitMQ messaging, then comprehensive monitoring requires gaining visibility into the broker itself. RabbitMQ exposes metrics for all of its main components, giving you insight into your message traffic and how it affects the rest of your system.

RabbitMQ runs as an Erlang runtime, called a node. A RabbitMQ server can include one or more nodes, and a cluster of nodes can operate across one machine or several. Connections to RabbitMQ take place through TCP, making RabbitMQ suitable for a distributed setup. While RabbitMQ supports a number of protocols, it implements AMQP (Advanced Message Queuing Protocol) and extends some of its concepts.

At the heart of RabbitMQ is the message. Messages feature a set of headers and a binary payload. Any sort of data can make up a message. It is up to your application to parse the headers and use this information to interpret the payload.

The parts of your application that join up with the RabbitMQ server are called producers and consumers. A producer is anything that publishes a message, which RabbitMQ then routes to another part of your application: the consumer. RabbitMQ clients are available in a range of languages, letting you implement messaging in most applications.

RabbitMQ passes messages through abstractions within the server called exchanges and queues. When your application publishes a message, it publishes to an exchange. An exchange routes a message to a queue. Queues wait for a consumer to be available, then deliver the message.

You’ll notice that a message going from a producer to a consumer moves through two intermediary points, an exchange and a queue. This separation lets you specify the logic of routing messages. There can be multiple exchanges per queue, multiple queues per exchange, or a one-to-one mapping of queues and exchanges. Which queue an exchange delivers to depends on the type of the exchange. While RabbitMQ defines the basic behaviors of topics and exchanges, how they relate is up to the needs of your application.

There are many possible design patterns. You might use work queues, a publish/subscribe pattern, or a Remote Procedure Call (as seen in OpenStack Nova), just to name examples from the official tutorial. The design of your RabbitMQ setup depends on how you configure its application objects (nodes, queues, exchanges…). RabbitMQ exposes metrics for each of these, letting you measure message traffic, resource use, and more.

With so many moving parts within the RabbitMQ server, and so much room for configuration, you’ll want to make sure your messaging setup is working as efficiently as possible. As we’ve seen, RabbitMQ has a whole cast of abstractions, and each has its own metrics. These include:

This post, the first in the series, is a tour through these metrics. In some cases, the metrics have to do with RabbitMQ-specific abstractions, such as queues and exchanges. Other components of a RabbitMQ application demand attention to the same metrics that you’d monitor in the rest of your infrastructure, such as storage and memory resources.

You can gather RabbitMQ metrics through a set of plugins and built-in tools. One is rabbitmqctl, a RabbitMQ command line interface that lists queues, exchanges, and so on, along with various metrics. Another is a management plugin that reports metrics from a local web server, as well as a Prometheus plugin that can transmit metrics in the OpenMetrics format. Several tools report events. We’ll tell you how to use these tools in Part 2.

Exchanges tell your messages where to go. Monitoring exchanges lets you see whether messages are being routed as expected.

In RabbitMQ, you specify how a message will move from an exchange to a queue by defining bindings. If a message falls outside the rules of your bindings, it is considered unroutable. In some cases, such as a Publish/Subscribe pattern, it may not be important for consumers to receive every message. In others, you may want to keep missed messages to a minimum. RabbitMQ’s implementation of AMQP includes a way to detect unroutable messages, sending them to a dedicated (‘Alternative’) exchange. In either of the plugins (see Part 2), capture the unroutable returns metric, constraining the count to a given time interval. If some messages have not been routed properly, the rate of publications into an exchange will also exceed the rate of publications out of the exchange, suggesting that some messages have been lost.

RabbitMQ runs inside an Erlang runtime system called a node. For this reason the node is the primary reference point for observing the resource use of your RabbitMQ setup.

When use of certain resources reaches a threshold, RabbitMQ triggers an alarm and blocks connections. These connections appear as blocking in built-in monitoring tools, but it is left to the user to set up notifications (see Part 2). For this reason, monitoring resource use across your RabbitMQ system is necessary for ensuring availability.

As you increase the number of connections to your RabbitMQ server, RabbitMQ uses a greater number of file descriptors and network sockets. Since RabbitMQ will block new connections for nodes that have reached their file descriptor limit, monitoring the available number of file descriptors helps you keep your system running (configuring the file descriptor limit depends on your system, as seen in the context of Linux here). On the front page of the management plugin UI, you’ll see a count of your file descriptors for each node. You can fetch this information through the HTTP API (see Part 2). This timeseries graph shows what happens to the count of file descriptors used when we add, then remove, connections to the RabbitMQ server.

RabbitMQ goes into a state of alarm when the available disk space of a given node drops below a threshold. Alarms notify your application by passing an AMQP method, connection.blocked, which RabbitMQ clients handle differently (e.g. Ruby, Python). The default threshold is 50MB, and the number is configurable. RabbitMQ checks the storage of a given drive or partition every 10 seconds, and checks more frequently closer to the threshold. Disk alarms impact your whole cluster: once one node hits its threshold, the rest will stop accepting messages. By monitoring storage at the level of the node, you can make sure your RabbitMQ cluster remains available. If storage becomes an issue, you can check queue-level metrics and see which parts of your RabbitMQ setup demand the most disk space.

As with storage, RabbitMQ alerts on memory. Once a node’s RAM utilization exceeds a threshold, RabbitMQ blocks all connections that are publishing messages. If your application requires a different threshold than the default of 40 percent, you can set the vm_memory_high_watermark in your RabbitMQ configuration file. Monitoring the memory your nodes consume can help you avoid surprise memory alarms and throttled connections.

The challenge for monitoring memory in RabbitMQ is that it’s used across your setup, at different scales and different points within your architecture, for application-level abstractions such as queues as well as for dependencies like Mnesia, Erlang’s internal database management system. A crucial step in monitoring memory is to break it down by use. In Part 2, we’ll cover tools that let you list application objects by memory and visualize that data in a graph.

Any traffic in RabbitMQ flows through a TCP connection. Messages in RabbitMQ implement the structure of the AMQP frame: a set of headers for attributes like content type and routing key, as well as a binary payload that contains the content of the message. RabbitMQ is well suited for a distributed network, and even single-machine setups work through local TCP connections. Like monitoring exchanges, monitoring your connections helps you understand your application’s messaging traffic. While exchange-level metrics are observable in terms of RabbitMQ-specific abstractions such as message rates, connection-level metrics are reported in terms of computational resources.

The logic of publishing, routing, queuing and subscribing is independent of a message’s size. RabbitMQ messages are always first-in, first-out, and require a consumer to parse their content. From the perspective of a queue, all messages are equal.

One way to get insight into the payloads of your messages, then, is by monitoring the data that travels through a connection. If you’re seeing a rise in memory or storage in your nodes, the messages moving to consumers through a connection may be holding a greater payload. Whether the messages use memory or storage depends on your persistence settings, which you can monitor along with your queues. A rise in the rate of sent octets may explain spikes in storage and memory use downstream.

Queues receive, push, and store messages. After the exchange, the queue is a message’s final stop within the RabbitMQ server before it reaches your application. In addition to observing your exchanges, then, you will want to monitor your queues. Since the message is the top-level unit of work in RabbitMQ, monitoring queue traffic is one way of measuring your application’s throughput and performance.

Queue depth, or the count of messages currently in the queue, tells you a lot and very little: a queue depth of zero can indicate that your consumers are behaving efficiently or that a producer has thrown an error. The usefulness of queue depth depends on your application’s expected performance, which you can compare against queue depths for messages in specific states.

For instance, messages_ready indicates the number of messages that your queues have exposed to subscribing consumers. Meanwhile, messages_unacknowledged tracks messages that have been delivered but remain in a queue pending explicit acknowledgment (an ack) by a consumer. By comparing the values of messages, messages_ready and messages_unacknowledged, you can understand the extent to which queue depth is due to success or failure elsewhere.

You can also retrieve rates for messages in different states of delivery. If your messages_unacknowledged rate is higher than usual, for example, there may be errors or performance issues downstream. If your deliveries per second are lower than usual, there may be issues with a producer, or your routing logic may have changed.

This dashboard shows message rates for three queues, all part of a test application that collects data about New York City.

A queue may persist messages in memory or on disk, preserving them as pairs of keys and values in a message store. The way RabbitMQ stores messages depends on whether your queues and messages are configured to be, respectively, durable and persistent. Transient messages are written to disk in conditions of memory pressure. Since a queue consumes both storage and memory, and does so dynamically, it’s important to keep track of your queues’ resource metrics. For instance you can compare two metrics, message_bytes_persistent and message_bytes_ram, to understand how your queue is allocating messages between resources.

Since you configure consumers manually, an application running as expected should have a stable consumer count. A lower-than-expected count of consumers can indicate failures or errors in your application.

A queue’s consumers are not always able to receive messages. If you have configured a consumer to acknowledge messages manually, you can stop your queues from releasing more than a certain number at a time before they are consumed. This is your channel’s prefetch setting. If a consumer encounters an error and terminates, the proportion of time in which it can receive messages will shrink. By measuring consumer utilization, which the management and Prometheus plugins (see Part 2) report as a percentage and as a decimal between 0 and 1, you can determine the availability of your consumers.

Much of the work that takes place in your RabbitMQ setup is only observable in terms of abstractions within the server, such as exchanges and queues. RabbitMQ reports metrics on these abstractions in their own terms, for instance counting the messages that move through them. Abstractions you can monitor, and the metrics RabbitMQ reports for them, include:

Monitoring your message traffic, you can make sure that the loosely coupled services within your application are communicating as intended.

You will also want to track the resources that your RabbitMQ setup consumes. Here you’ll monitor:

In Part 2 of this series, we’ll show you how to use a number of RabbitMQ monitoring tools. In Part 3, we’ll introduce you to comprehensive RabbitMQ monitoring with Datadog, including the RabbitMQ integration.