Introduction to RabbitMQ

RabbitMQ is an open-source message broker that implements the AMQP 0-9-1 protocol. Written in Erlang and maintained by Broadcom, it decouples producers from consumers a publisher sends a message without knowing which service will process it, and a consumer processes messages without knowing which service sent them.

RabbitMQ stores messages in queues and delivers them to consumers. It does not process messages itself that is the responsibility of your application code.

Architecture

graph LR
  Producer["Producer\n(publishes)"]
  Exchange["Exchange\n(routes)"]
  Q1["Queue A"]
  Q2["Queue B"]
  Consumer1["Consumer 1"]
  Consumer2["Consumer 2"]

  Producer -->|"message + routing key"| Exchange
  Exchange -->|"binding"| Q1
  Exchange -->|"binding"| Q2
  Q1 -->|"deliver"| Consumer1
  Q2 -->|"deliver"| Consumer2

Messages flow from producers to exchanges. Exchanges inspect the routing key and route messages to one or more bound queues. Consumers subscribe to queues and process messages independently.

Core AMQP concepts

Producer

An application that creates and publishes messages to an exchange. A producer declares which exchange to publish to and attaches a routing key that the exchange uses for routing decisions.

Exchange

The exchange receives messages from producers and routes them to queues based on its type and the message’s routing key. An exchange does not store messages if no queue is bound to it for a given routing key, the message is dropped (or returned to the producer if mandatory is set).

Queue

A named buffer that holds messages until a consumer retrieves them. Queues can be durable (survive broker restart) or transient. Multiple consumers can subscribe to the same queue, in which case RabbitMQ distributes messages round-robin between them.

Binding

A binding is a rule that connects an exchange to a queue. It may include a binding key (for direct and topic exchanges) or header attributes (for headers exchanges) that control which messages are routed to which queue.

Consumer

An application that subscribes to a queue and processes messages. A consumer registers a callback that is invoked when a message arrives. After processing, it sends an acknowledgement to tell RabbitMQ the message was handled.

Channel

A channel is a lightweight virtual connection multiplexed over a single TCP connection. Applications typically open one channel per thread rather than one TCP connection per thread opening TCP connections is expensive; opening channels is cheap.

Exchange types

Topic exchange pattern examples

Message acknowledgements

By default, RabbitMQ considers a message delivered once it is sent to the consumer. In production you should use manual acknowledgements so a message is only removed from the queue after your code has successfully processed it.

If a consumer disconnects without sending an ack, RabbitMQ requeues the message and delivers it to another consumer.

Dead-letter queues

When a message is rejected without requeue, expires (TTL), or exceeds the queue’s maximum length, RabbitMQ can route it to a dead-letter exchange (DLX). Binding a dead-letter queue to the DLX lets you inspect, alert on, or replay failed messages.

1
2
3
4
5
6
7
8
channel.queue_declare(
    queue='orders',
    durable=True,
    arguments={
        'x-dead-letter-exchange': 'orders.dlx',
        'x-message-ttl': 60000,  # 60 seconds
    }
)

Durability and persistence

By default, queues and messages are transient a broker restart loses them. For reliable messaging:

• Declare queues with durable=True the queue definition survives a restart.
• Publish messages with delivery_mode=2 (persistent) message bodies are written to disk.
• Enable publisher confirms the broker acknowledges each message once it has been written to disk, so you know it is safe.

All three are required together. A durable queue with non-persistent messages still loses undelivered messages on restart.

Practical example

Install pika

1
pip install pika

Publish a message

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost')
)
channel = connection.channel()

channel.exchange_declare(exchange='orders', exchange_type='direct', durable=True)
channel.queue_declare(queue='orders.created', durable=True)
channel.queue_bind(queue='orders.created', exchange='orders', routing_key='created')

channel.basic_publish(
    exchange='orders',
    routing_key='created',
    body=b'{"order_id": 42}',
    properties=pika.BasicProperties(delivery_mode=2),  # persistent
)

print('Message published')
connection.close()

Consume messages

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import pika

def on_message(channel, method, properties, body):
    print(f'Received: {body.decode()}')
    channel.basic_ack(delivery_tag=method.delivery_tag)  # manual ack

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost')
)
channel = connection.channel()

channel.basic_qos(prefetch_count=1)  # process one at a time
channel.basic_consume(queue='orders.created', on_message_callback=on_message)

print('Waiting for messages...')
channel.start_consuming()

basic_qos(prefetch_count=1) prevents RabbitMQ from dispatching more than one unacknowledged message to a consumer at a time, ensuring work is distributed evenly when consumers have different processing speeds.

What to avoid

Do not use auto-ack in production. With auto_ack=True, RabbitMQ marks messages as delivered the moment it sends them. If your consumer crashes mid-processing, the message is lost with no way to recover it.

Do not use fanout when you need targeted routing. Fanout sends every message to every bound queue regardless of content. Use direct or topic exchanges when only specific consumers should receive a message.

Do not skip dead-letter queues. Without a DLX, rejected or expired messages disappear silently. Always configure a dead-letter exchange so you can audit and replay failed messages.

Do not ignore publisher confirms for critical messages. Publishing to RabbitMQ is asynchronous by default. Without confirms, a network blip between your producer and the broker can silently drop messages. Enable channel.confirm_delivery() for financial or audit-critical flows.

Do not share one queue across unrelated consumers. If two unrelated services consume from the same queue, each receives roughly half the messages both will miss messages intended for the other. Give each logical consumer its own queue bound to a shared exchange instead.