Client Configuration

Working with RabbitMQ

RabbitMQ is a message broker that accepts and forwards messages. You can think about it as a post office: when you put the mail that you want sent into a post box, you can be sure that the letter carrier will eventually deliver the mail to your recipient. In this analogy, RabbitMQ is a post box, a post office, and a letter carrier.

A quick way to get started would be to look at this Python script: RabbitMQ consumer that logs to console and file · GitHub.

To configure your AMQP client first make sure you have the access credentials (see Getting Access). The credentials will include the host, vhost, username, password, and queue name.

Next, you will have to choose and set up the client library for your preferred programming language. A quick way to get started would be to look at this Python script: RabbitMQ consumer that logs to console and file · GitHub.

Using TLS

We have TLS enabled in front of our RabbitMQ clusters therefore clients need to use it. The configuration depends on the client drivers; for example, Python clients can simply use a connection string with amqps:// as the protocol:

amqps://{username}:{password}@{host}/{vhost}

while others might require the connection options port=5671, ssl=true to be set explicitly.

Connection examples

pika Python library with connection string:

>>> import pika
>>> consume_uri = 'amqps://myusername:mypassword@host.example.com/vhost?heartbeat=30'
>>> params = pika.URLParameters(consume_uri)
>>> connection = pika.BlockingConnection(params)

or with explicit options:

>>> import ssl
>>> ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
>>> ssl_options = pika.SSLOptions(context=cxt)
>>> credentials = pika.PlainCredentials('myusername', 'mypassword')
>>> options = pika.ConnectionParameters(
...     host='host.example.com',
...     virtual_host='vhost',
...     port=5671,
...     ssl_options=ssl_options,
...     heartbeat=30,
...     credentials=credentials)
>>> connection = pika.BlockingConnection(options)

kombu Python library with connection string:

>>> import kombu
>>> consume_uri = 'amqps://myusername:mypassword@host.example.com/vhost?heartbeat=30'
>>> connection = kombu.Connection(consume_uri, heartbeat=30)
>>> connection.connect()

or with explicit options:

>>> connection = kombu.Connection(
...     hostname='host.example.com',
...     port=5671,
...     ssl=True,
...     virtual_host='vhost',
...     userid='myusername',
...     password='mypassword',
...     heartbeat=30)
>>> connection.connect()

Important: Consumer/Publisher Permissions

Data consumers only have permissions to consume messages from their outbox queue.

📝 Note

When setting up the AMQP client library, make sure to disable queue auto-declare or other configure/write features. Otherwise your connections will be closing with ACCESS_REFUSED errors.

>>> queue = kombu.Queue('queue-name', no_declare=True)

AMQP Consumer Configuration Guidelines

• Consumers should use long-lived connections and avoid periodically opening/closing connections. They should configure a heartbeat option on RabbitMQ Connection of 20-30 seconds. Refer to your client library docs on how to set this up.
• Consumers should acknowledge the message once they're done processing it. Not doing so can re-queue messages and cause consumers to receive old duplicates.
• Consumer prefetch value can (and should) be adjusted to minimize network delay and increase throughput. Messages in outbox queues have a time-to-live (TTL) of 30 minutes. This means that unconsumed messages are dropped after this duration. It also means that when a consumer connects for the first time, it may receive messages from this time window.
• Clients should automatically reconnect in case the idle connection is terminated.
• Clients should not cache DNS responses for long, our TTL is 60s.