RabbitMQ advanced configuration and troubleshooting

This page provides information about advanced configuration scenarios and troubleshooting.

Continuous deployment strategy

How to synchronize new public instances

Public instance backups should be done when the SyncState message id is identical for all public nodes. You can then choose one public node and to do a dump. Once the dump is done, create a new historical queue and empty the old queue by consuming all messages. The instance will then be ready to receive new activation messages which post-date the last dump. 

Whenever you need to create a new instance, always use the latest dump. The new instance will be registered to the new historical queue. This will mean that the message id is in SyncState with the latest activation message. New activation messages will be sent to the new historical queue before they register in the old historical queue. The new historical queue will start getting all messages after the last message in old historical queue.

Adding new instances

This series of diagrams shows the best setup for scaling up when adding new public instances to the load balancer.

1. The basic idea is to always keep a spare queue (which does not have a consumer) connected to the exchange. 
   
2. The spare queue stores all activation messages until a new consumer connects. The newly created instance is created with the same initial data as Public 1.
   
3. Public 2 is created with the same data set as Public 1. Before Public 2 starts consuming a new queue (without a consumer) is created.
   

4. Once the remaining messages are consumed a backup is created. This serves as the new state for new instances. This is also the time to add the new public instance to the load balancer.

   When to do a backup?

   Each message has a version number or tag in its header. When a new public is added it can tell the author instance or the exchange about its presence by sending its identity before consuming starts. This enables the exchange or author to create the new queue for the public instance.

   From this point, all activation messages contain a new version or tag and the consumer starts consuming messages in the queue. When the consumer comes across a message with the newest version number, it does the backup.

5. Activation continues and the spare queue (queue_pub_3) is filled with new messages that represent the difference between Public 2 and newest content.
   

Adding a queue after backup

This example shows you how to add a new queue to the example testfanoutAck exchange:

1. Add the queue:

   curl -i -u guest:guest -H "content-type:application/json" \
     -XPUT -d'{"auto_delete":false,"durable":true,"arguments":{},"node":"rabbit@localhost"}' \
     http://localhost:55672/api/queues/%2f/fan2

2. Declare it to be bound to correct exchange:

   curl -i -u guest:guest -H "content-type:application/json" \
      -d'{"routing_key":"","arguments":[]}' \
     http://localhost:55672/api/bindings/%2f/fan2/testfanoutAck

See RabbitMQ Management HTTP API for more.

ACK queue blocked by unacked node

When a consumer encounters an issue because of an activation exception, the consumer stops consuming. The message is sent back to the queue with a non-ack, and an exception is sent over the ACK exchange. 

To illustrate this we deleted a node on the public instance and then tried to deactivate it on the author instance. 

The deletion and attempted deactivation resulted in the following error:

ERROR lia.rabbitmq.activation.jobs.ActivationConsumerJob: /bla

The exception is visible in the Public Monitoring app. You can see from the PathNotFoundException that the consumer did not find the path that was deleted directly of the public instance.

In the RabbitMQ console the:

• Message is not consumed in the queue.
• Message is marked unacked.
• Consumer is gone,

When more nodes are published into the queue they become blocked behind the unacked message.

There are two ways to solve this:

1. Recreate the node on the public instance and restart the consumer.
   •  The sequence number count remains correct. 
   •  You have to connect to the public instance and do "manual manipulation".
2. Remove the faulty activation message from the queue and restart the consumer.
   •  This can be done remotely and there is no need to connect "manually" to the public instance. 
   •  Because there is one "missing" message the sequence number is lost.

Here's a step-by-step solution using option 2 (because it demonstrates how to delete an unacked message):

1. Close the channel in the RabbitMQ console. This acks all unacked messages and is needed because messages can no longer be consumed.
    
2. Force close the connection
    
3. All messages are now acked in the queue.
    
4. Consume only the "first" message.
   

   

   

5. Restart the consumer on the public instance.

   curl http://localhost:8081/.rest/rbmqClients/restartAll

6. The instance is getting activated again.