Distributed RabbitMQ brokers
Federation
Federation is a way to connect queues and exchanges, which can reside on different brokers. It's a one-way type of connection - we define an "upstream", and that upstream defines where messages come from. Federation is set up on "destination" broker, nothing is configured on the source.
Federated exchanges
If two exchanges are federated, copies of messages are forwarded from source to destination. It's very similar to binding, except that destination exchange can be on a different broker. Behind the scene, this is supported by "upstream queue" on source broker - it's a queue which keeps messages which are not delivered yet. You'll recognize it by "federation:" prefix, and its name tells us for which destination it is.
We can use QueueExplorer to see or modify messages which are waiting for delivery. RabbitMQ automatically creates "federated" exchange, also on source broker:
Federated exchanges are fairly complex to set up. First, you have to define upstream, and then you have to create a policy which will apply federation. You can see more details about federated exchanges here:
https://www.rabbitmq.com/federation.html
https://www.rabbitmq.com/federated-exchanges.html
Federated queues
Federated queues are a bit different - destination queue acts as a consumer for source queue. When a message is consumed on a destination machine, it gets removed from source queue and no other consumer will see it. Therefore each message is processed only once, either by a consumer on a source machine, or by some of federated destination queues (there can be more that one), whichever receives it first. That is different from Federated exchanges, where messages are duplicated and all receivers receive all messages.
Effectively, all federated queues act as a single logical queue, with multiple receivers on multiple machines. So federated queues can be used for load balancing.
http://www.rabbitmq.com/federated-queues.html
Shovel
Shovel is a plugin which moves messages from one place to another. It's like a client which we could write (but don't have to), which receives messages from a queue and sends them to an exchange, including handling failures in that process. Queue and exchange can be on same or different virtual host, or on another broker. It can be set up on independent node, and configured however we like. Problem with shovel was that we had to restart a broker where it runs in order to reconfigure it, but dynamic shovels removed that limitation.
https://www.rabbitmq.com/shovel.html
Clustering
On RabbitMQ cluster is geared toward increasing scalability, not high availability. All nodes in a cluster work all the time, and load is distributed, which improves performance. Definitions (e.g. queue/exchange names/parameters, users, policies, etc) are distributed on all nodes. However, by default, queue messages reside on just one of the nodes. Such queue is not fault tolerant. If node fails, queues from that node won't be available anymore.
In order to get high availability, RabbitMQ has a feature called queue mirroring, which can be turned on after cluster is created. With mirroring turned on, messages will be synced to other nodes. However, they'll still be consumed only from their primary location (i.e. master queue) and other nodes will just maintain their own copy of queue. When node fails, one of mirrored nodes takes over.
Problem is if there are communication issues between nodes. They may end up thinking that other node(s) are down, and continue separately. If messages are not produced/consumed in the exactly same way on disconnected nodes, they'll end up with conflicting data. That state is known as a network partition. In these cases RabbitMQ cluster requires manual intervention after failures, although there are few auto modes where cluster would try to fix problems itself.
https://www.rabbitmq.com/clustering.html
QueueExplorer: Explorer-like management for RabbitMQ
We developed a software which greatly helps you with RabbitMQ. It helps you to understand and manage your system, figure out what went wrong, and fix it. There is a free trial as well.