Community Support Policy

• I have readRabbitMQ's Community Support Policy

RabbitMQ version used

4.2.x

How is RabbitMQ deployed?

Other

Steps to reproduce the behavior in question

When a cluster member is lost due to some hardware failures,rabbitmqctl forget_cluster_node (or other QQ shrink actions) can work through the set of QQs slowly. A hardware failure may cause a node to 'disappear' according to the other nodes in the cluster. Forgetting the lost member can take a long time per-queue.forget_cluster_node is an easy way to reproduce this but not the only one:rabbit_quorum_queue:shrink_all/1 is also used by peer discovery cleanup (opt-in feature) and QQCMR has a similar method of working through the queues withrabbit_quorum_queue:delete_member/2.

Reproduction

I have three EC2 nodes withservice tags set torabbitmq sharing an Erlang cookie and using this config:

# ~/config.confcluster_formation.peer_discovery_backend = awscluster_formation.aws.region = <region>cluster_formation.aws.access_key_id = <access key>cluster_formation.aws.secret_key = <secret key>cluster_formation.aws.instance_tags.service = rabbitmq

Then on each node I runmake RABBITMQ_CONFIG_FILE=~/config.conf run-broker usingmain and OTP 27. Then declare a large number of queues:perf-test -qq -qpf 1 -qpt 1000 -qp qq-%d -x 1 -y 0 --time 1.

Then we simulate a hardware failure where a node effectively becomes partitioned network-wise using IP-tables.

# block traffic to/from the other brokers (node C blocking A and B here):A=172.31.30.253B=172.31.17.251iptables -A INPUT -s$A -j DROP&& iptables -A OUTPUT -d$A -j DROP&& \    iptables -A INPUT -s$B -j DROP&& iptables -A OUTPUT -d$B -j DROP

If there are any QQs with leaders on that node, the majority-side of the cluster will soon-after take leadership. Then nodes A and B recognize that C is unreachable:

2025-12-03 16:12:29.936467+00:00 [error] <0.382.0> ** Node 'rabbit@ip-172-31-26-76' not responding **2025-12-03 16:12:29.936467+00:00 [error] <0.382.0> ** Removing (timedout) connection **2025-12-03 16:12:29.936467+00:00 [error] <0.382.0> 2025-12-03 16:12:29.939262+00:00 [info] <0.514.0> rabbit on node 'rabbit@ip-172-31-26-76' down2025-12-03 16:12:29.941314+00:00 [info] <0.514.0> node 'rabbit@ip-172-31-26-76' down: net_tick_timeout

Now if we runrabbitmqctl forget_cluster_node rabbit@ip-172-31-26-76 from A or B, we can see that C is removed from each QQ rather slowly:

2025-12-03 16:13:16.958658+00:00 [info] <0.114280.0> Will remove all queues from node rabbit@ip-172-31-26-76. The node is likely being removed from the cluster.2025-12-03 16:13:16.960172+00:00 [info] <0.119227.0> Asked to remove all quorum queue replicas from node rabbit@ip-172-31-26-762025-12-03 16:13:16.961481+00:00 [info] <0.119227.0> queue 'qq-71' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:23.965503+00:00 [info] <0.119227.0> queue 'qq-335' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:30.970583+00:00 [info] <0.119227.0> queue 'qq-476' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:37.976568+00:00 [info] <0.119227.0> queue 'qq-411' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:44.981565+00:00 [info] <0.119227.0> queue 'qq-610' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:51.986743+00:00 [info] <0.119227.0> queue 'qq-736' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:13:58.991849+00:00 [info] <0.119227.0> queue 'qq-566' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'2025-12-03 16:14:05.996656+00:00 [info] <0.119227.0> queue 'qq-244' in vhost '/': removing member (replica) on node 'rabbit@ip-172-31-26-76'

Note the seven seconds between each attempted queue membership removal. Leaving this test to run overnight, it takes around 1hr57min to finish shrinking the member off of the 1000 queues.

Analysis

rabbit_quorum_queue:delete_member/2 has three potentially expensive components:

1. ra:remove_member/3. This acts against the (maybe newly elected) leader, though, and is usually very quick.
2. rabbit_amqqueue:update/2 to remove the node from themembers list in the queue type state. This is also quick since the first step offorget_cluster_node is to remove the node from the metadata-store membership.
3. ra:force_delete_server/2 whenra:remove_member/3 succeeds. This is where the seven seconds come from.

ra:force_delete_server/2 ultimately callsra_server_sup_sup:stop_server/2 which performs anrpc:call/4 to the failed node. Because of the node disappeared at the network level, Erlang forgets about its distribution table entry and attempts to form a new connection to the node. By default the connect timeout isseven seconds innet_kernel, so this repeatedly waits for up to seven seconds if the destination is not reachable and is not responding. Shrinking is usually very very fast, completing in single-digit minute times even for thousands of queues, as long as the lost member hung up the network connection gracefully. But when it disappears from net_tick_timeout rather than connection_down, shrinking is slow.

This situation is benign since all QQs are have a quorum of active members. If there are other membership changes during this long window, like a new instance joining, the quorum can become threatened since the membership increases to four nodes for any QQs waiting to have the original node removed. If any instance then fails, the membership would drop to 2/4 active members on some QQs and progress on those QQs would be blocked.