Time-to-Live and Expiration
Key topics covered in this documentation guide are
- An overview of queue TTL and message TTL features supported by RabbitMQ
- Per-queuemessage TTL defined at the queue level
- Per-messageTTL defined by publishers
- Message TTL and dead lettering
- Message TTL applied retroactively
- Queue TTL (expiration of queues)
Time-To-Live Feature
With RabbitMQ, you can set a TTL (time-to-live) argument or policy for messages and queues. As the name suggests, TTL specifies the time period that the messages and queues "live for".
Message TTL determines how long messages can be retained in a queue. If the retention period of a message in a queue exceeds the message TTL of the queue, the message expires and is discarded.
"Discarded" means that the message will not be delivered to any of subscribed consumers and won't be accessible throughbasic.get method applied directly on queue. Message TTL can be applied to a single queue, a group of queues, or applied on the message-by-message basis.
TTL can also be set on queues, not just queue contents. This feature can be used together with the auto-delete queue property. Setting TTL (expiration) on queues generally only makes sense for transient (non-durable) classic queues. Streams do not support expiration.
Queues will expire after a period of time only when they are not used (a queue is used if it has online consumers).
TTL behavior is controlled byoptional queue arguments and the best way to configure it is using apolicy.
TTL settings also can be enforced byoperator policies.
Per-Queue Message TTL in Queues
Message TTL can be set for a given queue by setting themessage-ttl argument with apolicyor by specifying the same argument at the time of queue declaration.
A message that has been in the queue for longer than the configured TTL is said tobeexpired. Note that a message routed to multiple queuescan expire at different times, or not at all, in each queue inwhich it resides. The death of a message in one queue has noimpact on the life of the same message in other queues.
The serverguarantees that expired messages will not be deliveredusingbasic.deliver (to a consumer) or sent in response to a polling consumer(in abasic.get-ok response).
Further, the server will try to remove messages at orshortly after their TTL-based expiry.
The value of the TTL argument or policy must beanon-negative integer (equal to or greater than zero),describing the TTL period in milliseconds.
Thus a value of1000 means that a message added to the queue will live in thequeue for 1 second or until it is delivered to a consumer. Theargument can be of AMQP 0-9-1 typeshort-short-int,short-int,long-int, orlong-long-int.
Define Message TTL for Queues Using a Policy
To specify a TTL using policy, add the key "message-ttl" to apolicy definition:
| rabbitmqctl | |
|---|---|
| rabbitmqctl (Windows) | |
This applies a TTL of 60 seconds to all queues.
Define Message TTL for Queues Using x-arguments During Declaration
This example in Java creates a queue in which messages mayreside for at most 60 seconds:
Map<String,Object> args=newHashMap<String,Object>();
args.put("x-message-ttl",60000);
channel.queueDeclare("myqueue",false,false,false, args);
The same example in C#:
var args=newDictionary<string,object>();
args.Add("x-message-ttl",60000);
model.QueueDeclare("myqueue",false,false,false, args);
It is possible to apply a message TTL policy to a queue which alreadyhas messages in it but this involvessome caveats.
The original expiry time of a message is preserved if itis requeued (for example due to the use of an AMQP methodthat features a requeue parameter, or due to a channelclosure).
Setting the TTL to 0 causes messages to be expired upon reachinga queue unless they can be delivered to a consumerimmediately. Thus this provides an alternative totheimmediate publishing flag, whichthe RabbitMQ server does not support. Unlike that flag, nobasic.returns are issued, and if a dead letterexchange is set then messages will be dead-lettered.
Per-Message TTL in Publishers
A TTL can be specified on a per-message basis, by setting theexpiration property when publishing a message.
The value of theexpiration field describes theTTL period in milliseconds. The same constraints as forx-message-ttl apply. Since theexpiration field must be a string, the brokerwill (only) accept the string representation of the number.
When both a per-queue and a per-message TTL are specified, thelower value between the two will be chosen.
This example usesRabbitMQ Java clientto publish a message which can reside in the queue for at most 60 seconds:
byte[] messageBodyBytes="Hello, world!".getBytes();
AMQP.BasicProperties properties=newAMQP.BasicProperties.Builder()
.expiration("60000")
.build();
channel.basicPublish("my-exchange","routing-key", properties, messageBodyBytes);
The same example in C#:
byte[] messageBodyBytes= System.Text.Encoding.UTF8.GetBytes("Hello, world!");
IBasicProperties props= model.CreateBasicProperties();
props.ContentType="text/plain";
props.DeliveryMode=2;
props.Expiration="60000";
model.BasicPublish(exchangeName,
routingKey, props,
messageBodyBytes);
Per-Message TTL and Dead Lettering
Quorum Queues
Quorum queues dead letter expired messages when they reach the head of the queue.
Classic Queues
Classic queues dead letter expired messages in a few cases:
- When the message reaches the head of the queue
- When the queue is notified of a policy change that affects it
Per-message TTL Applied Retroactively (to an Existing Queue)
Queues that had a per-message TTL applied to themretroactively (when they already had messages) will discardthe messages when specific events occur.
Only when expired messages reach the head of a queue will they actually bediscarded (marked for deletion). Consumers will not haveexpired messages delivered to them. Keep in mind thatthere can be a natural race condition between message expirationand consumer delivery, e.g. a message can expireafter it was written to the socket but before it has reacheda consumer.
When setting per-message TTL expired messages can queue upbehind non-expired ones until the latter are consumed orexpired. Hence resources used by such expired messages willnot be freed, and they will be counted in queue statistics(e.g. the number of messages in the queue).
When retroactively applying a per-message TTL policy, it isrecommended to have consumers online to make sure themessages are discarded quicker.
Given this behaviour of per-message TTL settings on existingqueues, when the need to delete messages to free upresources arises, queue TTL should be used instead (or queuepurging, or queue deletion).
Queue TTL
TTL can also be set on queues, not just queue contents.This feature can be used together with theauto-delete queue property.
Setting TTL (expiration) on queues generally only makes sensefor transient (non-durable) classic queues. Streamsdo not support expiration.
Queues will expire after a period of time only when theyare not used (a queue is used if it has online consumers).
Expiry time can be set for a given queue by setting thex-expires argument toqueue.declare,or by setting theexpirespolicy. This controls forhow long a queue can be unused before it is automaticallydeleted. Unused means the queue has no consumers, thequeue has not been recently redeclared (redeclaring renews the lease),andbasic.get has not been invoked for a duration of at least the expirationperiod. This can be used, for example, for RPC-style replyqueues, where many queues can be created which may never bedrained.
The server guarantees that the queue will be deleted, ifunused for at least the expiration period. No guarantee isgiven as to how promptly the queue will be removed after theexpiration period has elapsed.
The value of thex-expires argument orexpires policy describes the expiration period inmilliseconds. It must be a positive integer (unlike messageTTL it cannot be 0). Thus a value of 1000 means a queue whichis unused for 1 second will be deleted.
Define Queue TTL for Queues Using a Policy
The following policy makes all queues expire after 30 minutes since last use:
| rabbitmqctl | |
|---|---|
| rabbitmqctl (Windows) | |
Define Queue TTL for Queues Using x-arguments During Declaration
This example in Java creates a queue which expires afterit has been unused for 30 minutes.
Map<String,Object> args=newHashMap<String,Object>();
args.put("x-expires",1800000);
channel.queueDeclare("myqueue",false,false,false, args);