Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork9.6k
Description
Symfony version(s) affected
6.x, 5.x
Description
Everytime a message is retried, the DoctrineSender adds anotherTransportMessageIdStamp with the new id to the envelope. This id (the latest id) is needed for thebin/console messenger:failed:show command.
If a message is retried many times, the new id is appended as a new stamp and the stamps get more and more, which leads to a bigger and bigger serialized envelope in the database. Ultimately this can lead to very big binlogs in MySQL.
This is especially noticeable, when you need to check a condition every second and therefore use the Messenger component. If that condition is not yet met, the message is retried by throwing aRecoverableMessageHandlingException inside a transport with a retry multiplier of 1 (see "How to reproduce").
How to reproduce
Example uses Symfony 6.2:
# config/packages/messenger.yamlframework:messenger:transports:check:dsn:'doctrine://default?queue_name=check'retry_strategy:delay:1000multiplier:1
# src/Message/CheckMessage.phpnamespaceApp\Message;class CheckMessage{publicfunction__construct() { }}
# src/Handler/CheckHandler.phpnamespaceApp\Handler;useApp\Message\CheckMessage;useSymfony\Component\Messenger\Attribute\AsMessageHandler;useSymfony\Component\Messenger\Exception\RecoverableMessageHandlingException;#[AsMessageHandler]class CheckHandler{publicfunction__invoke(CheckMessage$message): never {thrownewRecoverableMessageHandlingException(); }}
# src/Command/QueueCommand.phpnamespaceApp\Command;useApp\Message\CheckMessage;useSymfony\Component\Console\Attribute\AsCommand;useSymfony\Component\Console\Command\Command;useSymfony\Component\Console\Input\InputInterface;useSymfony\Component\Console\Output\OutputInterface;useSymfony\Component\Messenger\MessageBusInterface;#[AsCommand(name:'app:check:queue')]class QueueCommandextends Command{publicfunction__construct(privatereadonlyMessageBusInterface$bus, ) {parent::__construct(); }protectedfunctionexecute(InputInterface$input,OutputInterface$output):int {$this->bus->dispatch(newCheckMessage());return Command::SUCCESS; }}
- Insert a message into the queue:
bin/console app:check:queue- Handle this message 100 times:
bin/console messenger:consume check -l 100- Open the
bodyof the message from themessenger_messagestable. You see 100TransportMessageIdStamps:
(...) s:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\";a:100:{i:0;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311314;}i:1;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311315;}i:2;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311316;}i:3;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311317;}i:4;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311318;}i:5;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311319;}i:6;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311320;}i:7;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311321;}i:8;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311322;}i:9;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311323;}i:10;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311324;}i:11;O:57:\"Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\":1:{s:61:\"\0Symfony\\Component\\Messenger\\Stamp\\TransportMessageIdStamp\0id\";i:13311325;}(...)Possible Solution
As far as I can see, the id from theTransportMessageIdStamp is only used for the messenger:failed:show/retry/remove commands to be able to show/retry/remove specific messages.
For this command, theDoctrineReceiver is used and always adds its ownTransportMessageIdStamp with the current id to the envelope. The stamp from the received message from the database is not used at all.
It should be enough to only keep the latestTransportMessageIdStamp and remove all previousTransportMessageIdStamps inside theDoctrineSender andDoctrineReceiver.
Additional Context
No response