Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork9.7k
[Messenger] Adding failure transport support#30970
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
e63e80f to3a6080fCompare| * @param bool[] $sendAndHandle | ||
| */ | ||
| publicfunction__construct(array$senders,array$sendAndHandle = []) | ||
| publicfunction__construct(array$sendersMap,ContainerInterface$sendersLocator,array$sendAndHandle = []) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Instead of each sender class having its own service locator object of senders, this$sendersMap is now a simple array of string sender aliases (or service ids, if no alias) and$sendersLocator is a service locator that contains all senders.
src/Symfony/Component/Messenger/Event/WorkerMessageReceivedEvent.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Command/FailedMessagesRetryCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
| * | ||
| * @experimental in 4.3 | ||
| */ | ||
| class FailedMessagesRetryCommandextends Command |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Should we make the commands final?
src/Symfony/Component/Messenger/EventListener/SendFailedMessageToFailedTransportListener.phpShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Transport/Sender/SendersLocator.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Command/AbstractFailedMessagesCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Transport/Receiver/SingleMessageReceiver.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
8e203fc toe151d3cCompare9085c63 to9ccf3bcCompareweaverryan commentedApr 10, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
This is ready! And it worksawesome. Here are some technical notes:
|
c7e613d to16f2b32Comparesrc/Symfony/Component/Messenger/Command/AbstractFailedMessagesCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
16f2b32 to04a985aComparesrc/Symfony/Component/Messenger/Transport/Receiver/ReceiverInterface.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
2c448d8 to905febbCompare
weaverryan left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is ready! I think it's important to get into 4.3. We're beyond feature freeze, but this component is experimental and this contains one more BC break. This also contains various bug fixes, which I've outlined.
| $throwable =$throwable->getNestedExceptions()[0]; | ||
| } | ||
| $flattenedException =\class_exists(FlattenException::class) ? FlattenException::createFromThrowable($throwable) :null; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Using FlattenException was my way of making sure we had an Exception that was serializable. The original exception would be even better (as we already have logic to render this very "pretty" in the console), but I think this is the only safe way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
maybe at least pass file+line info when the Debug component is not found?
or maybe remove this and replace by the throwable cast as string?
| publicstaticfunctiongetSubscribedEvents() | ||
| { | ||
| return [ | ||
| WorkerMessageFailedEvent::class => ['onMessageFailed', -100], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Low priority so that users could handle it first if they wanted.
| $envelope =$envelope->with(newBusNameStamp($this->busName)); | ||
| if (null ===$envelope->last(BusNameStamp::class)) { | ||
| $envelope =$envelope->with(newBusNameStamp($this->busName)); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Fixes an unrelated "bug" where retries would cause many BusNameStamp to be added, which was just unnecessary and wasteful. Test was added for this.
| { | ||
| if (null !==$redeliveryStamp) { | ||
| return [ | ||
| $redeliveryStamp->getSenderClassOrAlias() =>$this->sendersLocator->getSenderByAlias($redeliveryStamp->getSenderClassOrAlias()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
For redelivery, the sender "alias" really becomes more important. This is communicated on the phpdoc ofSendersLocatorInterface::getSenders(), which mentions that the iterator should be keyed by a sender alias, when possible.
| publicfunctionisRetryable(Envelope$message):bool | ||
| { | ||
| if (0 ===$this->maxRetries) { | ||
| if (null ===$this->maxRetries) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Unrelated: changes behavior slightly so that0 can be used to mean "don't retry on this transport". Previously, there was no easy way to say "don't retry" as 0 meant "retry infinitely".
Test added for this.
| publicfunctiongetMessageCount():int | ||
| { | ||
| return ($this->receiver ??$this->getReceiver())->getMessageCount(); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Unrelated: the interface & this method were just missing.
src/Symfony/Component/Messenger/Transport/Receiver/SingleMessageReceiver.phpShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
| @trigger_error(sprintf('The 2nd argument to "%s::__construct()" requires ContainerInterface 2nd argument. Not passing that was deprecated in Symfony 4.3 and will be required in Symfony 5.0.',__CLASS__),E_USER_DEPRECATED); | ||
| $this->sendersLocator =newServiceLocator([]); | ||
| $this->sendAndHandle =$sendersLocator; | ||
| $this->useLegacyLookup =true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
BC added so that FW bundle 4.2 can work with Messenger 4.3 (otherwise cross-version tests fail)
| * @param bool[] $sendAndHandle | ||
| */ | ||
| publicfunction__construct(array$senders,array$sendAndHandle = []) | ||
| publicfunction__construct(array$sendersMap,/*ContainerInterface*/$sendersLocator =null,array$sendAndHandle = []) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is really the "messiest" part of the PR. Because we need to send to the failure transport, identified by its "name", we need a way to ask the system: "please give me the sender namedmy_failed_transport". The newgetSenderByAlias() accomplishes this. And it works great, but it was a "shift" in the system, as - until now - we only ever asked "Give me the senders for this envelope/message".
| "conflict": { | ||
| "symfony/event-dispatcher":"<4.3" | ||
| "symfony/event-dispatcher":"<4.3", | ||
| "symfony/debug":"<4.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Because, IF symfony/debug is available, we need theFlattenException::createFromThrowable() method.
src/Symfony/Component/Messenger/Command/FailedMessagesPurgeCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
weaverryan commentedApr 24, 2019
Ready again! |
chalasr left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Very nice work!
src/Symfony/Bundle/FrameworkBundle/Resources/config/messenger.xml OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Command/FailedMessagesRetryCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
src/Symfony/Component/Messenger/Command/FailedMessagesPurgeCommand.php OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
e6bb425 to1692281Compareweaverryan commentedApr 29, 2019
Thanks for the review@chalasr! This is once again ready to go! |
chalasr left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Retested locally, works well now.
👍 once my last comment addressed
Uh oh!
There was an error while loading.Please reload this page.
1692281 toa8923a9Compareweaverryan commentedApr 30, 2019
Comment addressed! |
a8923a9 todd1edf3Compareweaverryan commentedApr 30, 2019
Test failures are unrelated. Ready for review! |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
dd1edf3 to36487e5Comparefabpot commentedMay 1, 2019
Thank you@weaverryan. |
This PR was squashed before being merged into the 4.3-dev branch (closes#30970).Discussion----------[Messenger] Adding failure transport support| Q | A| ------------- | ---| Branch? | master| Bug fix? | yes| New feature? | yes| BC breaks? | yes| Deprecations? | no| Tests pass? | yes| Fixed tickets |#31231| License | MIT| Doc PR |symfony/symfony-docs#11236This adds "failure" transport support for messenger, so that messages that fail on *all* their retries can be collected in one spot and retried later if wanted:```ymlframework: messenger: failure_transport: failed transports: async: dsn: 'amqp://' failed: dsn: 'doctrine://default?queue_name=failed' routing: 'App\Message\SmsNotification': async```In this setup, `SmsNotification` would be retried 3 times on the `async` transport (current behavior) and then finally sent to the `failed` transport. The `failed` transport can be consumed like a normal transport, but should usually be handled & consumed by one of the new commands:**> bin/console messenger:failed:show**<img width="861" alt="Screen Shot 2019-04-10 at 3 15 45 PM" src="https://user-images.githubusercontent.com/121003/55917329-ddc54280-5ba3-11e9-878c-af3c653643de.png">**> bin/console messenger:failed:show 217**<img width="804" alt="Screen Shot 2019-04-10 at 3 15 55 PM" src="https://user-images.githubusercontent.com/121003/55917360-f33a6c80-5ba3-11e9-9f12-a8c57a9a7a4b.png">**> bin/console messenger:failed:purge 217**<img width="835" alt="Screen Shot 2019-04-10 at 3 16 07 PM" src="https://user-images.githubusercontent.com/121003/55917383-ff262e80-5ba3-11e9-9720-e24176b834f7.png">**> bin/console messenger:failed:retry 217**<img width="737" alt="Screen Shot 2019-04-10 at 3 16 29 PM" src="https://user-images.githubusercontent.com/121003/55917396-09482d00-5ba4-11e9-8d51-0bbe2b4ffc14.png">**> bin/console messenger:failed:retry 218 -vv**<img width="1011" alt="Screen Shot 2019-04-10 at 3 20 39 PM" src="https://user-images.githubusercontent.com/121003/55917503-6512b600-5ba4-11e9-9365-4ac87d858541.png">**Note** (This screenshot is ugly - need to make the dump of the message and the exception more attractive)Or you can run `bin/console messenger:failed:retry` without any argument, and it will consume the failed messages one-by-one and ask you if you want to retry/handle each. By passingCheers!Commits-------36487e5 [Messenger] Adding failure transport support
Uh oh!
There was an error while loading.Please reload this page.
Tobion commentedMay 5, 2019
If I see it correctly, the messenger:failed:show command cannot show messages when rabbitmq is used as a failed transport because it does not implement the listing and getting individual messages. |
weaverryan commentedMay 5, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Hi@Tobion! Yes, it’s something i talked about with Sam, but wasn’t ultimately sure about. The problem is if there aremany messages. Consuming them all, just to list them and put them back could be a big operation. I don’t have any experience in how rabbit behaves in this way. I’d love to hear any experiences related to this. Another consideration is that (from feedback I’ve heard) people could theoretically have many failed messages (due to a 3rd party service failure, for example) and want to retry all of them, or maybe all failed messages within a date range. Retrying all of them would work fine, but doing it by date range would require the “consume all” strategy. Finally, one other unrelated thing from feedback: retry should perhaps just requeue. This is especially important if we offer some sort of “bulk” retry/requeue: requeuing them and allowing their (potentially many) workers on the original queue do their work normally makes more sense then retrying them one-by-one of the server where someone is manually looking at and retrying the failed messages. |
weaverryan commentedMay 5, 2019
Also, generally speaking, I’m not sure if Amqp is a good candidate for the failure transport. The failure transport could get “large” - even in a normal app - if you rarely check it and handle those messages. That could hurt Amqp performance - it doesn’t (as I understand it) like to have queues with many messages. |
Tobion commentedMay 5, 2019
Hey@weaverryan. I love this feature.I think if we provide a way to configure the failed storage, then we should also try our best to make them all feature complete.
I don't think it's very common to have many failed messages after retry. But as a sanity check, we could just do the listing if the amount of messages is below a threshold and ask for confirmation if not.
There are use-cases for both approaches in my opinion. You either just want to retry one and see if it works now or why it doesn't work. Or the queue is full and you don't want to wait. |
weaverryan commentedMay 5, 2019
I'm going to work on a PR for the requeue. When I've open that, let's continue discussing there to make sure we're happy with everything. I also like the "retry" for at least some situations (just retry it right now and show me if it worked!). |
| publicfunctionget():iterable | ||
| { | ||
| if ($this->hasReceived) { | ||
| return []; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
since this method uses a generator, it should not return array. but justreturn;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
actually it does not seem worth to use a generator here, but just return the envolope in an array.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Uh oh!
There was an error while loading.Please reload this page.
This adds "failure" transport support for messenger, so that messages that fail onall their retries can be collected in one spot and retried later if wanted:
In this setup,
SmsNotificationwould be retried 3 times on theasynctransport (current behavior) and then finally sent to thefailedtransport. Thefailedtransport can be consumed like a normal transport, but should usually be handled & consumed by one of the new commands:> bin/console messenger:failed:show

> bin/console messenger:failed:show 217

> bin/console messenger:failed:purge 217

> bin/console messenger:failed:retry 217

> bin/console messenger:failed:retry 218 -vv

Note (This screenshot is ugly - need to make the dump of the message and the exception more attractive)
Or you can run
bin/console messenger:failed:retrywithout any argument, and it will consume the failed messages one-by-one and ask you if you want to retry/handle each. By passingCheers!