Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork9.7k
[Messenger] base64_encoding inside PhpSerializer to avoid null characters#30814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
fabpot approved these changesApr 1, 2019
Member
fabpot left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The only downside is obfuscation of the message that is not as readable as before, but we probably don't have a choice here.
MemberAuthor
weaverryan commentedApr 1, 2019
That's exactly what I was thinking :/. The messages were "readable" before, now it would require a tool. But I also can't see another way around that. |
Contributor
vincenttouzet commentedApr 1, 2019
I think we don't have another choice here :/ But if we want something readable we can still use the symfony serializer 😉 |
nicolas-grekas approved these changesApr 1, 2019
3 tasks
Member
fabpot commentedApr 1, 2019
Thank you@weaverryan. |
fabpot added a commit that referenced this pull requestApr 1, 2019
…null characters (weaverryan)This PR was merged into the 4.3-dev branch.Discussion----------[Messenger] base64_encoding inside PhpSerializer to avoid null characters| Q | A| ------------- | ---| Branch? | master| Bug fix? | yes| New feature? | no| BC breaks? | no| Deprecations? | no| Tests pass? | yes| Fixed tickets |#30805| License | MIT| Doc PR | not neededHi!As pointed out in#30805, the `PhpSerializer` creates strings with null bytes. This apparently causes problems on at least some database systems (I didn't notice, but@vincenttouzet did). I also read that, for example, SQS doesn't like null characters. And, in general, because we're sending this data over a transport, `base64_encoding` data is pretty standard.Does anyone see any downsides?Cheers!Commits-------fe7ad81 base64_encoding inside PhpSerializer to avoid null characters
symfony-splitter pushed a commit to symfony/messenger that referenced this pull requestApr 1, 2019
This PR was merged into the 4.3-dev branch.Discussion----------[Messenger] bug fixes in Doctrine Transport| Q | A| ------------- | ---| Branch? | master| Bug fix? | yes| New feature? | no| BC breaks? | no| Deprecations? | no| Tests pass? | yes| Fixed tickets || License | MIT| Doc PR |Just tested the new Doctrine transport and I've see 3 bugs so far :- [x] The message is not return by the transport- [x] The headers column must be of type TEXT and not just STRING- [ ] When using the PhpSerializer the message is truncated (PR:symfony/symfony#30814)The body in database looks like this :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"```The body given by the serializer is the following :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}}```Commits-------27466498d0 [Messenger] Fix get in Doctrine Transport
fabpot added a commit to fabpot/symfony that referenced this pull requestApr 1, 2019
…ttouzet)This PR was merged into the 4.3-dev branch.Discussion----------[Messenger] bug fixes in Doctrine Transport| Q | A| ------------- | ---| Branch? | master| Bug fix? | yes| New feature? | no| BC breaks? | no| Deprecations? | no| Tests pass? | yes| Fixed tickets || License | MIT| Doc PR |Just tested the new Doctrine transport and I've see 3 bugs so far :- [x] The message is not return by the transport- [x] The headers column must be of type TEXT and not just STRING- [ ] When using the PhpSerializer the message is truncated (PR:symfony#30814)The body in database looks like this :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"```The body given by the serializer is the following :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}}```Commits-------2746649 [Messenger] Fix get in Doctrine Transport
This was referencedApr 2, 2019
fabpot added a commit that referenced this pull requestApr 15, 2019
…verryan)This PR was merged into the 4.3-dev branch.Discussion----------[Messenger] Remove base64_encode & use addslashes| Q | A| ------------- | ---| Branch? | master| Bug fix? | no| New feature? | yes| BC breaks? | no| Deprecations? | no| Tests pass? | yes| Fixed tickets | none| License | MIT| Doc PR | already covered by existing issueIn#30814, we base64_encoded messages because some transports (specifically DoctrineTransport + Postgresql & SQS) do not allow binary data.The downside is that the messages become unreadable, which makes it much less convenient to debug your messages with 3rd party monitoring tools, for example.This PR replaces base64_encode with addslashes. Another alternative (that I first tried in this PR) was to use a blob type, which Drupal does in its code (https://www.drupal.org/project/drupal/issues/690746). But, it still meant that binary data could cause problems with other transports, like SQS.I also put all the serializer config under a nice, neat `serializer` key under messenger.Best seen with `?w=1`.Cheers!Commits-------70b448d Reorganizing messenger serializer config and replacing base64_encode with addslashes
symfony-splitter pushed a commit to symfony/messenger that referenced this pull requestJan 28, 2020
This PR was merged into the 4.3-dev branch.Discussion----------[Messenger] bug fixes in Doctrine Transport| Q | A| ------------- | ---| Branch? | master| Bug fix? | yes| New feature? | no| BC breaks? | no| Deprecations? | no| Tests pass? | yes| Fixed tickets || License | MIT| Doc PR |Just tested the new Doctrine transport and I've see 3 bugs so far :- [x] The message is not return by the transport- [x] The headers column must be of type TEXT and not just STRING- [ ] When using the PhpSerializer the message is truncated (PR:symfony/symfony#30814)The body in database looks like this :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"```The body given by the serializer is the following :```O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}}```Commits-------27466498d0 [Messenger] Fix get in Doctrine Transport
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi!
As pointed out in#30805, the
PhpSerializercreates strings with null bytes. This apparently causes problems on at least some database systems (I didn't notice, but@vincenttouzet did). I also read that, for example, SQS doesn't like null characters. And, in general, because we're sending this data over a transport,base64_encodingdata is pretty standard.Does anyone see any downsides?
Cheers!