Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[RateLimiter] Always store SlidingWindows with an expiration set#44996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
fabpot merged 1 commit intosymfony:5.4fromSeldaek:patch-21
Mar 29, 2022

Conversation

@Seldaek
Copy link
Member

@SeldaekSeldaek commentedJan 12, 2022
edited
Loading

QA
Branch?5.4
Bug fix?yes
New feature?no
Deprecations?no
TicketsRefs#44995
LicenseMIT
Doc PRsymfony/symfony-docs#...

See#44995 for more details. The short story is that this code causes the cache item to be overwritten without TTL in

$cacheItem =$this->pool->getItem(sha1($limiterState->getId()));
$cacheItem->set($limiterState);
if (null !== ($expireAfter =$limiterState->getExpirationTime())) {
$cacheItem->expiresAfter($expireAfter);
}
$this->pool->save($cacheItem);

So our redis DB is filling up with rate limiter data, which is not the expected outcome. I couldn't really figure out why the logic was the way it was so it seems safe for me to remove this, but maybe I missed something.

As a related side note, implementing __serialize/__unserialize in SlidingWindow and co would be good as it would reduce the storage requirements in the cache from 400 bytes to 100 bytes per rate limiter stored. I'm happy to help with that just let me know, I am not sure if there was a good reason not to implement this or if storage size was simply not a consideration. I see __sleep is implemented but that does not seem to help here, not sure what the purpose is/was.

@stof
Copy link
Member

I see __sleep is implemented but that does not seem to help here, not sure what the purpose is/was.

I think the purpose was precisely to reduce the storage size

@Seldaek
Copy link
MemberAuthor

It does not seem to have any effect though, so I think it was mostly there to exclude thecached property:https://3v4l.org/nHuAQ

FYI here is the full output with igbinary:

string 'O:3:"Noo":2:{s:6:"�*�bar";s:7:"sdfpods";s:7:"�*�bloo";s:8:"blkdsjds";}' (length=70)string 'O:3:"Foo":2:{s:6:"�*�bar";s:7:"sdfpods";s:7:"�*�bloo";s:8:"blkdsjds";}' (length=70)string 'O:3:"Bar":2:{i:0;s:7:"sdfpods";i:1;s:8:"blkdsjds";}' (length=51)string '���STXETBETXNooDC4STXDC1ACK�*�barDC1BELsdfpodsDC1BEL�*�blooDC1BSblkdsjds' (length=47)string '���STXETBETXFooDC4STXDC1ACK�*�barDC1BELsdfpodsDC1BEL�*�blooDC1BSblkdsjds' (length=47)string '���STXETBETXBarDC4STXACK�DC1BELsdfpodsACKSOHDC1BSblkdsjds' (length=34)

You can see it is quite a bit shorter with __serialize, but it's much more with the RateLimiter classes because somehow the serialized output includes the property names with full class name every time, it's extremely wasteful:

One example from my prod redis server:\x00\x00\x00\x02\x172Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x14\x05\x116\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00id\x11\x1clogin_per_ip-000.000.214.194\x11<\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00hitCount\x06\x01\x11E\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00intervalInSeconds\x06<\x11I\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00hitCountForLastWindow\x06\x00\x11?\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00windowEndAt\na\xde\xc1\xfa

@Seldaek
Copy link
MemberAuthor

In any case, I think we need@wouterj or@Nyholm to shine some light on this :)

@Nyholm
Copy link
Member

Oh no, you are very correct. This is indeed a bug. I’m not sure about your solution. I think there is a good reason why we didn’t set a new expectation time, Ie we don’t want to move the window. (Yes, it is “sliding” but we need a clear start and end of each window which is defined by the cache item)

I need to give this a second look asap to confirm. (I’m currently traveling and my computer broke down)

@Seldaek
Copy link
MemberAuthor

Seldaek commentedMar 1, 2022
edited
Loading

@Nyholm ping here? I'd be happy to get this resolved at some point, as our redis keeps filling up right now :)

Thanks to the bug right now, the cache item doesn't expire ever if you hit more than once.

So either the way I see it, either:

  • you hit once, then it disappears after 2x $intervalsInSeconds (SlidingWindow's interval), then you come back and you hit and it's a fresh SlidingWindow
  • you hit it twice, then if you hit a third time it is reloaded from cache, you are still in the active window and so things work using that existing one
  • you hit it twice, so it stays in the cache, but you hit a third time after the sliding window expired, then I assume createFromPreviousWindow is called to create a new one, which is correct behavior too, but if it was cleared from the cache because you come later than 2x $intervalsInSeconds it would simply create a blank one and that'd be correct too.

So the fix seems correct to me (it would only change the last bit of the last bullet point above), but maybe I misunderstood something of course.

@SeldaekSeldaek changed the base branch from5.3 to5.4March 16, 2022 15:40
@Seldaek
Copy link
MemberAuthor

Rebased/retargetted to 5.4 as 5.3 is now EOL

@Nyholm
Copy link
Member

Please let me delay this week too. I have an open source session planned on Sunday. If I fail to come back then just ignore me.

Sorry for being unavailable.

chalasr reacted with thumbs up emoji

@fabpot
Copy link
Member

Any news here?

Copy link
Member

@wouterjwouterj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks!

It does not seem to have any effect though, so I think it was mostly there to exclude the cached property

Indeed, I don't think storage size was considered here.

@nicolas-grekasnicolas-grekasforce-pushed thepatch-21 branch 2 times, most recently from5d96d1d to5336906CompareMarch 28, 2022 15:12
@fabpot
Copy link
Member

Thank you@Seldaek.

@fabpotfabpot merged commit5dd5dd2 intosymfony:5.4Mar 29, 2022
@SeldaekSeldaek deleted the patch-21 branchMarch 29, 2022 09:05
@Seldaek
Copy link
MemberAuthor

I tested the patch with the latest changes from@nicolas-grekas and the good news is it does work well doing:

"SETEX" "lIoZODSNyI:921c2aa99364ab8bbb0e4525c4297aa71c61d2c6" "119" "\x00\x00\x00\x02\x172Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x14\x01\x117\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00<login_per_ip_username-sdfsdf@fdjk-127.0.0.1\x0cA\xd8\x90\xb3\x9e\x10\x0es"

instead of:

"SET" "lIoZODSNyI:921c2aa99364ab8bbb0e4525c4297aa71c61d2c6" "\x00\x00\x00\x02\x172Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x14\x05\x116\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00id\x11\x16login_per_ip-127.0.0.1\x11<\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00hitCount\x06\x02\x11E\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00intervalInSeconds\x06<\x11I\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00hitCountForLastWindow\x06\x00\x11?\x00Symfony\\Component\\RateLimiter\\Policy\\SlidingWindow\x00windowEndAt\x0cA\xd8\x90\xb3#^z\xcc"

So much more efficient storage, and SETEX instead of SET 👍🏻

Bad news is the migration path is a bit broken (like you get a fatal error..) if it reloads existing data which was serialized with __sleep and tries to unserialize using the new __unserialize. I'll send a follow-up PR.

wouterj reacted with thumbs up emoji

@wouterj
Copy link
Member

thanks for the quick checking in a real app :)

@Seldaek
Copy link
MemberAuthor

Here#45876

nicolas-grekas added a commit that referenced this pull requestMar 29, 2022
…che (Seldaek)This PR was merged into the 5.4 branch.Discussion----------Add BC layer to handle old objects already present in cache| Q             | A| ------------- | ---| Branch?       | 5.4| Bug fix?      | yes| New feature?  | no <!-- please update src/**/CHANGELOG.md files -->| Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->| Tickets       | Refs#44996| License       | MIT| Doc PR        | symfony/symfony-docs#... <!-- required for new features -->When reloading an old RateLimiter instance from cache with the new code I did get an exception because windowEndAt was a string and getHitCount does windowEndAt - intervalInSeconds, and string - int fails.The reason is the new code assigns the id to windowEndAt via:```        $pack = key($data);        $this->windowEndAt = $data[$pack];```I dumped $this and  $data in __unserialize just FYI and you get this:```object(Symfony\Component\RateLimiter\Policy\SlidingWindow)[959]  private 'id' => null  private 'hitCount' => int 0  private 'hitCountForLastWindow' => int 0  private 'intervalInSeconds' => null  private 'windowEndAt' => null``````array (size=5)  '�Symfony\Component\RateLimiter\Policy\SlidingWindow�id' => string 'login_per_ip-127.0.0.1' (length=22)  '�Symfony\Component\RateLimiter\Policy\SlidingWindow�hitCount' => int 2  '�Symfony\Component\RateLimiter\Policy\SlidingWindow�intervalInSeconds' => int 60  '�Symfony\Component\RateLimiter\Policy\SlidingWindow�hitCountForLastWindow' => int 0  '�Symfony\Component\RateLimiter\Policy\SlidingWindow�windowEndAt' => float 1648544909.4762```I only tested the code in real conditions for SlidingWindow, but I do believe/hope the same applies for the other two policies.Commits-------f039ee6 Add BC layer to handle old objects already present in cache
This was referencedApr 2, 2022
@Seldaek
Copy link
MemberAuthor

Confirmed to work in prod now with 6.0.7 🥳 Thanks again everyone.

chalasr reacted with hooray emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@nicolas-grekasnicolas-grekasnicolas-grekas approved these changes

@wouterjwouterjwouterj approved these changes

@chalasrchalasrchalasr approved these changes

Assignees

No one assigned

Projects

None yet

Milestone

5.4

Development

Successfully merging this pull request may close these issues.

8 participants

@Seldaek@stof@Nyholm@fabpot@wouterj@nicolas-grekas@chalasr@carsonbot

[8]ページ先頭

©2009-2025 Movatter.jp