Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-95778: Add pre-check for int-to-str conversion#96537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gpshead merged 14 commits intopython:mainfrommdickinson:fix/int-to-str-dos
Sep 4, 2022

Conversation

@mdickinson
Copy link
Member

@mdickinsonmdickinson commentedSep 3, 2022
edited
Loading

On currentmain, converting a large enoughint to a decimal string raisesValueError as expected. However, the raise comesafter the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop.

This PR gives a proof-of-concept quick fix: essentially we catchmost values that exceed the threshold up front. Those that slip through will still be on the small side, and will get caught by the existing check.

For the record, here's the justification for the current check. The C code check is:

max_str_digits / (3*PyLong_SHIFT) <= (size_a-11) /10

In math-speak, writing$M$ formax_str_digits,$L$ forPyLong_SHIFT and$s$ forsize_a, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$

From this it follows that
$$\frac{M}{3L} &lt; \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} &gt; \frac{10}{3} &gt; \log_2(10).$$
So
$$2^{L(s-1)} &gt; 10^M.$$
But our input integer$a$ satisfies$|a| \ge 2^{L(s-1)}$, so$|a|$ is larger than$10^M$. This shows that we don't accidentally capture anythingbelow the intended limit in the check.

I don't think this is ready to merge as-is - there are some details to figure out, and I'll add line-by-line comments for those.

serhiy-storchaka and hujiawei34 reacted with thumbs up emoji
@mdickinson
Copy link
MemberAuthor

mdickinson commentedSep 3, 2022
edited
Loading

To give an idea of how much the crude check misses: assuming 30 bits per digit andint_max_str_digits value at its default of4300, the smallest integer for which the pre-check kicks in is2**(30 * 480), which is about6.79e4334. So values between1e4300 and6.79e4334 will end up getting converted andthen meeting the post-loop check and raisingValueError.


EDIT: Whoops, sorry; the bound is2**(30 * 480), not2**(30 * 481). Edited to fix.

@gpsheadgpshead self-assigned thisSep 4, 2022
@gpsheadgpshead added type-bugAn unexpected behavior, bug, or error type-securityA security issue release-blocker labelsSep 4, 2022
skip rather than fail if we find unexpectedly high performance.
@gpsheadgpshead added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelSep 4, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@gpshead for commit4dae3e0 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelSep 4, 2022
@gpsheadgpshead added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelSep 4, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@gpshead for commitadb1784 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelSep 4, 2022
@gpsheadgpshead merged commitb126196 intopython:mainSep 4, 2022
@miss-islington
Copy link
Contributor

Thanks@mdickinson for the PR, and@gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestSep 4, 2022
…GH-96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:pythongh-95778 -->* Issue:pythongh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>(cherry picked from commitb126196)Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
@bedevere-botbedevere-bot removed the needs backport to 3.11only security fixes labelSep 4, 2022
@bedevere-bot
Copy link

GH-96562 is a backport of this pull request to the3.11 branch.

@miss-islington
Copy link
Contributor

Sorry,@mdickinson and@gpshead, I could not cleanly backport this to3.10 due to a conflict.
Please backport usingcherry_picker on command line.
cherry_picker b126196838bbaf5f4d35120e0e6bcde435b0b480 3.10

gpshead pushed a commit to gpshead/cpython that referenced this pull requestSep 4, 2022
…ythonGH-96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:pythongh-95778 -->* Issue:pythongh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>(cherry picked from commitb126196)Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
@bedevere-botbedevere-bot removed the needs backport to 3.10only security fixes labelSep 4, 2022
@bedevere-bot
Copy link

GH-96563 is a backport of this pull request to the3.10 branch.

gpshead added a commit to gpshead/cpython that referenced this pull requestSep 4, 2022
…#96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:pythongh-95778 -->* Issue:pythongh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
miss-islington added a commit that referenced this pull requestSep 4, 2022
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:gh-95778 -->* Issue:gh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>(cherry picked from commitb126196)Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
gpshead added a commit to gpshead/cpython that referenced this pull requestSep 4, 2022
…#96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:pythongh-95778 -->* Issue:pythongh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
gpshead added a commit to gpshead/cpython that referenced this pull requestSep 4, 2022
…#96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:pythongh-95778 -->* Issue:pythongh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
gpshead added a commit that referenced this pull requestSep 4, 2022
) (#96563)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:gh-95778 -->* Issue:gh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>(cherry picked from commitb126196)Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
ambv pushed a commit that referenced this pull requestSep 5, 2022
* Correctly pre-check for int-to-str conversion (#96537)Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.The justification for the current check. The C code check is:```cmax_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10```In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$From this it follows that$$\frac{M}{3L} < \frac{s-1}{10}$$hence that$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$So$$2^{L(s-1)} > 10^M.$$But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.<!-- gh-issue-number:gh-95778 -->* Issue:gh-95778<!-- /gh-issue-number -->Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
@mdickinsonmdickinson deleted the fix/int-to-str-dos branchSeptember 8, 2022 08:07
facebook-github-bot pushed a commit to facebookincubator/cinder that referenced this pull requestJan 20, 2023
Summary:cherry-picked the upstream 3.10 backport```git cherry-pick8f0fa4beace09e```this one ispython/cpython#96537original commit message below--------Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.This PR comes fresh from a pile of work done in our private PSRT security response team repo.This backportspython/cpython#96499 aka511ca94Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).<!-- gh-issue-number: gh-95778 -->* Issue: gh-95778<!-- /gh-issue-number -->I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#).Reviewed By: alexmalyshevDifferential Revision: D39369518fbshipit-source-id:6ed2def
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadgpshead approved these changes

@tirantiranAwaiting requested review from tiran

Assignees

@gpsheadgpshead

Labels

release-blockerskip newstype-bugAn unexpected behavior, bug, or errortype-securityA security issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mdickinson@bedevere-bot@gpshead@miss-islington

[8]ページ先頭

©2009-2025 Movatter.jp