Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

adding benchmarking, currently only for http funcs#247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
albertedwardson wants to merge3 commits intoscrapy:master
base:master
Choose a base branch
Loading
fromalbertedwardson:pytest-benchmark

Conversation

@albertedwardson
Copy link
Contributor

@albertedwardsonalbertedwardson commentedJul 27, 2025•
edited
Loading

running

pytest --codspeed --codspeed-warmup-time=1 --codspeed-max-rounds=10000 --codspeed-max-time=10

onmicroopts branch

                                                   Benchmark Results                                                   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓┃                                                        Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃     Iters ┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩│          TestBenchmarkHttp::test_bench_dict_to_raw[long_headers] │         2ns │       15.3% │    9.54s │ 6,960,000 ││   TestBenchmarkHttp::test_bench_dict_to_raw[many_unique_headers] │     2,765ns │        1.4% │   10.00s │   190,000 ││ TestBenchmarkHttp::test_bench_dict_to_raw[many_repeated_headers] │       917ns │        3.6% │   10.00s │   330,000 ││          TestBenchmarkHttp::test_bench_raw_to_dict[long_headers] │         2ns │        6.5% │    9.36s │ 6,360,000 ││   TestBenchmarkHttp::test_bench_raw_to_dict[many_unique_headers] │     1,303ns │       10.4% │   10.00s │   280,000 ││ TestBenchmarkHttp::test_bench_raw_to_dict[many_repeated_headers] │     1,120ns │        1.5% │   10.00s │   300,000 │└──────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────────┘

onmicroopts branch commit789e5de

                                                   Benchmark Results                                                   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓┃                                                        Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃     Iters ┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩│          TestBenchmarkHttp::test_bench_dict_to_raw[long_headers] │         3ns │        4.4% │    9.36s │ 5,590,000 ││   TestBenchmarkHttp::test_bench_dict_to_raw[many_unique_headers] │     1,743ns │        1.4% │   10.00s │   240,000 ││ TestBenchmarkHttp::test_bench_dict_to_raw[many_repeated_headers] │       441ns │        8.1% │    9.91s │   470,000 ││          TestBenchmarkHttp::test_bench_raw_to_dict[long_headers] │         2ns │        4.3% │    9.59s │ 6,530,000 ││   TestBenchmarkHttp::test_bench_raw_to_dict[many_unique_headers] │     1,298ns │        2.8% │   10.00s │   280,000 ││ TestBenchmarkHttp::test_bench_raw_to_dict[many_repeated_headers] │     1,177ns │        1.7% │   10.00s │   290,000 │└──────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────────┘

onmain

                                                   Benchmark Results                                                   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓┃                                                        Benchmark ┃ Time (best) ┃ Rel. StdDev ┃ Run time ┃     Iters ┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩│          TestBenchmarkHttp::test_bench_dict_to_raw[long_headers] │         4ns │        6.5% │    9.50s │ 4,810,000 ││   TestBenchmarkHttp::test_bench_dict_to_raw[many_unique_headers] │     3,580ns │        7.2% │   10.00s │   170,000 ││ TestBenchmarkHttp::test_bench_dict_to_raw[many_repeated_headers] │       251ns │       11.9% │   10.00s │   630,000 ││          TestBenchmarkHttp::test_bench_raw_to_dict[long_headers] │         8ns │        3.7% │    9.67s │ 3,420,000 ││   TestBenchmarkHttp::test_bench_raw_to_dict[many_unique_headers] │     1,918ns │        7.0% │   10.00s │   230,000 ││ TestBenchmarkHttp::test_bench_raw_to_dict[many_repeated_headers] │     1,595ns │        7.7% │   10.00s │   250,000 │└──────────────────────────────────────────────────────────────────┴─────────────┴─────────────┴──────────┴───────────┘

headers_dict_to_raw performance regressed inmany_repeated_headers (e.g. cookies) case

I believe it`s cause of using bytes leads to frequent object recreation and copying, which is more expensive than growing a bytearray, especially when building larger payloads.

butheaders_raw_to_dict shows improvements across all cases.

given that real-world headers are rarely tiny, I plan to stick with bytearray for now, as it provides more consistent performance under realistic conditions.

albertedwardson added a commit to albertedwardson/w3lib that referenced this pull requestJul 27, 2025
@codecov
Copy link

codecovbot commentedJul 27, 2025•
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.96%. Comparing base (f45e3ff) to head (d8f400b).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@##           master     #247   +/-   ##=======================================  Coverage   97.96%   97.96%           =======================================  Files           9        9             Lines         491      491             Branches       83       83           =======================================  Hits          481      481             Misses          6        6             Partials        4        4
🚀 New features to boost your workflow:
  • ❄️Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@albertedwardson
[8]ページ先頭
©2009-2025 Movatter.jp