Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-118761: Improve import time ofsqlite3#131796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
Wulian233 wants to merge1 commit intopython:main
base:main
Choose a base branch
Loading
fromWulian233:sqlite

Conversation

Wulian233
Copy link
Contributor

@Wulian233Wulian233 commentedMar 27, 2025
edited
Loading

Improve import time ofsqlite3 by around 2 times

Benchmark on Windows10, Python official 3.13.1

D:\Python313>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'"Benchmark 1: ./python -c 'from sqlite3 import *'  Time (mean ± σ):     697.1 µs ± 1059.7 µs    [User: 2577.5 µs, System: 3063.5 µs]  Range (min … max):     0.0 µs … 10960.7 µs    391 runsBenchmark 2: ./python -c 'from sqlite3_new import *'  Time (mean ± σ):     635.0 µs ± 755.4 µs    [User: 1821.0 µs, System: 3691.3 µs]  Range (min … max):     0.0 µs … 3622.5 µs    399 runsSummary  ./python -c 'from sqlite3_new import *' ran    1.10 ± 2.12 times faster than ./python -c 'from sqlite3 import *'

I just realized today that I accidentally deleted an unmerged branch last month, which resulted in closing#129118 . My apologies for the oversight. I've now recreated the pull request - the content remains exactly the same as before

@eendebakpt
Copy link
Contributor

@Wulian233 In your benchmark thesqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Alsotime is a builtin module. Is that one really a bottleneck?

erlend-aasland reacted with thumbs up emoji

Comment on lines 23 to +25
import time
import collections.abc
import datetime
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

One of these things doesn't belong ;-)

Suggested change
importtime
importcollections.abc
importdatetime
importcollections.abc
importdatetime

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

A small import sort, which has been approved before :)
#129118

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Sorry, I should have been more explicit. I was referring toimport time that's still present - I'm assuming you want to remove the module level import (as you did in#129118)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It's particularly noticeable since the continued presence ofimport time makes this "sort the imports" stand out like a sore thumb due to not being sorted in the end.

@brianschubert
Copy link
Contributor

brianschubert commentedMar 27, 2025
edited
Loading

I'm also a bit surprised thattime would be a bottleneck. When I run./python -X importtime -c 'import sqlite3' on my system, I get

import time: self [us] | cumulative | imported package[...]import time:        80 |         80 |   time[...]import time:      2011 |       8096 | sqlite3

Which, if I'm reading it right, says that importingtime can only account for only a small fractionsqlite3's import time?

@Wulian233
Copy link
ContributorAuthor

Wulian233 commentedMar 27, 2025
edited
Loading

@Wulian233 In your benchmark thesqlite3_new is slower (takes more time to import). Based on the name I would expect it to be the faster one.

Alsotime is a builtin module. Is that one really a bottleneck?

I think I got the name wrong at the time, it was a copy of the previous PR content, and here is the test I just run

D:\Python313>hyperfine -i --warmup 8 "./python -c 'from sqlite3 import *'" "./python -c 'from sqlite3_new import *'"Benchmark 1: ./python -c 'from sqlite3 import *'  Time (mean ± σ):     697.1 µs ± 1059.7 µs    [User: 2577.5 µs, System: 3063.5 µs]  Range (min … max):     0.0 µs … 10960.7 µs    391 runs  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.  Warning: Ignoring non-zero exit code.  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.Benchmark 2: ./python -c 'from sqlite3_new import *'  Time (mean ± σ):     635.0 µs ± 755.4 µs    [User: 1821.0 µs, System: 3691.3 µs]  Range (min … max):     0.0 µs … 3622.5 µs    399 runs  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.  Warning: Ignoring non-zero exit code.Summary  ./python -c 'from sqlite3_new import *' ran    1.10 ± 2.12 times faster than ./python -c 'from sqlite3 import *'

@@ -20,9 +20,9 @@
# misrepresented as being the original software.
# 3. This notice may not be removed or altered from any source distribution.

import datetime
import time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What's the point of the PR if the import is not removed? Lazy imports are therefore not used, then why should any speedup be expected, a slowdown is more likely?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Already pointed out in#131796 (comment) :)

(But the PR author self-reviewed the review comment by declaring the review comment to be resolved, which is a controversial feature for github to even allow.)

@picnixz
Copy link
Member

picnixz commentedMar 29, 2025
edited
Loading

To be precise, it's not importingsqlite3 that is faster, it's importingsqlite3and loading its content in the global namespace (the benchmarks are done forfrom sqlite3 import * not for a plainimport sqlite3 statement).

Also, the benchmarks are quite unstable. The standard deviation is much higher than the mean itself! In practice, I think we don't gain anything else than stability (the stdev becomes smaller with the new implementation but again this could be noise).

Can you check ifimport sqlite3 has less noise?


And to be precise, we're not gaining a 2x speed-up. On average, we're only gaining a 1.1x speed-up and the standard deviation of this speed-up is$\pm2$, which is again not really indicative =/

And yes, I'm also surprised that we're gaining "that much" when we just makeimport time local. Considering it's a built-in module (though not sure if it's present at startup), I don't think we need to do this change (we're maybe gaining a small speed-up but maybe not much; and sqlite3 is already is a heavy module to import). So benchmarks using-X importtime here are much more important IMO compared to interpreter's startup as well.

erlend-aasland reacted with thumbs up emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@eli-schwartzeli-schwartzeli-schwartz left review comments

@brianschubertbrianschubertbrianschubert left review comments

@StanFromIrelandStanFromIrelandStanFromIreland left review comments

@berkerpeksagberkerpeksagAwaiting requested review from berkerpeksagberkerpeksag is a code owner

@erlend-aaslanderlend-aaslandAwaiting requested review from erlend-aaslanderlend-aasland is a code owner

Assignees
No one assigned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

7 participants
@Wulian233@eendebakpt@brianschubert@picnixz@eli-schwartz@StanFromIreland@ZeroIntensity

[8]ページ先頭

©2009-2025 Movatter.jp