I have a branch which adds a filtering logic to the completer (seehttps://discuss.python.org/t/repl-introduce-generic-filters-to-filter-auto-completion-matches/77553) and I wondered whether we could expand the Completer interface in order to have a more generic hook category. Both your work and mine can be orthogonal but we might take this opportunity to make the completer class more generic (I haven't looked at the PR in details as I'm travelling now)

Right, I remember seeing your dpo post before! I think it's a good idea :) With this PR I decided to target PyREPL since it is less stable than rlcompleter and thus it's easier to expand/make changes. Though we could definitely think about unifying the interface with that of rlcompleter. For now the API is a bit ad hoc so I'd welcome some discussion in that direction 🙂

1. On macOS, typeimport re<tab>:

>>>import re[ complete but not unique ]

2. Press<tab> again:

>>>import rere        reprlib   readline  resource  requests

3. Then change your mind and pressctrl+C, and you get:

>>>import reKeyboardInterrupt   readline  resource  requests>>>re        reprlib   readline  resource  requests

I'd expect something more like:

>>> import reKeyboardInterrupt>>>

4. Pressctrl+C a couple more times:

>>>import reKeyboardInterrupt   readline  resource  requests>>>KeyboardInterrupt   readline  resource  requests>>>KeyboardInterrupt   readline  resource  requests>>>re        reprlib   readline  resource  requests

I'd expect something more like:

>>>import reKeyboardInterrupt>>>KeyboardInterrupt>>>KeyboardInterrupt>>>

Answering here what I wrote on Discord - thectrl+C behaviour is present for the existing name/attribute autocomplete as well, this PR is not changing that. I think it's something we could add in a separate PR though!

I only did a quick look at the code. Are you trying to import the module during completion?

I only did a quick look at the code. Are you trying to import the module during completion?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g.import foo.<tab>). IPython doessomething similar. Is it a concern?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g.import foo.<tab>). IPython doessomething similar. Is it a concern?

I think so. Implicitly import a module during auto-completion seems a bit unintentional to me. For example, importingtorch could take 2~3 seconds, and to user it would be a freeze during completion. Some libraries monkeypatch stdlib or even Python during import, and that's some hidden side effect. Personally I don't like this implicit behavior. Also, will the import only triggered if. is in the text? Or for all possible completion? Say I doimport t<tab> does it import all the modules starting with at?

Say I do import t does it import all the modules starting with a t?

This PR usespkgutil.iter_modules so just typingimport t<tab> does not actually import anything. The results ofpkgutil.iter_modules are also cached so the top-level packages are only searched once.

Ido import in case you typeimport torch.x<tab>, but I think I might be able to avoid that.

What I think I cannot avoid is an import in case you typefrom foo import x<tab>. In order for me to know which objects are available for import, I need to actually import the module. I don't think there is a way around that? What do you think?

Personally, I would prefer if the auto-completernever imports any module without my knowledge, even if that means it can't do its job in some cases. It's okay if it completes when the module already imported, but importing a new module during auto-completion seems wrong to me. I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I admit I do not fully understand the implications of this so I think getting more opinions is a good idea :) Do you want me to start a discussion on Discord?

Discord or/and dpo maybe? We should at least get some opinions from repl owners.

ok! It's getting a bit late here, so I'll do it tomorrow 🙂

I updated the PR following the discussion on DPO:https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

• Modules are no longer imported. Modules are discovered using a combination ofpkgutil.iter_modules andsubmodule_search_locations.
• Removed support for module attributes. This relied on actually importing the modules. We can add this once we have decided in which way we should discover module attributes.

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

Thanks for taking the time to review this, I really appreciate it!

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

When it comes to performance, the biggest bottleneck is the call topkgutil.iter_modules() to find all top-level packages.
(the result is cached so we pay the cost only the first time you hit TAB).

Here are some timings for lots of packages. I installed the top 1000 pypi packages (at least those that can be installed on 3.14 so 836 packages installed):

~ pip freeze| wc -l836

The timings are on a laptop with Intel ultra 9 185H CPU and a new-ish SSD (relevant becausepkgutil.iter_modules interacts with the file system).

For those ~800 packages in a single location,pkgutil.iter_modules takes about 0.03s:

>>>importtime,pkgutil>>>start=time.time();list(pkgutil.iter_modules());time.time()-start0.03072071075439453

Typingimport <tab> takes about 0.03s as well so most of the cost is really insidepkgutil.iter_modules:

>>>import<tab>0.03162884712219238

Subsequent completion requests are much faster since the results are cached:

>>>import<tab>0.002469778060913086

Another thing I tried is multiple search locations (i.e. multiplesys.path entries):

5 different locations and ~4000 packages total:

`pkgutil.iter_modules`:0.3535459041595459sInitial`import <tab>`:0.3592982292175293sSubsequent`import <tab>`:0.0021576881408691406s

10 different locations and ~8000 packages total:

`pkgutil.iter_modules`:0.6836235523223877sInitial`import <tab>`:0.6652779579162598sSubsequent`import <tab>`:0.0002522468566894531s

The initial search is about 0.35s and 0.68s respectively, while the subsequent ones are again very cheap.

I still get this failure locally:

======================================================================FAIL:test_import_completions (test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_import_completions) (code='import path\t\n')----------------------------------------------------------------------Traceback (most recent call last):  File"/Users/hugo/github/python/cpython/main/Lib/test/test_pyrepl/test_pyrepl.py", line938, intest_import_completionsself.assertEqual(output, expected)~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^AssertionError:'import path' != 'import pathlib'- import path+ import pathlib?            +++

It's because my env isn't a clean slate because I've pip installed some packages for testing on 3.14, and a dependency is interfering:

Python 3.14.0a7+ (heads/completer:f4e290a03f1, Apr 23 2025, 13:32:14) [Clang 16.0.0 (clang-1600.0.26.6)] on darwinType "help", "copyright", "credits" or "license" for more information.>>>import path<tab>pathlib       pathvalidate

❯./python.exe -m pip freeze| grep pathpathvalidate==3.2.0

I don't think this needs to block initial merge before the freeze, but would be good to fix.

(PS there's a merge conflict on some imports.)

Weird, I tested this locally with some scripts onsys.path and it passed for me. Sorry about that, I'll take a look later today!

Ok soiter_modules also gets FileFinders fromsys.path so I had to change the approach but itshould work now. Tested locally withpathvalidate installed and the tests are passing.

Passing now, thanks!

LGTM, fantastic job@tomasr8