Movatterモバイル変換

[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

• Previous message:[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
• Next message:[Python-Dev] PEP 471 -- os.scandir() function -- a better andfaster directory iterator
• Messages sorted by:[ date ][ thread ][ subject ][ author ]

Hello,On Thu, 26 Jun 2014 21:52:43 -0400Ben Hoyt <benhoyt at gmail.com> wrote:[]> It's a fair point that os.walk() can be implemented efficiently> without adding a new function and API. However, often you'll want more> info, like the file size, which scandir() can give you via> DirEntry.lstat(), which is free on Windows. So opening up this> efficient API is beneficial.>> In CPython, I think the DirEntry objects are as lightweight as> stat_result objects.>> I'm an embedded developer by background, so I know the constraints> here, but I really don't think Python's development should be tailored> to fit MicroPython. If os.scandir() is not very efficient on> MicroPython, so be it -- 99% of all desktop/server users will gain> from it.Surely, tailoring Python to MicroPython's needs is completely not whatI suggest. It was an example of alternative implementation whichoptimized os.walk() without need for any additional public module APIs.Vice-versa, high-level nature of API call like os.walk() andunderspecification of low-level details (like which functionimplemented in terms of which others) allow MicroPython provideoptimized implementation even with its resource constraints. So, powerof high-level interfaces and underspecification should not beunderestimated ;-).But I don't want to argue that os.scandir() is "not needed", becausethat's hardly productive. Something I'd like to prototype in uPy andideally lead further up to PEP status is to add iterator-based stringmethods, and I pretty much can expect "we lived without it" response,so don't want to go the same way regarding addition of otheriterator-based APIs - it's clear that more iterator/generator based APIsis a good direction for Python to evolve.> > It would be better if os.scandir() was specified to return a struct> > (named tuple) compatible with return value of os.stat() (with only> > fields relevant to underlying readdir()-like system call). The> > grounds for that are obvious: it's already existing data interface> > in module "os", which is also based on open standard for operating> > systems - POSIX, so if one is to expect something about file> > attributes, it's what one can reasonably base expectations on.>> Yes, we considered this early on (see the python-ideas and python-dev> threads referenced in the PEP), but decided it wasn't a great API to> overload stat_result further, and have most of the attributes None or> not present on Linux.>[]>> However, for scandir() to be useful, you also need the name. My> original version of this directory iterator returned two-tuples of> (name, stat_result). But most people didn't like the API, and I don't> really either. You could overload stat_result with a .name attribute> in this case, but it still isn't a nice API to have most of the> attributes None, and then you have to test for that, etc.Yes, returning (name, stat_result) would be my first motion too, Idon't see why someone wouldn't like pair of 2 values, with each valueof obvious type and semantics within "os" module. Regarding statresult, os.stat() provides full information about a file,and intuitively, one may expect that os.scandir() would provide subsetof that info, asymptotically reaching volume of what os.stat() mayprovide, depending on OS capabilities. So, if truly OS-independentinterface is wanted to salvage more data from a dir scanning, usingos.stat struct as data interface is hard to ignore.But well, if it was rejected already, what can be said? Perhaps, atleast the PEP could be extended to explicitly mention other approachedwhich were discussed and rejected, not just link to a discussionarchive (from experience with reading other PEPs, they oftentimescontained such subsections, so hope this suggestion is not ungrounded).>> So basically we tweaked the API to do what was best, and ended up with> it returning DirEntry objects with is_file() and similar methods.>> Hope that helps give a bit more context. If you haven't read the> relevant python-ideas and python-dev threads, those are interesting> too.>> -Ben-- Best regards, Paul                          mailto:pmiscml at gmail.com

• Previous message:[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
• Next message:[Python-Dev] PEP 471 -- os.scandir() function -- a better andfaster directory iterator
• Messages sorted by:[ date ][ thread ][ subject ][ author ]