This article explains the new features in Python 3.3, compared to 3.2.Python 3.3 was released on September 29, 2012. For full details,see thechangelog.
PEP 393: Flexible String Representation¶
The Unicode string type is changed to support multiple internalrepresentations, depending on the character with the largest Unicode ordinal(1, 2, or 4 bytes) in the represented string. This allows a space-efficientrepresentation in common cases, but gives access to full UCS-4 on allsystems. For compatibility with existing APIs, several representations mayexist in parallel; over time, this compatibility should be phased out.
On the Python side, there should be no downside to this change.
On the C API side, PEP 393 is fully backward compatible. The legacy APIshould remain available at least five years. Applications using the legacyAPI will not fully benefit of the memory reduction, or - worse - may usea bit more memory, because Python may have to maintain two versions of eachstring (in the legacy format and in the new efficient storage).
Functionality¶
Changes introduced byPEP 393 are the following:
- Python now always supports the full range of Unicode codepoints, includingnon-BMP ones (i.e. fromU+0000 toU+10FFFF). The distinction betweennarrow and wide builds no longer exists and Python now behaves like a widebuild, even under Windows.
- With the death of narrow builds, the problems specific to narrow builds havealso been fixed, for example:
- len() now always returns 1 for non-BMP characters,solen('\U0010FFFF')==1;
- surrogate pairs are not recombined in string literals,so'\uDBFF\uDFFF'!='\U0010FFFF';
- indexing or slicing non-BMP characters returns the expected value,so'\U0010FFFF'[0] now returns'\U0010FFFF' and not'\uDBFF';
- all other functions in the standard library now correctly handlenon-BMP codepoints.
- The value ofsys.maxunicode is now always1114111 (0x10FFFFin hexadecimal). ThePyUnicode_GetMax() function still returnseither0xFFFF or0x10FFFF for backward compatibility, and it shouldnot be used with the new Unicode API (seeissue 13054).
- The./configure flag--with-wide-unicode has been removed.
Performance and resource usage¶
The storage of Unicode strings now depends on the highest codepoint in the string:
- pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per codepoint;
- BMP strings (U+0000-U+FFFF) use 2 bytes per codepoint;
- non-BMP strings (U+10000-U+10FFFF) use 4 bytes per codepoint.
The net effect is that for most applications, memory usage of stringstorage should decrease significantly - especially compared to formerwide unicode builds - as, in many cases, strings will be pure ASCIIeven in international contexts (because many strings store non-humanlanguage data, such as XML fragments, HTTP headers, JSON-encoded data,etc.). We also hope that it will, for the same reasons, increase CPUcache efficiency on non-trivial applications. The memory usage ofPython 3.3 is two to three times smaller than Python 3.2, and a littlebit better than Python 2.7, on a Django benchmark (see the PEP fordetails).
See also
- PEP 393 - Flexible String Representation
- PEP written by Martin von Löwis; implementation by Torsten Beckerand Martin von Löwis.
Using importlib as the Implementation of Import¶
issue 2377 - Replace __import__ w/ importlib.__import__issue 13959 - Re-implement parts ofimp in pure Pythonissue 14605 - Make import machinery explicitissue 14646 - Require loaders set __loader__ and __package__
The__import__() function is now powered byimportlib.__import__().This work leads to the completion of “phase 2” ofPEP 302. There aremultiple benefits to this change. First, it has allowed for more of themachinery powering import to be exposed instead of being implicit and hiddenwithin the C code. It also provides a single implementation for all Python VMssupporting Python 3.3 to use, helping to end any VM-specific deviations inimport semantics. And finally it eases the maintenance of import, allowing forfuture growth to occur.
For the common user, there should be no visible change in semantics. Forthose whose code currently manipulates import or calls importprogrammatically, the code changes that might possibly be required are coveredin thePorting Python code section of this document.
Visible Changes¶
For potential required changes to code, see thePorting Python codesection.
Beyond the expanse of whatimportlib now exposes, there are othervisible changes to import. The biggest is thatsys.meta_path andsys.path_hooks now store all of the meta path finders and path entryhooks used by import. Previously the finders were implicit and hidden withinthe C code of import instead of being directly exposed. This means that one cannow easily remove or change the order of the various finders to fit one’s needs.
Another change is that all modules have a__loader__ attribute, storing theloader used to create the module.PEP 302 has been updated to make thisattribute mandatory for loaders to implement, so in the future once 3rd-partyloaders have been updated people will be able to rely on the existence of theattribute. Until such time, though, import is setting the module post-load.
Loaders are also now expected to set the__package__ attribute fromPEP 366. Once again, import itself is already setting this on all loadersfromimportlib and import itself is setting the attribute post-load.
None is now inserted intosys.path_importer_cache when no findercan be found onsys.path_hooks. Sinceimp.NullImporter is notdirectly exposed onsys.path_hooks it could no longer be relied upon toalways be available to use as a value representing no finder found.
All other changes relate to semantic changes which should be taken intoconsideration when updating code for Python 3.3, and thus should be read aboutin thePorting Python code section of this document.
(Implementation by Brett Cannon)
Improved Modules¶
abc¶
Improved support for abstract base classes containing descriptors composed withabstract methods. The recommended approach to declaring abstract descriptors isnow to provide__isabstractmethod__ as a dynamically updatedproperty. The built-in descriptors have been updated accordingly.
(Contributed by Darren Dale inissue 11610)
abc.ABCMeta.register() now returns the registered subclass, which meansit can now be used as a class decorator (issue 10868).
array¶
Thearray module supports thelonglong type usingq andQ type codes.
(Contributed by Oren Tirosh and Hirokazu Yamamoto inissue 1172711)
base64¶
ASCII-only Unicode strings are now accepted by the decoding functions of thebase64 modern interface. For example,base64.b64decode('YWJj')returnsb'abc'. (Contributed by Catalin Iacob inissue 13641.)
binascii¶
In addition to the binary objects they normally accept, thea2b_ functionsnow all also accept ASCII-only strings as input. (Contributed by AntoinePitrou inissue 13637.)
bz2¶
Thebz2 module has been rewritten from scratch. In the process, severalnew features have been added:
Newbz2.open() function: open a bzip2-compressed file in binary ortext mode.
bz2.BZ2File can now read from and write to arbitrary file-likeobjects, by means of its constructor’sfileobj argument.
(Contributed by Nadeem Vawda inissue 5863)
bz2.BZ2File andbz2.decompress() can now decompressmulti-stream inputs (such as those produced by thepbzip2 tool).bz2.BZ2File can now also be used to create this type of file, usingthe'a' (append) mode.
(Contributed by Nir Aides inissue 1625)
bz2.BZ2File now implements all of theio.BufferedIOBase API,except for thedetach() andtruncate() methods.
codecs¶
Thembcs codec has been rewritten to handle correctlyreplace andignore error handlers on all Windows versions. Thembcs codec now supports all error handlers, instead of onlyreplace to encode andignore to decode.
A new Windows-only codec has been added:cp65001 (issue 13216). It is theWindows code page 65001 (Windows UTF-8,CP_UTF8). For example, it is usedbysys.stdout if the console output code page is set to cp65001 (e.g., usingchcp65001 command).
Multibyte CJK decoders now resynchronize faster. They only ignore the firstbyte of an invalid byte sequence. For example,b'\xff\n'.decode('gb2312','replace') now returns a\n after the replacement character.
(issue 12016)
Incremental CJK codec encoders are no longer reset at each call to theirencode() methods. For example:
$ ./python -q>>> import codecs>>> encoder = codecs.getincrementalencoder('hz')('strict')>>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')b'~{NpJ)l6HK!#~} Bye.'This example givesb'~{Np~}~{J)~}~{l6~}~{HK~}~{!#~}Bye.' with older Pythonversions.
(issue 12100)
Theunicode_internal codec has been deprecated.
collections¶
Addition of a newChainMap class to allow treating anumber of mappings as a single unit. (Written by Raymond Hettinger forissue 11089, made public inissue 11297)
The abstract base classes have been moved in a newcollections.abcmodule, to better differentiate between the abstract and the concretecollections classes. Aliases for ABCs are still present in thecollections module to preserve existing imports. (issue 11085)
TheCounter class now supports the unary+ and-operators, as well as the in-place operators+=,-=,|=, and&=. (Contributed by Raymond Hettinger inissue 13121.)
contextlib¶
ExitStack now provides a solid foundation forprogrammatic manipulation of context managers and similar cleanupfunctionality. Unlike the previouscontextlib.nested API (which wasdeprecated and removed), the new API is designed to work correctlyregardless of whether context managers acquire their resources intheir__init__ method (for example, file objects) or in their__enter__ method (for example, synchronisation objects from thethreading module).
(issue 13585)
curses¶
- If thecurses module is linked to the ncursesw library, use Unicodefunctions when Unicode strings or characters are passed (e.g.waddwstr()), and bytes functions otherwise (e.g.waddstr()).
- Use the locale encoding instead ofutf-8 to encode Unicode strings.
- curses.window has a newcurses.window.encoding attribute.
- Thecurses.window class has a newget_wch()method to get a wide character
- Thecurses module has a newunget_wch() function topush a wide character so the nextget_wch() will returnit
(Contributed by Iñigo Serna inissue 6755)
decimal¶
- issue 7652 - integrate fast native decimal arithmetic.
- C-module and libmpdec written by Stefan Krah.
The new C version of the decimal module integrates the high speed libmpdeclibrary for arbitrary precision correctly-rounded decimal floating pointarithmetic. libmpdec conforms to IBM’s General Decimal Arithmetic Specification.
Performance gains range from 10x for database applications to 100x fornumerically intensive applications. These numbers are expected gainsfor standard precisions used in decimal floating point arithmetic. Sincethe precision is user configurable, the exact figures may vary. For example,in integer bignum arithmetic the differences can be significantly higher.
The following table is meant as an illustration. Benchmarks are availableathttp://www.bytereef.org/mpdecimal/quickstart.html.
| | decimal.py | _decimal | speedup |
|---|
| pi | 42.02s | 0.345s | 120x |
| telco | 172.19s | 5.68s | 30x |
| psycopg | 3.57s | 0.29s | 12x |
Features¶
- TheFloatOperation signal optionally enables strictersemantics for mixing floats and Decimals.
- If Python is compiled without threads, the C version automaticallydisables the expensive thread local context machinery. In this case,the variableHAVE_THREADS is set toFalse.
API changes¶
The C module has the following context limits, depending on the machinearchitecture:
| | 32-bit | 64-bit |
|---|
| MAX_PREC | 425000000 | 999999999999999999 |
| MAX_EMAX | 425000000 | 999999999999999999 |
| MIN_EMIN | -425000000 | -999999999999999999 |
In the context templates (DefaultContext,BasicContext andExtendedContext)the magnitude ofEmax andEmin has changed to999999.
TheDecimal constructor in decimal.py does not observethe context limits and converts values with arbitrary exponents or precisionexactly. Since the C version has internal limits, the following scheme isused: If possible, values are converted exactly, otherwiseInvalidOperation is raised and the result is NaN. In thelatter case it is always possible to usecreate_decimal()in order to obtain a rounded or inexact value.
The power function in decimal.py is always correctly-rounded. In theC version, it is defined in terms of the correctly-roundedexp() andln() functions,but the final result is only “almost always correctly rounded”.
In the C version, the context dictionary containing the signals is aMutableMapping. For speed reasons,flags andtraps alwaysrefer to the sameMutableMapping that the contextwas initialized with. If a new signal dictionary is assigned,flags andtrapsare updated with the new values, but they do not reference the RHSdictionary.
Pickling aContext produces a different output in orderto have a common interchange format for the Python and C versions.
The order of arguments in theContext constructor has beenchanged to match the order displayed byrepr().
Thewatchexp parameter in thequantize() methodis deprecated.
email¶
Policy Framework¶
The email package now has apolicy framework. APolicy is an object with several methods and propertiesthat control how the email package behaves. The primary policy for Python 3.3is theCompat32 policy, which provides backwardcompatibility with the email package in Python 3.2. Apolicy can bespecified when an email message is parsed by aparser, or when aMessage object is created, or when an email isserialized using agenerator. Unless overridden, a policy passedto aparser is inherited by all theMessage object and sub-objectscreated by theparser. By default agenerator will use the policy oftheMessage object it is serializing. The default policy iscompat32.
The minimum set of controls implemented by allpolicy objects are:
| max_line_length | The maximum length, excluding the linesep character(s),individual lines may have when aMessage isserialized. Defaults to 78. |
| linesep | The character used to separate individual lines when aMessage is serialized. Defaults to\n. |
| cte_type | 7bit or8bit.8bit applies only to aBytesgenerator, and means that non-ASCII maybe used where allowed by the protocol (or where itexists in the original input). |
| raise_on_defect | Causes aparser to raise error when defects areencountered instead of adding them to theMessageobject’sdefects list. |
A new policy instance, with new settings, is created using theclone() method of policy objects.clone takesany of the above controls as keyword arguments. Any control not specified inthe call retains its default value. Thus you can create a policy that uses\r\n linesep characters like this:
mypolicy=compat32.clone(linesep='\r\n')
Policies can be used to make the generation of messages in the format needed byyour application simpler. Instead of having to remember to specifylinesep='\r\n' in all the places you call agenerator, you can specifyit once, when you set the policy used by theparser or theMessage,whichever your program uses to createMessage objects. On the other hand,if you need to generate messages in multiple forms, you can still specify theparameters in the appropriategenerator call. Or you can have custompolicy instances for your different cases, and pass those in when you createthegenerator.
Provisional Policy with New Header API¶
While the policy framework is worthwhile all by itself, the main motivation forintroducing it is to allow the creation of new policies that implement newfeatures for the email package in a way that maintains backward compatibilityfor those who do not use the new policies. Because the new policies introduce anew API, we are releasing them in Python 3.3 as aprovisional policy. Backwards incompatible changes (up to and includingremoval of the code) may occur if deemed necessary by the core developers.
The new policies are instances ofEmailPolicy,and add the following additional controls:
| refold_source | Controls whether or not headers parsed by aparser are refolded by thegenerator. It can benone,long,orall. The default islong, which means thatsource headers with a line longer thanmax_line_length get refolded.none means noline get refolded, andall means that all linesget refolded. |
| header_factory | A callable that take aname andvalue andproduces a custom header object. |
Theheader_factory is the key to the new features provided by the newpolicies. When one of the new policies is used, any header retrieved fromaMessage object is an object produced by theheader_factory, and anytime you set a header on aMessage it becomes an object produced byheader_factory. All such header objects have aname attribute equalto the header name. Address and Date headers have additional attributesthat give you access to the parsed data of the header. This means you can nowdo things like this:
>>>m=Message(policy=SMTP)>>>m['To']='Éric <foo@example.com>'>>>m['to']'Éric <foo@example.com>'>>>m['to'].addresses(Address(display_name='Éric', username='foo', domain='example.com'),)>>>m['to'].addresses[0].username'foo'>>>m['to'].addresses[0].display_name'Éric'>>>m['Date']=email.utils.localtime()>>>m['Date'].datetimedatetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))>>>m['Date']'Fri, 25 May 2012 21:44:27 -0400'>>>print(m)To: =?utf-8?q?=C3=89ric?= <foo@example.com>Date: Fri, 25 May 2012 21:44:27 -0400
You will note that the unicode display name is automatically encoded asutf-8 when the message is serialized, but that when the header is accesseddirectly, you get the unicode version. This eliminates any need to deal withtheemail.headerdecode_header() ormake_header() functions.
You can also create addresses from parts:
>>>m['cc']=[Group('pals',[Address('Bob','bob','example.com'),...Address('Sally','sally','example.com')]),...Address('Bonzo',addr_spec='bonz@laugh.com')]>>>print(m)To: =?utf-8?q?=C3=89ric?= <foo@example.com>Date: Fri, 25 May 2012 21:44:27 -0400cc: pals: Bob <bob@example.com>, Sally <sally@example.com>;, Bonzo <bonz@laugh.com>Decoding to unicode is done automatically:
>>>m2=message_from_string(str(m))>>>m2['to']'Éric <foo@example.com>'
When you parse a message, you can use theaddresses andgroupsattributes of the header objects to access the groups and individualaddresses:
>>>m2['cc'].addresses(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))>>>m2['cc'].groups(Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))
In summary, if you use one of the new policies, header manipulation works theway it ought to: your application works with unicode strings, and the emailpackage transparently encodes and decodes the unicode to and from the RFCstandard Content Transfer Encodings.
Other API Changes¶
NewBytesHeaderParser, added to theparsermodule to complementHeaderParser and complete the BytesAPI.
New utility functions:
ftplib¶
- ftplib.FTP now accepts asource_address keyword argument tospecify the(host,port) to use as the source address in the bind callwhen creating the outgoing socket. (Contributed by Giampaolo Rodolàinissue 8594.)
- TheFTP_TLS class now provides a newccc() function to revert control channel back toplaintext. This can be useful to take advantage of firewalls that know howto handle NAT with non-secure FTP without opening fixed ports. (Contributedby Giampaolo Rodolà inissue 12139)
- Addedftplib.FTP.mlsd() method which provides a parsable directorylisting format and deprecatesftplib.FTP.nlst() andftplib.FTP.dir(). (Contributed by Giampaolo Rodolà inissue 11072)
functools¶
Thefunctools.lru_cache() decorator now accepts atyped keywordargument (that defaults toFalse to ensure that it caches values ofdifferent types that compare equal in separate cache slots. (Contributedby Raymond Hettinger inissue 13227.)
It is now possible to register callbacks invoked by the garbage collectorbefore and after collection using the newcallbacks list.
hmac¶
A newcompare_digest() function has been added to prevent sidechannel attacks on digests through timing analysis. (Contributed by NickCoghlan and Christian Heimes inissue 15061)
html¶
html.parser.HTMLParser is now able to parse broken markup withoutraising errors, therefore thestrict argument of the constructor and theHTMLParseError exception are now deprecated.The ability to parse broken markup is the result of a number of bug fixes thatare also available on the latest bug fix releases of Python 2.7/3.2.(Contributed by Ezio Melotti inissue 15114, andissue 14538,issue 13993,issue 13960,issue 13358,issue 1745761,issue 755670,issue 13357,issue 12629,issue 1200313,issue 670664,issue 13273,issue 12888,issue 7311)
A newhtml5 dictionary that maps HTML5 named characterreferences to the equivalent Unicode character(s) (e.g.html5['gt;']=='>') has been added to thehtml.entities module. The dictionary isnow also used byHTMLParser. (Contributed by EzioMelotti inissue 11113 andissue 15156)
imaplib¶
TheIMAP4_SSL constructor now accepts an SSLContextparameter to control parameters of the secure channel.
(Contributed by Sijin Joseph inissue 8808)
inspect¶
A newgetclosurevars() function has been added. This functionreports the current binding of all names referenced from the function body andwhere those names were resolved, making it easier to verify correct internalstate when testing code that relies on stateful closures.
(Contributed by Meador Inge and Nick Coghlan inissue 13062)
A newgetgeneratorlocals() function has been added. Thisfunction reports the current binding of local variables in the generator’sstack frame, making it easier to verify correct internal state when testinggenerators.
(Contributed by Meador Inge inissue 15153)
Theopen() function has a new'x' mode that can be used toexclusively create a new file, and raise aFileExistsError if the filealready exists. It is based on the C11 ‘x’ mode to fopen().
(Contributed by David Townshend inissue 12760)
The constructor of theTextIOWrapper class has a newwrite_through optional argument. Ifwrite_through isTrue, calls towrite() are guaranteed not to be buffered: any datawritten on theTextIOWrapper object is immediately handled to itsunderlying binary buffer.
itertools¶
accumulate() now takes an optionalfunc argument forproviding a user-supplied binary function.
logging¶
ThebasicConfig() function now supports an optionalhandlersargument taking an iterable of handlers to be added to the root logger.
A class level attributeappend_nul hasbeen added toSysLogHandler to allow control of theappending of theNUL (\000) byte to syslog records, since for somedeamons it is required while for others it is passed through to the log.
math¶
Themath module has a new function,log2(), which returnsthe base-2 logarithm ofx.
(Written by Mark Dickinson inissue 11888).
mmap¶
Theread() method is now more compatible with other file-likeobjects: if the argument is omitted or specified asNone, it returns thebytes from the current file position to the end of the mapping. (Contributedby Petri Lehtinen inissue 12021.)
nntplib¶
Thenntplib.NNTP class now supports the context manager protocol tounconditionally consumesocket.error exceptions and to close the NNTPconnection when done:
>>>fromnntplibimportNNTP>>>withNNTP('news.gmane.org')asn:...n.group('gmane.comp.python.committers')...('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')>>>(Contributed by Giampaolo Rodolà inissue 9795)
Theos module has a newpipe2() function that makes itpossible to create a pipe withO_CLOEXEC orO_NONBLOCK flags set atomically. This is especially useful toavoid race conditions in multi-threaded programs.
Theos module has a newsendfile() function which providesan efficent “zero-copy” way for copying data from one file (or socket)descriptor to another. The phrase “zero-copy” refers to the fact that all ofthe copying of data between the two descriptors is done entirely by thekernel, with no copying of data into userspace buffers.sendfile()can be used to efficiently copy data from a file on disk to a network socket,e.g. for downloading a file.
(Patch submitted by Ross Lagerwall and Giampaolo Rodolà inissue 10882.)
To avoid race conditions like symlink attacks and issues with temporaryfiles and directories, it is more reliable (and also faster) to manipulatefile descriptors instead of file names. Python 3.3 enhances existing functionsand introduces new functions to work on file descriptors (issue 4761,issue 10755 andissue 14626).
- Theos module has a newfwalk() function similar towalk() except that it also yields file descriptors referring to thedirectories visited. This is especially useful to avoid symlink races.
- The following functions get new optionaldir_fd (paths relative todirectory descriptors) and/orfollow_symlinks (notfollowing symlinks):access(),chflags(),chmod(),chown(),link(),lstat(),mkdir(),mkfifo(),mknod(),open(),readlink(),remove(),rename(),replace(),rmdir(),stat(),symlink(),unlink(),utime(). Platformsupport for using these parameters can be checked via the setsos.supports_dir_fd andos.supports_follows_symlinks.
- The following functions now support a file descriptor for their path argument:chdir(),chmod(),chown(),execve(),listdir(),pathconf(),exists(),stat(),statvfs(),utime(). Platform supportfor this can be checked via theos.supports_fd set.
access() accepts aneffective_ids keyword argument to turn onusing the effective uid/gid rather than the real uid/gid in the access check.Platform support for this can be checked via thesupports_effective_ids set.
Theos module has two new functions:getpriority() andsetpriority(). They can be used to get or set processniceness/priority in a fashion similar toos.nice() but extended to allprocesses instead of just the current one.
(Patch submitted by Giampaolo Rodolà inissue 10784.)
The newos.replace() function allows cross-platform renaming of afile with overwriting the destination. Withos.rename(), an existingdestination file is overwritten under POSIX, but raises an error underWindows.(Contributed by Antoine Pitrou inissue 8828.)
The stat family of functions (stat(),fstat(),andlstat()) now support reading a file’s timestampswith nanosecond precision. Symmetrically,utime()can now write file timestamps with nanosecond precision. (Contributed byLarry Hastings inissue 14127.)
The newos.get_terminal_size() function queries the size of theterminal attached to a file descriptor. See alsoshutil.get_terminal_size().(Contributed by Zbigniew Jędrzejewski-Szmek inissue 13609.)
- New functions to support Linux extended attributes (issue 12720):getxattr(),listxattr(),removexattr(),setxattr().
- New interface to the scheduler. These functionscontrol how a process is allocated CPU time by the operating system. Newfunctions:sched_get_priority_max(),sched_get_priority_min(),sched_getaffinity(),sched_getparam(),sched_getscheduler(),sched_rr_get_interval(),sched_setaffinity(),sched_setparam(),sched_setscheduler(),sched_yield(),
- New functions to control the file system:
- posix_fadvise(): Announces an intention to access data in aspecific pattern thus allowing the kernel to make optimizations.
- posix_fallocate(): Ensures that enough disk space is allocatedfor a file.
- sync(): Force write of everything to disk.
- Additional new posix functions:
- lockf(): Apply, test or remove a POSIX lock on an open file descriptor.
- pread(): Read from a file descriptor at an offset, the fileoffset remains unchanged.
- pwrite(): Write to a file descriptor from an offset, leavingthe file offset unchanged.
- readv(): Read from a file descriptor into a number of writable buffers.
- truncate(): Truncate the file corresponding topath, so thatit is at mostlength bytes in size.
- waitid(): Wait for the completion of one or more child processes.
- writev(): Write the contents ofbuffers to a file descriptor,wherebuffers is an arbitrary sequence of buffers.
- getgrouplist() (issue 9344): Return list of group ids thatspecified user belongs to.
- times() anduname(): Return type changed from a tuple toa tuple-like object with named attributes.
- Some platforms now support additional constants for thelseek()function, such asos.SEEK_HOLE andos.SEEK_DATA.
- New constantsRTLD_LAZY,RTLD_NOW,RTLD_GLOBAL,RTLD_LOCAL,RTLD_NODELETE,RTLD_NOLOAD, andRTLD_DEEPBIND are available onplatforms that support them. These are for use with thesys.setdlopenflags() function, and supersede the similar constantsdefined inctypes andDLFCN. (Contributed by Victor Stinnerinissue 13226.)
- os.symlink() now accepts (and ignores) thetarget_is_directorykeyword argument on non-Windows platforms, to ease cross-platform support.
pdb¶
Tab-completion is now available not only for command names, but also theirarguments. For example, for thebreak command, function and file namesare completed.
(Contributed by Georg Brandl inissue 14210)
pydoc¶
The Tk GUI and theserve() function have been removed from thepydoc module:pydoc-g andserve() have been deprecatedin Python 3.2.
str regular expressions now support\u and\U escapes.
(Contributed by Serhiy Storchaka inissue 3665.)
sched¶
- run() now accepts ablocking parameter which whenset to False makes the method execute the scheduled events due to expiresoonest (if any) and then return immediately.This is useful in case you want to use thescheduler innon-blocking applications. (Contributed by Giampaolo Rodolà inissue 13449)
- scheduler class can now be safely used in multi-threadedenvironments. (Contributed by Josiah Carlson and Giampaolo Rodolà inissue 8684)
- timefunc anddelayfunct parameters ofscheduler classconstructor are now optional and defaults totime.time() andtime.sleep() respectively. (Contributed by Chris Clark inissue 13245)
- enter() andenterabs()argument parameter is now optional. (Contributed by Chris Clark inissue 13245)
- enter() andenterabs()now accept akwargs parameter. (Contributed by Chris Clark inissue 13245)
select¶
Solaris and derivative platforms have a new classselect.devpollfor high performance asynchronous sockets via/dev/poll.(Contributed by Jesús Cea Avión inissue 6397.)
shlex¶
The previously undocumented helper functionquote from thepipes modules has been moved to theshlex module anddocumented.quote() properly escapes all characters in a stringthat might be otherwise given special meaning by the shell.
shutil¶
- New functions:
- disk_usage(): provides total, used and free disk spacestatistics. (Contributed by Giampaolo Rodolà inissue 12442)
- chown(): allows one to change user and/or group of the givenpath also specifying the user/group names and not only their numericids. (Contributed by Sandro Tosi inissue 12191)
- shutil.get_terminal_size(): returns the size of the terminal windowto which the interpreter is attached. (Contributed by ZbigniewJędrzejewski-Szmek inissue 13609.)
- copy2() andcopystat() now preserve filetimestamps with nanosecond precision on platforms that support it.They also preserve file “extended attributes” on Linux. (Contributedby Larry Hastings inissue 14127 andissue 15238.)
- Several functions now take an optionalsymlinks argument: when thatparameter is true, symlinks aren’t dereferenced and the operation insteadacts on the symlink itself (or creates one, if relevant).(Contributed by Hynek Schlawack inissue 12715.)
- When copying files to a different file system,move() nowhandles symlinks the way the posixmv command does, recreating thesymlink rather than copying the target file contents. (Contributed byJonathan Niehof inissue 9993.)move() now also returnsthedst argument as its result.
- rmtree() is now resistant to symlink attacks on platformswhich support the newdir_fd parameter inos.open() andos.unlink(). (Contributed by Martin von Löwis and Hynek Schlawackinissue 4489.)
signal¶
- Thesignal module has new functions:
- The signal handler writes the signal number as a single byte instead ofa nul byte into the wakeup file descriptor. So it is possible to wait morethan one signal and know which signals were raised.
- signal.signal() andsignal.siginterrupt() raise an OSError,instead of a RuntimeError: OSError has an errno attribute.
smtpd¶
Thesmtpd module now supportsRFC 5321 (extended SMTP) andRFC 1870(size extension). Per the standard, these extensions are enabled if and onlyif the client initiates the session with anEHLO command.
(InitialELHO support by Alberto Trevino. Size extension by JuhanaJauhiainen. Substantial additional work on the patch contributed by MicheleOrrù and Dan Boswell.issue 8739)
smtplib¶
TheSMTP,SMTP_SSL, andLMTP classes now accept asource_address keyword argumentto specify the(host,port) to use as the source address in the bind callwhen creating the outgoing socket. (Contributed by Paulo Scardine inissue 11281.)
SMTP now supports the context manager protocol, allowing anSMTP instance to be used in awith statement. (Contributedby Giampaolo Rodolà inissue 11289.)
TheSMTP_SSL constructor and thestarttls()method now accept an SSLContext parameter to control parameters of the securechannel. (Contributed by Kasun Herath inissue 8809)
stat¶
The undocumented tarfile.filemode function has been moved tostat.filemode(). It can be used to convert a file’s mode to a string ofthe form ‘-rwxrwxrwx’.
(Contributed by Giampaolo Rodolà inissue 14807)
struct¶
Thestruct module now supportsssize_t andsize_t via thenew codesn andN, respectively. (Contributed by Antoine Pitrouinissue 3163.)
subprocess¶
Command strings can now be bytes objects on posix platforms. (Contributed byVictor Stinner inissue 8513.)
A new constantDEVNULL allows suppressing output in aplatform-independent fashion. (Contributed by Ross Lagerwall inissue 5870.)
textwrap¶
Thetextwrap module has a newindent() that makesit straightforward to add a common prefix to selected lines in a blockof text (issue 13857).
time¶
ThePEP 418 added new functions to thetime module:
- get_clock_info(): Get information on a clock.
- monotonic(): Monotonic clock (cannot go backward), not affectedby system clock updates.
- perf_counter(): Performance counter with the highest availableresolution to measure a short duration.
- process_time(): Sum of the system and user CPU time of thecurrent process.
Other new functions:
To improve cross platform consistency,sleep() now raises aValueError when passed a negative sleep value. Previously this was anerror on posix, but produced an infinite sleep on Windows.
urllib¶
TheRequest class, now accepts amethod argumentused byget_method() to determine what HTTP methodshould be used. For example, this will send a'HEAD' request:
>>>urlopen(Request('http://www.python.org',method='HEAD'))(issue 1673007)
webbrowser¶
Thewebbrowser module supports more “browsers”: Google Chrome (namedchrome,chromium,chrome-browser orchromium-browser depending on the version and operating system),and the generic launchersxdg-open, from the FreeDesktop.orgproject, andgvfs-open, which is the default URI handler for GNOME3. (The former contributed by Arnaud Calmettes inissue 13620, the latterby Matthias Klose inissue 14493)
xml.etree.ElementTree¶
Thexml.etree.ElementTree module now imports its C accelerator bydefault; there is no longer a need to explicitly importxml.etree.cElementTree (this module stays for backwards compatibility,but is now deprecated). In addition, theiter family of methods ofElement has been optimized (rewritten in C).The module’s documentation has also been greatly improved with added examplesand a more detailed reference.
zlib¶
New attributezlib.Decompress.eof makes it possible to distinguishbetween a properly-formed compressed stream and an incomplete or truncated one.(Contributed by Nadeem Vawda inissue 12646.)
New attributezlib.ZLIB_RUNTIME_VERSION reports the version string ofthe underlyingzlib library that is loaded at runtime. (Contributed byTorsten Landschoff inissue 12306.)
Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
