11.1.Output Formatting¶

Thereprlib module provides a version ofrepr() customized forabbreviated displays of large or deeply nested containers:

>>>importreprlib>>>reprlib.repr(set('supercalifragilisticexpialidocious'))"{'a', 'c', 'd', 'e', 'f', 'g', ...}"

Thepprint module offers more sophisticated control over printing bothbuilt-in and user defined objects in a way that is readable by the interpreter.When the result is longer than one line, the “pretty printer” adds line breaksand indentation to more clearly reveal data structure:

>>>importpprint>>>t=[[[['black','cyan'],'white',['green','red']],[['magenta',...'yellow'],'blue']]]...>>>pprint.pprint(t,width=30)[[[['black', 'cyan'],   'white',   ['green', 'red']],  [['magenta', 'yellow'],   'blue']]]

Thetextwrap module formats paragraphs of text to fit a given screenwidth:

>>>importtextwrap>>>doc="""The wrap() method is just like fill() except that it returns...a list of strings instead of one big string with newlines to separate...the wrapped lines."""...>>>print(textwrap.fill(doc,width=40))The wrap() method is just like fill()except that it returns a list of stringsinstead of one big string with newlinesto separate the wrapped lines.

Thelocale module accesses a database of culture specific data formats.The grouping attribute of locale’s format function provides a direct way offormatting numbers with group separators:

>>>importlocale>>>locale.setlocale(locale.LC_ALL,'English_United States.1252')'English_United States.1252'>>>conv=locale.localeconv()# get a mapping of conventions>>>x=1234567.8>>>locale.format_string("%d",x,grouping=True)'1,234,567'>>>locale.format_string("%s%.*f",(conv['currency_symbol'],...conv['frac_digits'],x),grouping=True)'$1,234,567.80'

11.2.Templating¶

Thestring module includes a versatileTemplate classwith a simplified syntax suitable for editing by end-users. This allows usersto customize their applications without having to alter the application.

The format uses placeholder names formed by$ with valid Python identifiers(alphanumeric characters and underscores). Surrounding the placeholder withbraces allows it to be followed by more alphanumeric letters with no interveningspaces. Writing$$ creates a single escaped$:

>>>fromstringimportTemplate>>>t=Template('${village}folk send $$10 to $cause.')>>>t.substitute(village='Nottingham',cause='the ditch fund')'Nottinghamfolk send $10 to the ditch fund.'

Thesubstitute() method raises aKeyError when aplaceholder is not supplied in a dictionary or a keyword argument. Formail-merge style applications, user supplied data may be incomplete and thesafe_substitute() method may be more appropriate —it will leave placeholders unchanged if data is missing:

>>>t=Template('Return the $item to $owner.')>>>d=dict(item='unladen swallow')>>>t.substitute(d)Traceback (most recent call last):...KeyError:'owner'>>>t.safe_substitute(d)'Return the unladen swallow to $owner.'

Template subclasses can specify a custom delimiter. For example, a batchrenaming utility for a photo browser may elect to use percent signs forplaceholders such as the current date, image sequence number, or file format:

>>>importtime,os.path>>>photofiles=['img_1074.jpg','img_1076.jpg','img_1077.jpg']>>>classBatchRename(Template):...delimiter='%'...>>>fmt=input('Enter rename style (%d-date %n-seqnum%f-format):  ')Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f>>>t=BatchRename(fmt)>>>date=time.strftime('%d%b%y')>>>fori,filenameinenumerate(photofiles):...base,ext=os.path.splitext(filename)...newname=t.substitute(d=date,n=i,f=ext)...print('{0} -->{1}'.format(filename,newname))img_1074.jpg --> Ashley_0.jpgimg_1076.jpg --> Ashley_1.jpgimg_1077.jpg --> Ashley_2.jpg

Another application for templating is separating program logic from the detailsof multiple output formats. This makes it possible to substitute customtemplates for XML files, plain text reports, and HTML web reports.

11.3.Working with Binary Data Record Layouts¶

Thestruct module providespack() andunpack() functions for working with variable length binaryrecord formats. The following example showshow to loop through header information in a ZIP file without using thezipfile module. Pack codes"H" and"I" represent two and fourbyte unsigned numbers respectively. The"<" indicates that they arestandard size and in little-endian byte order:

importstructwithopen('myfile.zip','rb')asf:data=f.read()start=0foriinrange(3):# show the first 3 file headersstart+=14fields=struct.unpack('<IIIHH',data[start:start+16])crc32,comp_size,uncomp_size,filenamesize,extra_size=fieldsstart+=16filename=data[start:start+filenamesize]start+=filenamesizeextra=data[start:start+extra_size]print(filename,hex(crc32),comp_size,uncomp_size)start+=extra_size+comp_size# skip to the next header

11.4.Multi-threading¶

Threading is a technique for decoupling tasks which are not sequentiallydependent. Threads can be used to improve the responsiveness of applicationsthat accept user input while other tasks run in the background. A related usecase is running I/O in parallel with computations in another thread.

The following code shows how the high levelthreading module can runtasks in background while the main program continues to run:

importthreading,zipfileclassAsyncZip(threading.Thread):def__init__(self,infile,outfile):threading.Thread.__init__(self)self.infile=infileself.outfile=outfiledefrun(self):f=zipfile.ZipFile(self.outfile,'w',zipfile.ZIP_DEFLATED)f.write(self.infile)f.close()print('Finished background zip of:',self.infile)background=AsyncZip('mydata.txt','myarchive.zip')background.start()print('The main program continues to run in foreground.')background.join()# Wait for the background task to finishprint('Main program waited until background was done.')

The principal challenge of multi-threaded applications is coordinating threadsthat share data or other resources. To that end, the threading module providesa number of synchronization primitives including locks, events, conditionvariables, and semaphores.

While those tools are powerful, minor design errors can result in problems thatare difficult to reproduce. So, the preferred approach to task coordination isto concentrate all access to a resource in a single thread and then use thequeue module to feed that thread with requests from other threads.Applications usingQueue objects for inter-thread communication andcoordination are easier to design, more readable, and more reliable.

11.5.Logging¶

Thelogging module offers a full featured and flexible logging system.At its simplest, log messages are sent to a file or tosys.stderr:

importlogginglogging.debug('Debugging information')logging.info('Informational message')logging.warning('Warning:config file%s not found','server.conf')logging.error('Error occurred')logging.critical('Critical error -- shutting down')

This produces the following output:

WARNING:root:Warning:config file server.conf not foundERROR:root:Error occurredCRITICAL:root:Critical error -- shutting down

By default, informational and debugging messages are suppressed and the outputis sent to standard error. Other output options include routing messagesthrough email, datagrams, sockets, or to an HTTP Server. New filters can selectdifferent routing based on message priority:DEBUG,INFO,WARNING,ERROR,andCRITICAL.

The logging system can be configured directly from Python or can be loaded froma user editable configuration file for customized logging without altering theapplication.

11.6.Weak References¶

Python does automatic memory management (reference counting for most objects andgarbage collection to eliminate cycles). The memory is freed shortlyafter the last reference to it has been eliminated.

This approach works fine for most applications but occasionally there is a needto track objects only as long as they are being used by something else.Unfortunately, just tracking them creates a reference that makes them permanent.Theweakref module provides tools for tracking objects without creating areference. When the object is no longer needed, it is automatically removedfrom a weakref table and a callback is triggered for weakref objects. Typicalapplications include caching objects that are expensive to create:

>>>importweakref,gc>>>classA:...def__init__(self,value):...self.value=value...def__repr__(self):...returnstr(self.value)...>>>a=A(10)# create a reference>>>d=weakref.WeakValueDictionary()>>>d['primary']=a# does not create a reference>>>d['primary']# fetch the object if it is still alive10>>>dela# remove the one reference>>>gc.collect()# run garbage collection right away0>>>d['primary']# entry was automatically removedTraceback (most recent call last):  File"<stdin>", line1, in<module>d['primary']# entry was automatically removed  File"C:/python313/lib/weakref.py", line46, in__getitem__o=self.data[key]()KeyError:'primary'

11.7.Tools for Working with Lists¶

Many data structure needs can be met with the built-in list type. However,sometimes there is a need for alternative implementations with differentperformance trade-offs.

Thearray module provides anarray object that is likea list that stores only homogeneous data and stores it more compactly. Thefollowing example shows an array of numbers stored as two byte unsigned binarynumbers (typecode"H") rather than the usual 16 bytes per entry for regularlists of Python int objects:

>>>fromarrayimportarray>>>a=array('H',[4000,10,700,22222])>>>sum(a)26932>>>a[1:3]array('H', [10, 700])

Thecollections module provides adeque objectthat is like a list with faster appends and pops from the left side but slowerlookups in the middle. These objects are well suited for implementing queuesand breadth first tree searches:

>>>fromcollectionsimportdeque>>>d=deque(["task1","task2","task3"])>>>d.append("task4")>>>print("Handling",d.popleft())Handling task1

unsearched=deque([starting_node])defbreadth_first_search(unsearched):node=unsearched.popleft()formingen_moves(node):ifis_goal(m):returnmunsearched.append(m)

In addition to alternative list implementations, the library also offers othertools such as thebisect module with functions for manipulating sortedlists:

>>>importbisect>>>scores=[(100,'perl'),(200,'tcl'),(400,'lua'),(500,'python')]>>>bisect.insort(scores,(300,'ruby'))>>>scores[(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]

Theheapq module provides functions for implementing heaps based onregular lists. The lowest valued entry is always kept at position zero. Thisis useful for applications which repeatedly access the smallest element but donot want to run a full list sort:

>>>fromheapqimportheapify,heappop,heappush>>>data=[1,3,5,7,9,2,4,6,8,0]>>>heapify(data)# rearrange the list into heap order>>>heappush(data,-5)# add a new entry>>>[heappop(data)foriinrange(3)]# fetch the three smallest entries[-5, 0, 1]

11.8.Decimal Floating-Point Arithmetic¶

Thedecimal module offers aDecimal datatype fordecimal floating-point arithmetic. Compared to the built-infloatimplementation of binary floating point, the class is especially helpful for

• financial applications and other uses which require exact decimalrepresentation,

• control over precision,

• control over rounding to meet legal or regulatory requirements,

• tracking of significant decimal places, or

• applications where the user expects the results to match calculations done byhand.

For example, calculating a 5% tax on a 70 cent phone charge gives differentresults in decimal floating point and binary floating point. The differencebecomes significant if the results are rounded to the nearest cent:

>>>fromdecimalimport*>>>round(Decimal('0.70')*Decimal('1.05'),2)Decimal('0.74')>>>round(.70*1.05,2)0.73

TheDecimal result keeps a trailing zero, automaticallyinferring four place significance from multiplicands with two placesignificance. Decimal reproduces mathematics as done by hand and avoidsissues that can arise when binary floating point cannot exactly representdecimal quantities.

Exact representation enables theDecimal class to performmodulo calculations and equality tests that are unsuitable for binary floatingpoint:

>>>Decimal('1.00')%Decimal('.10')Decimal('0.00')>>>1.00%0.100.09999999999999995>>>sum([Decimal('0.1')]*10)==Decimal('1.0')True>>>0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1+0.1==1.0False

Thedecimal module provides arithmetic with as much precision as needed:

>>>getcontext().prec=36>>>Decimal(1)/Decimal(7)Decimal('0.142857142857142857142857142857142857')