Abstract

This PEP proposes adding two new keywords to Python, ‘synchronize’and ‘asynchronize’.

Pronouncement

This PEP is rejected in favor ofPEP 343.

The ‘synchronize’ Keyword

The concept of code synchronization in Python is too low-level.To synchronize code a programmer must be aware of the details ofthe following pseudo-code pattern:

initialize_lock()...acquire_lock()try:change_shared_data()finally:release_lock()

This synchronized block pattern is not the only pattern (morediscussed below) but it is very common. This PEP proposesreplacing the above code with the following equivalent:

synchronize:change_shared_data()

The advantages of this scheme are simpler syntax and less room foruser error. Currently users are required to write code aboutacquiring and releasing thread locks in ‘try/finally’ blocks;errors in this code can cause notoriously difficult concurrentthread locking issues.

The ‘asynchronize’ Keyword

While executing a ‘synchronize’ block of code a programmer maywant to “drop back” to running asynchronously momentarily to runblocking input/output routines or something else that might take anindeterminate amount of time and does not require synchronization.This code usually follows the pattern:

initialize_lock()...acquire_lock()try:change_shared_data()release_lock()# become asyncdo_blocking_io()acquire_lock()# sync againchange_shared_data2()finally:release_lock()

The asynchronous section of the code is not very obvious visually,so it is marked up with comments. Using the proposed‘asynchronize’ keyword this code becomes much cleaner, easier tounderstand, and less prone to error:

synchronize:change_shared_data()asynchronize:do_blocking_io()change_shared_data2()

Encountering an ‘asynchronize’ keyword inside a non-synchronizedblock can raise either an error or issue a warning (as all codeblocks are implicitly asynchronous anyway). It is important tonote that the above example isnot the same as:

synchronize:change_shared_data()do_blocking_io()synchronize:change_shared_data2()

Because both synchronized blocks of code may be running inside thesame iteration of a loop, Consider:

whilein_main_loop():synchronize:change_shared_data()asynchronize:do_blocking_io()change_shared_data2()

Many threads may be looping through this code. Without the‘asynchronize’ keyword one thread cannot stay in the loop andrelease the lock at the same time while blocking IO is going on.This pattern of releasing locks inside a main loop to do blockingIO is used extensively inside the CPython interpreter itself.

Synchronization Targets

As proposed the ‘synchronize’ and ‘asynchronize’ keywordssynchronize a block of code. However programmers may want tospecify a target object that threads synchronize on. Any objectcan be a synchronization target.

Consider a two-way queue object: two different objects are used bythe same ‘synchronize’ code block to synchronize both queuesseparately in the ‘get’ method:

classTwoWayQueue:def__init__(self):self.front=[]self.rear=[]defputFront(self,item):self.put(item,self.front)defgetFront(self):item=self.get(self.front)returnitemdefputRear(self,item):self.put(item,self.rear)defgetRear(self):item=self.get(self.rear)returnitemdefput(self,item,queue):synchronizequeue:queue.append(item)defget(self,queue):synchronizequeue:item=queue[0]delqueue[0]returnitem

Here is the equivalent code in Python as it is now without a‘synchronize’ keyword:

importthreadclassLockableQueue:def__init__(self):self.queue=[]self.lock=thread.allocate_lock()classTwoWayQueue:def__init__(self):self.front=LockableQueue()self.rear=LockableQueue()defputFront(self,item):self.put(item,self.front)defgetFront(self):item=self.get(self.front)returnitemdefputRear(self,item):self.put(item,self.rear)defgetRear(self):item=self.get(self.rear)returnitemdefput(self,item,queue):queue.lock.acquire()try:queue.append(item)finally:queue.lock.release()defget(self,queue):queue.lock.acquire()try:item=queue[0]delqueue[0]returnitemfinally:queue.lock.release()

The last example had to define an extra class to associate a lockwith the queue where the first example the ‘synchronize’ keyworddoes this association internally and transparently.

Other Patterns that Synchronize

There are some situations where the ‘synchronize’ and‘asynchronize’ keywords cannot entirely replace the use of lockmethods likeacquire andrelease. Some examples are if theprogrammer wants to provide arguments foracquire or if a lockis acquired in one code block but released in another, as shownbelow.

Here is a class from Zope modified to use both the ‘synchronize’and ‘asynchronize’ keywords and also uses a pool of explicit locksthat are acquired and released in different code blocks and thusdon’t use ‘synchronize’:

importthreadfromZServerPublisherimportZServerPublisherclassZRendevous:def__init__(self,n=1):pool=[]self._lists=pool,[],[]synchronize:whilen>0:l=thread.allocate_lock()l.acquire()pool.append(l)thread.start_new_thread(ZServerPublisher,(self.accept,))n=n-1defaccept(self):synchronize:pool,requests,ready=self._listswhilenotrequests:l=pool[-1]delpool[-1]ready.append(l)asynchronize:l.acquire()pool.append(l)r=requests[0]delrequests[0]returnrdefhandle(self,name,request,response):synchronize:pool,requests,ready=self._listsrequests.append((name,request,response))ifready:l=ready[-1]delready[-1]l.release()

Here is the original class as found in the‘Zope/ZServer/PubCore/ZRendevous.py’ module. The “convenience” ofthe ‘_a’ and ‘_r’ shortcut names obscure the code:

importthreadfromZServerPublisherimportZServerPublisherclassZRendevous:def__init__(self,n=1):sync=thread.allocate_lock()self._a=sync.acquireself._r=sync.releasepool=[]self._lists=pool,[],[]self._a()try:whilen>0:l=thread.allocate_lock()l.acquire()pool.append(l)thread.start_new_thread(ZServerPublisher,(self.accept,))n=n-1finally:self._r()defaccept(self):self._a()try:pool,requests,ready=self._listswhilenotrequests:l=pool[-1]delpool[-1]ready.append(l)self._r()l.acquire()self._a()pool.append(l)r=requests[0]delrequests[0]returnrfinally:self._r()defhandle(self,name,request,response):self._a()try:pool,requests,ready=self._listsrequests.append((name,request,response))ifready:l=ready[-1]delready[-1]l.release()finally:self._r()

In particular the asynchronize section of theaccept method isnot very obvious. To beginner programmers, ‘synchronize’ and‘asynchronize’ remove many of the problems encountered whenjuggling multipleacquire andrelease methods on differentlocks in differenttry/finally blocks.

Formal Syntax

Python syntax is defined in a modified BNF grammar notationdescribed in the Python Language Reference[1]. This sectiondescribes the proposed synchronization syntax using this grammar:

synchronize_stmt:'synchronize'[test]':'suiteasynchronize_stmt:'asynchronize'[test]':'suitecompound_stmt:...|synchronized_stmt|asynchronize_stmt

(The ‘…’ indicates other compound statements elided).

Proposed Implementation

The author of this PEP has not explored an implementation yet.There are several implementation issues that must be resolved.The main implementation issue is what exactly gets locked andunlocked during a synchronized block.

During an unqualified synchronized block (the use of the‘synchronize’ keyword without a target argument) a lock could becreated and associated with the synchronized code block object.Any threads that are to execute the block must first acquire thecode block lock.

When an ‘asynchronize’ keyword is encountered in a ‘synchronize’block the code block lock is unlocked before the inner block isexecuted and re-locked when the inner block terminates.

When a synchronized block target is specified the object isassociated with a lock. How this is implemented cleanly isprobably the highest risk of this proposal. Java Virtual Machinestypically associate a special hidden lock object with targetobject and use it to synchronized the block around the targetonly.

Backward Compatibility

Backward compatibility is solved with the newfrom__future__Python syntax (PEP 236), and the new warning framework (PEP 230)to evolve thePython language into phasing out any conflicting names that usethe new keywords ‘synchronize’ and ‘asynchronize’. To use thesyntax now, a developer could use the statement:

from__future__importthreadsync# or whatever

In addition, any code that uses the keyword ‘synchronize’ or‘asynchronize’ as an identifier will be issued a warning fromPython. After the appropriate period of time, the syntax wouldbecome standard, the above import statement would do nothing, andany identifiers named ‘synchronize’ or ‘asynchronize’ would raisean exception.

PEP 310 Reliable Acquisition/Release Pairs

PEP 310 proposes the ‘with’ keyword that can serve the samefunction as ‘synchronize’ (but no facility for ‘asynchronize’).The pattern:

initialize_lock()withthe_lock:change_shared_data()

is equivalent to the proposed:

synchronizethe_lock:change_shared_data()

PEP 310 must synchronize on an existing lock, while this PEPproposes that unqualified ‘synchronize’ statements synchronize ona global, internal, transparent lock in addition to qualified‘synchronize’ statements. The ‘with’ statement also requires lockinitialization, while the ‘synchronize’ statement can synchronizeon any target objectincluding locks.

While limited in this fashion, the ‘with’ statement is moreabstract and serves more purposes than synchronization. Forexample, transactions could be used with the ‘with’ keyword:

initialize_transaction()withmy_transaction:do_in_transaction()# when the block terminates, the transaction is committed.

The ‘synchronize’ and ‘asynchronize’ keywords cannot serve this orany other general acquire/release pattern other than threadsynchronization.

How Java Does It

Java defines a ‘synchronized’ keyword (note the grammatical tensedifferent between the Java keyword and this PEP’s ‘synchronize’)which must be qualified on any object. The syntax is:

synchronized(Expression)Block

Expression must yield a valid object (null raises an error andexceptions during ‘Expression’ terminate the ‘synchronized’ blockfor the same reason) upon which ‘Block’ is synchronized.

How Jython Does It

Jython uses a ‘synchronize’ class with the static method‘make_synchronized’ that accepts one callable argument and returnsa newly created, synchronized, callable “wrapper” around theargument.

Summary of Proposed Changes to Python

Adding new ‘synchronize’ and ‘asynchronize’ keywords to thelanguage.

Risks

This PEP proposes adding two keywords to the Python language. Thismay break code.

There is no implementation to test.

It’s not the most important problem facing Python programmerstoday (although it is a fairly notorious one).

The equivalent Java keyword is the past participle ‘synchronized’.This PEP proposes the present tense, ‘synchronize’ as being morein spirit with Python (there being less distinction betweencompile-time and run-time in Python than Java).

Dissenting Opinion

This PEP has not been discussed on python-dev.

References

Copyright

This document has been placed in the public domain.