9.1.A Word About Names and Objects¶

Objects have individuality, and multiple names (in multiple scopes) can be boundto the same object. This is known as aliasing in other languages. This isusually not appreciated on a first glance at Python, and can be safely ignoredwhen dealing with immutable basic types (numbers, strings, tuples). However,aliasing has a possibly surprising effect on the semantics of Python codeinvolving mutable objects such as lists, dictionaries, and most other types.This is usually used to the benefit of the program, since aliases behave likepointers in some respects. For example, passing an object is cheap since only apointer is passed by the implementation; and if a function modifies an objectpassed as an argument, the caller will see the change — this eliminates theneed for two different argument passing mechanisms as in Pascal.

9.2.Python Scopes and Namespaces¶

Before introducing classes, I first have to tell you something about Python’sscope rules. Class definitions play some neat tricks with namespaces, and youneed to know how scopes and namespaces work to fully understand what’s going on.Incidentally, knowledge about this subject is useful for any advanced Pythonprogrammer.

Let’s begin with some definitions.

Anamespace is a mapping from names to objects. Most namespaces are currentlyimplemented as Python dictionaries, but that’s normally not noticeable in anyway (except for performance), and it may change in the future. Examples ofnamespaces are: the set of built-in names (containing functions such asabs(), andbuilt-in exception names); the global names in a module; and the local names ina function invocation. In a sense the set of attributes of an object also forma namespace. The important thing to know about namespaces is that there isabsolutely no relation between names in different namespaces; for instance, twodifferent modules may both define a functionmaximize without confusion —users of the modules must prefix it with the module name.

By the way, I use the wordattribute for any name following a dot — forexample, in the expressionz.real,real is an attribute of the objectz. Strictly speaking, references to names in modules are attributereferences: in the expressionmodname.funcname,modname is a moduleobject andfuncname is an attribute of it. In this case there happens to bea straightforward mapping between the module’s attributes and the global namesdefined in the module: they share the same namespace![1]

Attributes may be read-only or writable. In the latter case, assignment toattributes is possible. Module attributes are writable: you can writemodname.the_answer=42. Writable attributes may also be deleted with thedel statement. For example,delmodname.the_answer will removethe attributethe_answer from the object named bymodname.

Namespaces are created at different moments and have different lifetimes. Thenamespace containing the built-in names is created when the Python interpreterstarts up, and is never deleted. The global namespace for a module is createdwhen the module definition is read in; normally, module namespaces also lastuntil the interpreter quits. The statements executed by the top-levelinvocation of the interpreter, either read from a script file or interactively,are considered part of a module called__main__, so they have their ownglobal namespace. (The built-in names actually also live in a module; this iscalledbuiltins.)

The local namespace for a function is created when the function is called, anddeleted when the function returns or raises an exception that is not handledwithin the function. (Actually, forgetting would be a better way to describewhat actually happens.) Of course, recursive invocations each have their ownlocal namespace.

Ascope is a textual region of a Python program where a namespace is directlyaccessible. “Directly accessible” here means that an unqualified reference to aname attempts to find the name in the namespace.

Although scopes are determined statically, they are used dynamically. At anytime during execution, there are 3 or 4 nested scopes whose namespaces aredirectly accessible:

• the innermost scope, which is searched first, contains the local names

• the scopes of any enclosing functions, which are searched starting with thenearest enclosing scope, contain non-local, but also non-global names

• the next-to-last scope contains the current module’s global names

• the outermost scope (searched last) is the namespace containing built-in names

If a name is declared global, then all references and assignments go directly tothe next-to-last scope containing the module’s global names. To rebind variablesfound outside of the innermost scope, thenonlocal statement can beused; if not declared nonlocal, those variables are read-only (an attempt towrite to such a variable will simply create anew local variable in theinnermost scope, leaving the identically named outer variable unchanged).

Usually, the local scope references the local names of the (textually) currentfunction. Outside functions, the local scope references the same namespace asthe global scope: the module’s namespace. Class definitions place yet anothernamespace in the local scope.

It is important to realize that scopes are determined textually: the globalscope of a function defined in a module is that module’s namespace, no matterfrom where or by what alias the function is called. On the other hand, theactual search for names is done dynamically, at run time — however, thelanguage definition is evolving towards static name resolution, at “compile”time, so don’t rely on dynamic name resolution! (In fact, local variables arealready determined statically.)

A special quirk of Python is that – if noglobal ornonlocalstatement is in effect – assignments to names always go into the innermost scope.Assignments do not copy data — they just bind names to objects. The same is truefor deletions: the statementdelx removes the binding ofx from thenamespace referenced by the local scope. In fact, all operations that introducenew names use the local scope: in particular,import statements andfunction definitions bind the module or function name in the local scope.

Theglobal statement can be used to indicate that particularvariables live in the global scope and should be rebound there; thenonlocal statement indicates that particular variables live inan enclosing scope and should be rebound there.

defscope_test():defdo_local():spam="local spam"defdo_nonlocal():nonlocalspamspam="nonlocal spam"defdo_global():globalspamspam="global spam"spam="test spam"do_local()print("After local assignment:",spam)do_nonlocal()print("After nonlocal assignment:",spam)do_global()print("After global assignment:",spam)scope_test()print("In global scope:",spam)

After local assignment: test spamAfter nonlocal assignment: nonlocal spamAfter global assignment: nonlocal spamIn global scope: global spam

9.3.A First Look at Classes¶

Classes introduce a little bit of new syntax, three new object types, and somenew semantics.

classClassName:<statement-1>...<statement-N>

classMyClass:"""A simple example class"""i=12345deff(self):return'hello world'

x=MyClass()

def__init__(self):self.data=[]

x=MyClass()

>>>classComplex:...def__init__(self,realpart,imagpart):...self.r=realpart...self.i=imagpart...>>>x=Complex(3.0,-4.5)>>>x.r,x.i(3.0, -4.5)

x.counter=1whilex.counter<10:x.counter=x.counter*2print(x.counter)delx.counter

x.f()

xf=x.fwhileTrue:print(xf())

classDog:kind='canine'# class variable shared by all instancesdef__init__(self,name):self.name=name# instance variable unique to each instance>>>d=Dog('Fido')>>>e=Dog('Buddy')>>>d.kind# shared by all dogs'canine'>>>e.kind# shared by all dogs'canine'>>>d.name# unique to d'Fido'>>>e.name# unique to e'Buddy'

classDog:tricks=[]# mistaken use of a class variabledef__init__(self,name):self.name=namedefadd_trick(self,trick):self.tricks.append(trick)>>>d=Dog('Fido')>>>e=Dog('Buddy')>>>d.add_trick('roll over')>>>e.add_trick('play dead')>>>d.tricks# unexpectedly shared by all dogs['roll over','play dead']

classDog:def__init__(self,name):self.name=nameself.tricks=[]# creates a new empty list for each dogdefadd_trick(self,trick):self.tricks.append(trick)>>>d=Dog('Fido')>>>e=Dog('Buddy')>>>d.add_trick('roll over')>>>e.add_trick('play dead')>>>d.tricks['roll over']>>>e.tricks['play dead']

9.4.Random Remarks¶

If the same attribute name occurs in both an instance and in a class,then attribute lookup prioritizes the instance:

>>>classWarehouse:...purpose='storage'...region='west'...>>>w1=Warehouse()>>>print(w1.purpose,w1.region)storage west>>>w2=Warehouse()>>>w2.region='east'>>>print(w2.purpose,w2.region)storage east

Data attributes may be referenced by methods as well as by ordinary users(“clients”) of an object. In other words, classes are not usable to implementpure abstract data types. In fact, nothing in Python makes it possible toenforce data hiding — it is all based upon convention. (On the other hand,the Python implementation, written in C, can completely hide implementationdetails and control access to an object if necessary; this can be used byextensions to Python written in C.)

Clients should use data attributes with care — clients may mess up invariantsmaintained by the methods by stamping on their data attributes. Note thatclients may add data attributes of their own to an instance object withoutaffecting the validity of the methods, as long as name conflicts are avoided —again, a naming convention can save a lot of headaches here.

There is no shorthand for referencing data attributes (or other methods!) fromwithin methods. I find that this actually increases the readability of methods:there is no chance of confusing local variables and instance variables whenglancing through a method.

Often, the first argument of a method is calledself. This is nothing morethan a convention: the nameself has absolutely no special meaning toPython. Note, however, that by not following the convention your code may beless readable to other Python programmers, and it is also conceivable that aclass browser program might be written that relies upon such a convention.

Any function object that is a class attribute defines a method for instances ofthat class. It is not necessary that the function definition is textuallyenclosed in the class definition: assigning a function object to a localvariable in the class is also ok. For example:

# Function defined outside the classdeff1(self,x,y):returnmin(x,x+y)classC:f=f1defg(self):return'hello world'h=g

Nowf,g andh are all attributes of classC that refer tofunction objects, and consequently they are all methods of instances ofC —h being exactly equivalent tog. Note that this practiceusually only serves to confuse the reader of a program.

Methods may call other methods by using method attributes of theselfargument:

classBag:def__init__(self):self.data=[]defadd(self,x):self.data.append(x)defaddtwice(self,x):self.add(x)self.add(x)

Methods may reference global names in the same way as ordinary functions. Theglobal scope associated with a method is the module containing itsdefinition. (A class is never used as a global scope.) While onerarely encounters a good reason for using global data in a method, there aremany legitimate uses of the global scope: for one thing, functions and modulesimported into the global scope can be used by methods, as well as functions andclasses defined in it. Usually, the class containing the method is itselfdefined in this global scope, and in the next section we’ll find some goodreasons why a method would want to reference its own class.

Each value is an object, and therefore has aclass (also called itstype).It is stored asobject.__class__.

9.5.Inheritance¶

Of course, a language feature would not be worthy of the name “class” withoutsupporting inheritance. The syntax for a derived class definition looks likethis:

classDerivedClassName(BaseClassName):<statement-1>...<statement-N>

The nameBaseClassName must be defined in anamespace accessible from the scope containing thederived class definition. In place of a base class name, other arbitraryexpressions are also allowed. This can be useful, for example, when the baseclass is defined in another module:

classDerivedClassName(modname.BaseClassName):

Execution of a derived class definition proceeds the same as for a base class.When the class object is constructed, the base class is remembered. This isused for resolving attribute references: if a requested attribute is not foundin the class, the search proceeds to look in the base class. This rule isapplied recursively if the base class itself is derived from some other class.

There’s nothing special about instantiation of derived classes:DerivedClassName() creates a new instance of the class. Method referencesare resolved as follows: the corresponding class attribute is searched,descending down the chain of base classes if necessary, and the method referenceis valid if this yields a function object.

Derived classes may override methods of their base classes. Because methodshave no special privileges when calling other methods of the same object, amethod of a base class that calls another method defined in the same base classmay end up calling a method of a derived class that overrides it. (For C++programmers: all methods in Python are effectivelyvirtual.)

An overriding method in a derived class may in fact want to extend rather thansimply replace the base class method of the same name. There is a simple way tocall the base class method directly: just callBaseClassName.methodname(self,arguments). This is occasionally useful to clients as well. (Note that thisonly works if the base class is accessible asBaseClassName in the globalscope.)

Python has two built-in functions that work with inheritance:

• Useisinstance() to check an instance’s type:isinstance(obj,int)will beTrue only ifobj.__class__ isint or some classderived fromint.

• Useissubclass() to check class inheritance:issubclass(bool,int)isTrue sincebool is a subclass ofint. However,issubclass(float,int) isFalse sincefloat is not asubclass ofint.

classDerivedClassName(Base1,Base2,Base3):<statement-1>...<statement-N>

9.6.Private Variables¶

“Private” instance variables that cannot be accessed except from inside anobject don’t exist in Python. However, there is a convention that is followedby most Python code: a name prefixed with an underscore (e.g._spam) shouldbe treated as a non-public part of the API (whether it is a function, a methodor a data member). It should be considered an implementation detail and subjectto change without notice.

Since there is a valid use-case for class-private members (namely to avoid nameclashes of names with names defined by subclasses), there is limited support forsuch a mechanism, calledname mangling. Any identifier of the form__spam (at least two leading underscores, at most one trailing underscore)is textually replaced with_classname__spam, whereclassname is thecurrent class name with leading underscore(s) stripped. This mangling is donewithout regard to the syntactic position of the identifier, as long as itoccurs within the definition of a class.

Name mangling is helpful for letting subclasses override methods withoutbreaking intraclass method calls. For example:

classMapping:def__init__(self,iterable):self.items_list=[]self.__update(iterable)defupdate(self,iterable):foriteminiterable:self.items_list.append(item)__update=update# private copy of original update() methodclassMappingSubclass(Mapping):defupdate(self,keys,values):# provides new signature for update()# but does not break __init__()foriteminzip(keys,values):self.items_list.append(item)

The above example would work even ifMappingSubclass were to introduce a__update identifier since it is replaced with_Mapping__update in theMapping class and_MappingSubclass__update in theMappingSubclassclass respectively.

Note that the mangling rules are designed mostly to avoid accidents; it still ispossible to access or modify a variable that is considered private. This caneven be useful in special circumstances, such as in the debugger.

Notice that code passed toexec() oreval() does not consider theclassname of the invoking class to be the current class; this is similar to theeffect of theglobal statement, the effect of which is likewise restrictedto code that is byte-compiled together. The same restriction applies togetattr(),setattr() anddelattr(), as well as when referencing__dict__ directly.

9.7.Odds and Ends¶

Sometimes it is useful to have a data type similar to the Pascal “record” or C“struct”, bundling together a few named data items. The idiomatic approachis to usedataclasses for this purpose:

fromdataclassesimportdataclass@dataclassclassEmployee:name:strdept:strsalary:int

>>>john=Employee('john','computer lab',1000)>>>john.dept'computer lab'>>>john.salary1000

A piece of Python code that expects a particular abstract data type can often bepassed a class that emulates the methods of that data type instead. Forinstance, if you have a function that formats some data from a file object, youcan define a class with methodsread() andreadline() that get thedata from a string buffer instead, and pass it as an argument.

Instance method objects have attributes, too:m.__self__ is the instanceobject with the methodm(), andm.__func__ isthefunction objectcorresponding to the method.

9.8.Iterators¶

By now you have probably noticed that most container objects can be looped overusing afor statement:

forelementin[1,2,3]:print(element)forelementin(1,2,3):print(element)forkeyin{'one':1,'two':2}:print(key)forcharin"123":print(char)forlineinopen("myfile.txt"):print(line,end='')

This style of access is clear, concise, and convenient. The use of iteratorspervades and unifies Python. Behind the scenes, thefor statementcallsiter() on the container object. The function returns an iteratorobject that defines the method__next__() which accesseselements in the container one at a time. When there are no more elements,__next__() raises aStopIteration exception which tells thefor loop to terminate. You can call the__next__() methodusing thenext() built-in function; this example shows how it all works:

>>>s='abc'>>>it=iter(s)>>>it<str_iterator object at 0x10c90e650>>>>next(it)'a'>>>next(it)'b'>>>next(it)'c'>>>next(it)Traceback (most recent call last):  File"<stdin>", line1, in<module>next(it)StopIteration

Having seen the mechanics behind the iterator protocol, it is easy to additerator behavior to your classes. Define an__iter__() method whichreturns an object with a__next__() method. If the classdefines__next__(), then__iter__() can just returnself:

classReverse:"""Iterator for looping over a sequence backwards."""def__init__(self,data):self.data=dataself.index=len(data)def__iter__(self):returnselfdef__next__(self):ifself.index==0:raiseStopIterationself.index=self.index-1returnself.data[self.index]

>>>rev=Reverse('spam')>>>iter(rev)<__main__.Reverse object at 0x00A1DB50>>>>forcharinrev:...print(char)...maps

9.9.Generators¶

Generators are a simple and powerful tool for creating iterators. Theyare written like regular functions but use theyield statementwhenever they want to return data. Each timenext() is called on it, thegenerator resumes where it left off (it remembers all the data values and whichstatement was last executed). An example shows that generators can be triviallyeasy to create:

defreverse(data):forindexinrange(len(data)-1,-1,-1):yielddata[index]

>>>forcharinreverse('golf'):...print(char)...flog

Anything that can be done with generators can also be done with class-basediterators as described in the previous section. What makes generators socompact is that the__iter__() and__next__() methodsare created automatically.

Another key feature is that the local variables and execution state areautomatically saved between calls. This made the function easier to write andmuch more clear than an approach using instance variables likeself.indexandself.data.

In addition to automatic method creation and saving program state, whengenerators terminate, they automatically raiseStopIteration. Incombination, these features make it easy to create iterators with no more effortthan writing a regular function.

9.10.Generator Expressions¶

Some simple generators can be coded succinctly as expressions using a syntaxsimilar to list comprehensions but with parentheses instead of square brackets.These expressions are designed for situations where the generator is used rightaway by an enclosing function. Generator expressions are more compact but lessversatile than full generator definitions and tend to be more memory friendlythan equivalent list comprehensions.

Examples:

>>>sum(i*iforiinrange(10))# sum of squares285>>>xvec=[10,20,30]>>>yvec=[7,5,3]>>>sum(x*yforx,yinzip(xvec,yvec))# dot product260>>>unique_words=set(wordforlineinpageforwordinline.split())>>>valedictorian=max((student.gpa,student.name)forstudentingraduates)>>>data='golf'>>>list(data[i]foriinrange(len(data)-1,-1,-1))['f', 'l', 'o', 'g']

Footnotes

Except for one thing. Module objects have a secret read-only attribute called__dict__ which returns the dictionary used to implement the module’snamespace; the name__dict__ is an attribute but not a global name.Obviously, using this violates the abstraction of namespace implementation, andshould be restricted to things like post-mortem debuggers.