12.1.1.Relationship to other Python modules¶

12.1.2.Data stream format¶

The data format used bypickle is Python-specific. This has theadvantage that there are no restrictions imposed by external standards such asJSON or XDR (which can’t represent pointer sharing); however it means thatnon-Python programs may not be able to reconstruct pickled Python objects.

By default, thepickle data format uses a relatively compact binaryrepresentation. If you need optimal size characteristics, you can efficientlycompress pickled data.

The modulepickletools contains tools for analyzing data streamsgenerated bypickle.pickletools source code has extensivecomments about opcodes used by pickle protocols.

There are currently 5 different protocols which can be used for pickling.The higher the protocol used, the more recent the version of Python neededto read the pickle produced.

• Protocol version 0 is the original “human-readable” protocol and isbackwards compatible with earlier versions of Python.

• Protocol version 1 is an old binary format which is also compatible withearlier versions of Python.

• Protocol version 2 was introduced in Python 2.3. It provides much moreefficient pickling ofnew-style classes. Refer toPEP 307 forinformation about improvements brought by protocol 2.

• Protocol version 3 was added in Python 3.0. It has explicit support forbytes objects and cannot be unpickled by Python 2.x. This isthe default protocol, and the recommended protocol when compatibility withother Python 3 versions is required.

• Protocol version 4 was added in Python 3.4. It adds support for very largeobjects, pickling more kinds of objects, and some data formatoptimizations. Refer toPEP 3154 for information about improvementsbrought by protocol 4.

12.1.3.Module Interface¶

To serialize an object hierarchy, you simply call thedumps() function.Similarly, to de-serialize a data stream, you call theloads() function.However, if you want more control over serialization and de-serialization,you can create aPickler or anUnpickler object, respectively.

Thepickle module provides the following constants:

pickle.HIGHEST_PROTOCOL¶

An integer, the highestprotocol versionavailable. This value can be passed as aprotocol value to functionsdump() anddumps() as well as thePicklerconstructor.

pickle.DEFAULT_PROTOCOL¶

An integer, the defaultprotocol version usedfor pickling. May be less thanHIGHEST_PROTOCOL. Currently thedefault protocol is 3, a new protocol designed for Python 3.

Thepickle module provides the following functions to make the picklingprocess more convenient:

pickle.dump(obj,file,protocol=None,*,fix_imports=True)¶

Write a pickled representation ofobj to the openfile objectfile.This is equivalent toPickler(file,protocol).dump(obj).

The optionalprotocol argument, an integer, tells the pickler to usethe given protocol; supported protocols are 0 toHIGHEST_PROTOCOL.If not specified, the default isDEFAULT_PROTOCOL. If a negativenumber is specified,HIGHEST_PROTOCOL is selected.

Thefile argument must have a write() method that accepts a single bytesargument. It can thus be an on-disk file opened for binary writing, anio.BytesIO instance, or any other custom object that meets thisinterface.

Iffix_imports is true andprotocol is less than 3, pickle will try tomap the new Python 3 names to the old module names used in Python 2, sothat the pickle data stream is readable with Python 2.

pickle.dumps(obj,protocol=None,*,fix_imports=True)¶

Return the pickled representation of the object as abytes object,instead of writing it to a file.

Argumentsprotocol andfix_imports have the same meaning as indump().

pickle.load(file,*,fix_imports=True,encoding="ASCII",errors="strict")¶

Read a pickled object representation from the openfile objectfile and return the reconstituted object hierarchy specified therein.This is equivalent toUnpickler(file).load().

The protocol version of the pickle is detected automatically, so noprotocol argument is needed. Bytes past the pickled object’srepresentation are ignored.

The argumentfile must have two methods, a read() method that takes aninteger argument, and a readline() method that requires no arguments. Bothmethods should return bytes. Thusfile can be an on-disk file opened forbinary reading, anio.BytesIO object, or any other custom objectthat meets this interface.

Optional keyword arguments arefix_imports,encoding anderrors,which are used to control compatibility support for pickle stream generatedby Python 2. Iffix_imports is true, pickle will try to map the oldPython 2 names to the new names used in Python 3. Theencoding anderrors tell pickle how to decode 8-bit string instances pickled by Python2; these default to ‘ASCII’ and ‘strict’, respectively. Theencoding canbe ‘bytes’ to read these 8-bit string instances as bytes objects.Usingencoding='latin1' is required for unpickling NumPy arrays andinstances ofdatetime,date andtime pickled by Python 2.

pickle.loads(bytes_object,*,fix_imports=True,encoding="ASCII",errors="strict")¶

Read a pickled object hierarchy from abytes object and return thereconstituted object hierarchy specified therein.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed. Bytes past the pickled object’srepresentation are ignored.

Optional keyword arguments arefix_imports,encoding anderrors,which are used to control compatibility support for pickle stream generatedby Python 2. Iffix_imports is true, pickle will try to map the oldPython 2 names to the new names used in Python 3. Theencoding anderrors tell pickle how to decode 8-bit string instances pickled by Python2; these default to ‘ASCII’ and ‘strict’, respectively. Theencoding canbe ‘bytes’ to read these 8-bit string instances as bytes objects.Usingencoding='latin1' is required for unpickling NumPy arrays andinstances ofdatetime,date andtime pickled by Python 2.

Thepickle module defines three exceptions:

exceptionpickle.PickleError¶

Common base class for the other pickling exceptions. It inheritsException.

exceptionpickle.PicklingError¶

Error raised when an unpicklable object is encountered byPickler.It inheritsPickleError.

Refer toWhat can be pickled and unpickled? to learn what kinds of objects can bepickled.

exceptionpickle.UnpicklingError¶

Error raised when there is a problem unpickling an object, such as a datacorruption or a security violation. It inheritsPickleError.

Note that other exceptions may also be raised during unpickling, including(but not necessarily limited to) AttributeError, EOFError, ImportError, andIndexError.

Thepickle module exports two classes,Pickler andUnpickler:

classpickle.Pickler(file,protocol=None,*,fix_imports=True)¶

This takes a binary file for writing a pickle data stream.

The optionalprotocol argument, an integer, tells the pickler to usethe given protocol; supported protocols are 0 toHIGHEST_PROTOCOL.If not specified, the default isDEFAULT_PROTOCOL. If a negativenumber is specified,HIGHEST_PROTOCOL is selected.

Thefile argument must have a write() method that accepts a single bytesargument. It can thus be an on-disk file opened for binary writing, anio.BytesIO instance, or any other custom object that meets thisinterface.

Iffix_imports is true andprotocol is less than 3, pickle will try tomap the new Python 3 names to the old module names used in Python 2, sothat the pickle data stream is readable with Python 2.

dump(obj)¶

Write a pickled representation ofobj to the open file object given inthe constructor.

persistent_id(obj)¶

Do nothing by default. This exists so a subclass can override it.

Ifpersistent_id() returnsNone,obj is pickled as usual. Anyother value causesPickler to emit the returned value as apersistent ID forobj. The meaning of this persistent ID should bedefined byUnpickler.persistent_load(). Note that the valuereturned bypersistent_id() cannot itself have a persistent ID.

SeePersistence of External Objects for details and examples of uses.

dispatch_table¶

A pickler object’s dispatch table is a registry ofreductionfunctions of the kind which can be declared usingcopyreg.pickle(). It is a mapping whose keys are classesand whose values are reduction functions. A reduction functiontakes a single argument of the associated class and shouldconform to the same interface as a__reduce__()method.

By default, a pickler object will not have adispatch_table attribute, and it will instead use theglobal dispatch table managed by thecopyreg module.However, to customize the pickling for a specific pickler objectone can set thedispatch_table attribute to a dict-likeobject. Alternatively, if a subclass ofPickler has adispatch_table attribute then this will be used as thedefault dispatch table for instances of that class.

SeeDispatch Tables for usage examples.

New in version 3.3.

fast¶

Deprecated. Enable fast mode if set to a true value. The fast modedisables the usage of memo, therefore speeding the pickling process by notgenerating superfluous PUT opcodes. It should not be used withself-referential objects, doing otherwise will causePickler torecurse infinitely.

Usepickletools.optimize() if you need more compact pickles.

classpickle.Unpickler(file,*,fix_imports=True,encoding="ASCII",errors="strict")¶

This takes a binary file for reading a pickle data stream.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed.

The argumentfile must have two methods, a read() method that takes aninteger argument, and a readline() method that requires no arguments. Bothmethods should return bytes. Thusfile can be an on-disk file objectopened for binary reading, anio.BytesIO object, or any othercustom object that meets this interface.

Optional keyword arguments arefix_imports,encoding anderrors,which are used to control compatibility support for pickle stream generatedby Python 2. Iffix_imports is true, pickle will try to map the oldPython 2 names to the new names used in Python 3. Theencoding anderrors tell pickle how to decode 8-bit string instances pickled by Python2; these default to ‘ASCII’ and ‘strict’, respectively. Theencoding canbe ‘bytes’ to read these 8-bit string instances as bytes objects.

load()¶

Read a pickled object representation from the open file object given inthe constructor, and return the reconstituted object hierarchy specifiedtherein. Bytes past the pickled object’s representation are ignored.

persistent_load(pid)¶

Raise anUnpicklingError by default.

If defined,persistent_load() should return the object specified bythe persistent IDpid. If an invalid persistent ID is encountered, anUnpicklingError should be raised.

SeePersistence of External Objects for details and examples of uses.

find_class(module,name)¶

Importmodule if necessary and return the object calledname from it,where themodule andname arguments arestr objects. Note,unlike its name suggests,find_class() is also used for findingfunctions.

Subclasses may override this to gain control over what type of objects andhow they can be loaded, potentially reducing security risks. Refer toRestricting Globals for details.

12.1.4.What can be pickled and unpickled?¶

The following types can be pickled:

• None,True, andFalse

• integers, floating point numbers, complex numbers

• strings, bytes, bytearrays

• tuples, lists, sets, and dictionaries containing only picklable objects

• functions defined at the top level of a module (usingdef, notlambda)

• built-in functions defined at the top level of a module

• classes that are defined at the top level of a module

• instances of such classes whose__dict__ or the result ofcalling__getstate__() is picklable (see sectionPickling Class Instances fordetails).

Attempts to pickle unpicklable objects will raise thePicklingErrorexception; when this happens, an unspecified number of bytes may have alreadybeen written to the underlying file. Trying to pickle a highly recursive datastructure may exceed the maximum recursion depth, aRecursionError will beraised in this case. You can carefully raise this limit withsys.setrecursionlimit().

Note that functions (built-in and user-defined) are pickled by “fully qualified”name reference, not by value.2 This means that only the function name ispickled, along with the name of the module the function is defined in. Neitherthe function’s code, nor any of its function attributes are pickled. Thus thedefining module must be importable in the unpickling environment, and the modulemust contain the named object, otherwise an exception will be raised.3

Similarly, classes are pickled by named reference, so the same restrictions inthe unpickling environment apply. Note that none of the class’s code or data ispickled, so in the following example the class attributeattr is notrestored in the unpickling environment:

classFoo:attr='A class attribute'picklestring=pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined inthe top level of a module.

Similarly, when class instances are pickled, their class’s code and data are notpickled along with them. Only the instance data are pickled. This is done onpurpose, so you can fix bugs in a class or add methods to the class and stillload objects that were created with an earlier version of the class. If youplan to have long-lived objects that will see many versions of a class, it maybe worthwhile to put a version number in the objects so that suitableconversions can be made by the class’s__setstate__() method.

12.1.5.Pickling Class Instances¶

In this section, we describe the general mechanisms available to you to define,customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable. Bydefault, pickle will retrieve the class and the attributes of an instance viaintrospection. When a class instance is unpickled, its__init__() methodis usuallynot invoked. The default behaviour first creates an uninitializedinstance and then restores the saved attributes. The following code shows animplementation of this behaviour:

defsave(obj):return(obj.__class__,obj.__dict__)defload(cls,attributes):obj=cls.__new__(cls)obj.__dict__.update(attributes)returnobj

Classes can alter the default behaviour by providing one or several specialmethods:

object.__getnewargs_ex__()¶

In protocols 2 and newer, classes that implements the__getnewargs_ex__() method can dictate the values passed to the__new__() method upon unpickling. The method must return a pair(args,kwargs) whereargs is a tuple of positional argumentsandkwargs a dictionary of named arguments for constructing theobject. Those will be passed to the__new__() method uponunpickling.

You should implement this method if the__new__() method of yourclass requires keyword-only arguments. Otherwise, it is recommended forcompatibility to implement__getnewargs__().

Changed in version 3.6:__getnewargs_ex__() is now used in protocols 2 and 3.

object.__getnewargs__()¶

This method serves a similar purpose as__getnewargs_ex__(), butsupports only positional arguments. It must return a tuple of argumentsargs which will be passed to the__new__() method upon unpickling.

__getnewargs__() will not be called if__getnewargs_ex__() isdefined.

Changed in version 3.6:Before Python 3.6,__getnewargs__() was called instead of__getnewargs_ex__() in protocols 2 and 3.

object.__getstate__()¶

Classes can further influence how their instances are pickled; if the classdefines the method__getstate__(), it is called and the returned objectis pickled as the contents for the instance, instead of the contents of theinstance’s dictionary. If the__getstate__() method is absent, theinstance’s__dict__ is pickled as usual.

object.__setstate__(state)¶

Upon unpickling, if the class defines__setstate__(), it is called withthe unpickled state. In that case, there is no requirement for the stateobject to be a dictionary. Otherwise, the pickled state must be a dictionaryand its items are assigned to the new instance’s dictionary.

Note

If__getstate__() returns a false value, the__setstate__()method will not be called upon unpickling.

Refer to the sectionHandling Stateful Objects for more information about how to usethe methods__getstate__() and__setstate__().

As we shall see, pickle does not use directly the methods described above. Infact, these methods are part of the copy protocol which implements the__reduce__() special method. The copy protocol provides a unifiedinterface for retrieving the data necessary for pickling and copyingobjects.4

Although powerful, implementing__reduce__() directly in your classes iserror prone. For this reason, class designers should use the high-levelinterface (i.e.,__getnewargs_ex__(),__getstate__() and__setstate__()) whenever possible. We will show, however, cases whereusing__reduce__() is the only option or leads to more efficient picklingor both.

object.__reduce__()¶

The interface is currently defined as follows. The__reduce__() methodtakes no argument and shall return either a string or preferably a tuple (thereturned object is often referred to as the “reduce value”).

If a string is returned, the string should be interpreted as the name of aglobal variable. It should be the object’s local name relative to itsmodule; the pickle module searches the module namespace to determine theobject’s module. This behaviour is typically useful for singletons.

When a tuple is returned, it must be between two and five items long.Optional items can either be omitted, orNone can be provided as theirvalue. The semantics of each item are in order:

• A callable object that will be called to create the initial version of theobject.

• A tuple of arguments for the callable object. An empty tuple must be givenif the callable does not accept any argument.

• Optionally, the object’s state, which will be passed to the object’s__setstate__() method as previously described. If the object has nosuch method then, the value must be a dictionary and it will be added tothe object’s__dict__ attribute.

• Optionally, an iterator (and not a sequence) yielding successive items.These items will be appended to the object either usingobj.append(item) or, in batch, usingobj.extend(list_of_items).This is primarily used for list subclasses, but may be used by otherclasses as long as they haveappend() andextend() methods withthe appropriate signature. (Whetherappend() orextend() isused depends on which pickle protocol version is used as well as the numberof items to append, so both must be supported.)

• Optionally, an iterator (not a sequence) yielding successive key-valuepairs. These items will be stored to the object usingobj[key]=value. This is primarily used for dictionary subclasses, but may be usedby other classes as long as they implement__setitem__().

object.__reduce_ex__(protocol)¶

Alternatively, a__reduce_ex__() method may be defined. The onlydifference is this method should take a single integer argument, the protocolversion. When defined, pickle will prefer it over the__reduce__()method. In addition,__reduce__() automatically becomes a synonym forthe extended version. The main use for this method is to providebackwards-compatible reduce values for older Python releases.

# Simple example presenting how persistent ID can be used to pickle# external objects by reference.importpickleimportsqlite3fromcollectionsimportnamedtuple# Simple class representing a record in our database.MemoRecord=namedtuple("MemoRecord","key, task")classDBPickler(pickle.Pickler):defpersistent_id(self,obj):# Instead of pickling MemoRecord as a regular class instance, we emit a# persistent ID.ifisinstance(obj,MemoRecord):# Here, our persistent ID is simply a tuple, containing a tag and a# key, which refers to a specific record in the database.return("MemoRecord",obj.key)else:# If obj does not have a persistent ID, return None. This means obj# needs to be pickled as usual.returnNoneclassDBUnpickler(pickle.Unpickler):def__init__(self,file,connection):super().__init__(file)self.connection=connectiondefpersistent_load(self,pid):# This method is invoked whenever a persistent ID is encountered.# Here, pid is the tuple returned by DBPickler.cursor=self.connection.cursor()type_tag,key_id=pidiftype_tag=="MemoRecord":# Fetch the referenced record from the database and return it.cursor.execute("SELECT * FROM memos WHERE key=?",(str(key_id),))key,task=cursor.fetchone()returnMemoRecord(key,task)else:# Always raises an error if you cannot return the correct object.# Otherwise, the unpickler will think None is the object referenced# by the persistent ID.raisepickle.UnpicklingError("unsupported persistent object")defmain():importioimportpprint# Initialize and populate our database.conn=sqlite3.connect(":memory:")cursor=conn.cursor()cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)")tasks=('give food to fish','prepare group meeting','fight with a zebra',)fortaskintasks:cursor.execute("INSERT INTO memos VALUES(NULL, ?)",(task,))# Fetch the records to be pickled.cursor.execute("SELECT * FROM memos")memos=[MemoRecord(key,task)forkey,taskincursor]# Save the records using our custom DBPickler.file=io.BytesIO()DBPickler(file).dump(memos)print("Pickled records:")pprint.pprint(memos)# Update a record, just for good measure.cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1")# Load the records from the pickle data stream.file.seek(0)memos=DBUnpickler(file,conn).load()print("Unpickled records:")pprint.pprint(memos)if__name__=='__main__':main()

f=io.BytesIO()p=pickle.Pickler(f)p.dispatch_table=copyreg.dispatch_table.copy()p.dispatch_table[SomeClass]=reduce_SomeClass

classMyPickler(pickle.Pickler):dispatch_table=copyreg.dispatch_table.copy()dispatch_table[SomeClass]=reduce_SomeClassf=io.BytesIO()p=MyPickler(f)

copyreg.pickle(SomeClass,reduce_SomeClass)f=io.BytesIO()p=pickle.Pickler(f)

classTextReader:"""Print and number lines in a text file."""def__init__(self,filename):self.filename=filenameself.file=open(filename)self.lineno=0defreadline(self):self.lineno+=1line=self.file.readline()ifnotline:returnNoneifline.endswith('\n'):line=line[:-1]return"%i:%s"%(self.lineno,line)def__getstate__(self):# Copy the object's state from self.__dict__ which contains# all our instance attributes. Always use the dict.copy()# method to avoid modifying the original state.state=self.__dict__.copy()# Remove the unpicklable entries.delstate['file']returnstatedef__setstate__(self,state):# Restore instance attributes (i.e., filename and lineno).self.__dict__.update(state)# Restore the previously opened file's state. To do so, we need to# reopen it and read from it until the line count is restored.file=open(self.filename)for_inrange(self.lineno):file.readline()# Finally, save the file.self.file=file

>>>reader=TextReader("hello.txt")>>>reader.readline()'1: Hello world!'>>>reader.readline()'2: I am line number two.'>>>new_reader=pickle.loads(pickle.dumps(reader))>>>new_reader.readline()'3: Goodbye!'

12.1.6.Restricting Globals¶

By default, unpickling will import any class or function that it finds in thepickle data. For many applications, this behaviour is unacceptable as itpermits the unpickler to import and invoke arbitrary code. Just consider whatthis hand-crafted pickle data stream does when loaded:

>>>importpickle>>>pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")hello world0

In this example, the unpickler imports theos.system() function and thenapply the string argument “echo hello world”. Although this example isinoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizingUnpickler.find_class(). Unlike its name suggests,Unpickler.find_class() is called whenever a global (i.e., a class ora function) is requested. Thus it is possible to either completely forbidglobals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from thebuiltins module to be loaded:

importbuiltinsimportioimportpicklesafe_builtins={'range','complex','set','frozenset','slice',}classRestrictedUnpickler(pickle.Unpickler):deffind_class(self,module,name):# Only allow safe classes from builtins.ifmodule=="builtins"andnameinsafe_builtins:returngetattr(builtins,name)# Forbid everything else.raisepickle.UnpicklingError("global '%s.%s' is forbidden"%(module,name))defrestricted_loads(s):"""Helper function analogous to pickle.loads()."""returnRestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working has intended:

>>>restricted_loads(pickle.dumps([1,2,range(15)]))[1, 2, range(0, 15)]>>>restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")Traceback (most recent call last):...pickle.UnpicklingError:global 'os.system' is forbidden>>>restricted_loads(b'cbuiltins\neval\n'...b'(S\'getattr(__import__("os"), "system")'...b'("echo hello world")\'\ntR.')Traceback (most recent call last):...pickle.UnpicklingError:global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to beunpickled. Therefore if security is a concern, you may want to consideralternatives such as the marshalling API inxmlrpc.client orthird-party solutions.

12.1.7.Performance¶

Recent versions of the pickle protocol (from protocol 2 and upwards) featureefficient binary encodings for several common features and built-in types.Also, thepickle module has a transparent optimizer written in C.

12.1.8.Examples¶

For the simplest code, use thedump() andload() functions.

importpickle# An arbitrary collection of objects supported by pickle.data={'a':[1,2.0,3,4+6j],'b':("character string",b"byte string"),'c':{None,True,False}}withopen('data.pickle','wb')asf:# Pickle the 'data' dictionary using the highest protocol available.pickle.dump(data,f,pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data.

importpicklewithopen('data.pickle','rb')asf:# The protocol version used is detected automatically, so we do not# have to specify it.data=pickle.load(f)

Footnotes

1

Don’t confuse this with themarshal module

2

This is whylambda functions cannot be pickled: alllambda functions share the same name:<lambda>.

3

The exception raised will likely be anImportError or anAttributeError but it could be something else.

4

Thecopy module uses this protocol for shallow and deep copyingoperations.

5

The limitation on alphanumeric characters is due to the factthe persistent IDs, in protocol 0, are delimited by the newlinecharacter. Therefore if any kind of newline characters occurs inpersistent IDs, the resulting pickle will become unreadable.