Relationship to other Python modules¶

Data stream format¶

The data format used bypickle is Python-specific. This has theadvantage that there are no restrictions imposed by external standards such asJSON (which can’t represent pointer sharing); however it means thatnon-Python programs may not be able to reconstruct pickled Python objects.

By default, thepickle data format uses a relatively compact binaryrepresentation. If you need optimal size characteristics, you can efficientlycompress pickled data.

The modulepickletools contains tools for analyzing data streamsgenerated bypickle.pickletools source code has extensivecomments about opcodes used by pickle protocols.

There are currently 6 different protocols which can be used for pickling.The higher the protocol used, the more recent the version of Python neededto read the pickle produced.

• Protocol version 0 is the original “human-readable” protocol and isbackwards compatible with earlier versions of Python.

• Protocol version 1 is an old binary format which is also compatible withearlier versions of Python.

• Protocol version 2 was introduced in Python 2.3. It provides much moreefficient pickling ofnew-style classes. Refer toPEP 307 forinformation about improvements brought by protocol 2.

• Protocol version 3 was added in Python 3.0. It has explicit support forbytes objects and cannot be unpickled by Python 2.x. This wasthe default protocol in Python 3.0–3.7.

• Protocol version 4 was added in Python 3.4. It adds support for very largeobjects, pickling more kinds of objects, and some data formatoptimizations. It is the default protocol starting with Python 3.8.Refer toPEP 3154 for information about improvements brought byprotocol 4.

• Protocol version 5 was added in Python 3.8. It adds support for out-of-banddata and speedup for in-band data. Refer toPEP 574 for information aboutimprovements brought by protocol 5.

Module Interface¶

To serialize an object hierarchy, you simply call thedumps() function.Similarly, to de-serialize a data stream, you call theloads() function.However, if you want more control over serialization and de-serialization,you can create aPickler or anUnpickler object, respectively.

Thepickle module provides the following constants:

pickle.HIGHEST_PROTOCOL¶

An integer, the highestprotocol versionavailable. This value can be passed as aprotocol value to functionsdump() anddumps() as well as thePicklerconstructor.

pickle.DEFAULT_PROTOCOL¶

An integer, the defaultprotocol version usedfor pickling. May be less thanHIGHEST_PROTOCOL. Currently thedefault protocol is 4, first introduced in Python 3.4 and incompatiblewith previous versions.

Changed in version 3.0:The default protocol is 3.

Changed in version 3.8:The default protocol is 4.

Thepickle module provides the following functions to make the picklingprocess more convenient:

pickle.dump(obj,file,protocol=None,*,fix_imports=True,buffer_callback=None)¶

Write the pickled representation of the objectobj to the openfile objectfile. This is equivalent toPickler(file,protocol).dump(obj).

Argumentsfile,protocol,fix_imports andbuffer_callback havethe same meaning as in thePickler constructor.

Changed in version 3.8:Thebuffer_callback argument was added.

pickle.dumps(obj,protocol=None,*,fix_imports=True,buffer_callback=None)¶

Return the pickled representation of the objectobj as abytes object,instead of writing it to a file.

Argumentsprotocol,fix_imports andbuffer_callback have the samemeaning as in thePickler constructor.

Changed in version 3.8:Thebuffer_callback argument was added.

pickle.load(file,*,fix_imports=True,encoding='ASCII',errors='strict',buffers=None)¶

Read the pickled representation of an object from the openfile objectfile and return the reconstituted object hierarchy specified therein.This is equivalent toUnpickler(file).load().

The protocol version of the pickle is detected automatically, so noprotocol argument is needed. Bytes past the pickled representationof the object are ignored.

Argumentsfile,fix_imports,encoding,errors,strict andbuffershave the same meaning as in theUnpickler constructor.

Changed in version 3.8:Thebuffers argument was added.

pickle.loads(data,/,*,fix_imports=True,encoding='ASCII',errors='strict',buffers=None)¶

Return the reconstituted object hierarchy of the pickled representationdata of an object.data must be abytes-like object.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed. Bytes past the pickled representationof the object are ignored.

Argumentsfix_imports,encoding,errors,strict andbuffershave the same meaning as in theUnpickler constructor.

Changed in version 3.8:Thebuffers argument was added.

Thepickle module defines three exceptions:

exceptionpickle.PickleError¶

Common base class for the other pickling exceptions. It inherits fromException.

exceptionpickle.PicklingError¶

Error raised when an unpicklable object is encountered byPickler.It inherits fromPickleError.

Refer toWhat can be pickled and unpickled? to learn what kinds of objects can bepickled.

exceptionpickle.UnpicklingError¶

Error raised when there is a problem unpickling an object, such as a datacorruption or a security violation. It inherits fromPickleError.

Note that other exceptions may also be raised during unpickling, including(but not necessarily limited to) AttributeError, EOFError, ImportError, andIndexError.

Thepickle module exports three classes,Pickler,Unpickler andPickleBuffer:

classpickle.Pickler(file,protocol=None,*,fix_imports=True,buffer_callback=None)¶

This takes a binary file for writing a pickle data stream.

The optionalprotocol argument, an integer, tells the pickler to usethe given protocol; supported protocols are 0 toHIGHEST_PROTOCOL.If not specified, the default isDEFAULT_PROTOCOL. If a negativenumber is specified,HIGHEST_PROTOCOL is selected.

Thefile argument must have a write() method that accepts a single bytesargument. It can thus be an on-disk file opened for binary writing, anio.BytesIO instance, or any other custom object that meets thisinterface.

Iffix_imports is true andprotocol is less than 3, pickle will try tomap the new Python 3 names to the old module names used in Python 2, sothat the pickle data stream is readable with Python 2.

Ifbuffer_callback isNone (the default), buffer views areserialized intofile as part of the pickle stream.

Ifbuffer_callback is notNone, then it can be called any numberof times with a buffer view. If the callback returns a false value(such asNone), the given buffer isout-of-band;otherwise the buffer is serialized in-band, i.e. inside the pickle stream.

It is an error ifbuffer_callback is notNone andprotocol isNone or smaller than 5.

Changed in version 3.8:Thebuffer_callback argument was added.

dump(obj)¶

Write the pickled representation ofobj to the open file object given inthe constructor.

persistent_id(obj)¶

Do nothing by default. This exists so a subclass can override it.

Ifpersistent_id() returnsNone,obj is pickled as usual. Anyother value causesPickler to emit the returned value as apersistent ID forobj. The meaning of this persistent ID should bedefined byUnpickler.persistent_load(). Note that the valuereturned bypersistent_id() cannot itself have a persistent ID.

SeePersistence of External Objects for details and examples of uses.

Changed in version 3.13:Add the default implementation of this method in the C implementationofPickler.

dispatch_table¶

A pickler object’s dispatch table is a registry ofreductionfunctions of the kind which can be declared usingcopyreg.pickle(). It is a mapping whose keys are classesand whose values are reduction functions. A reduction functiontakes a single argument of the associated class and shouldconform to the same interface as a__reduce__()method.

By default, a pickler object will not have adispatch_table attribute, and it will instead use theglobal dispatch table managed by thecopyreg module.However, to customize the pickling for a specific pickler objectone can set thedispatch_table attribute to a dict-likeobject. Alternatively, if a subclass ofPickler has adispatch_table attribute then this will be used as thedefault dispatch table for instances of that class.

SeeDispatch Tables for usage examples.

Added in version 3.3.

reducer_override(obj)¶

Special reducer that can be defined inPickler subclasses. Thismethod has priority over any reducer in thedispatch_table. Itshould conform to the same interface as a__reduce__() method, andcan optionally returnNotImplemented to fallback ondispatch_table-registered reducers to pickleobj.

For a detailed example, seeCustom Reduction for Types, Functions, and Other Objects.

Added in version 3.8.

fast¶

Deprecated. Enable fast mode if set to a true value. The fast modedisables the usage of memo, therefore speeding the pickling process by notgenerating superfluous PUT opcodes. It should not be used withself-referential objects, doing otherwise will causePickler torecurse infinitely.

Usepickletools.optimize() if you need more compact pickles.

classpickle.Unpickler(file,*,fix_imports=True,encoding='ASCII',errors='strict',buffers=None)¶

This takes a binary file for reading a pickle data stream.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed.

The argumentfile must have three methods, a read() method that takes aninteger argument, a readinto() method that takes a buffer argumentand a readline() method that requires no arguments, as in theio.BufferedIOBase interface. Thusfile can be an on-disk fileopened for binary reading, anio.BytesIO object, or any othercustom object that meets this interface.

The optional argumentsfix_imports,encoding anderrors are usedto control compatibility support for pickle stream generated by Python 2.Iffix_imports is true, pickle will try to map the old Python 2 namesto the new names used in Python 3. Theencoding anderrors tellpickle how to decode 8-bit string instances pickled by Python 2;these default to ‘ASCII’ and ‘strict’, respectively. Theencoding canbe ‘bytes’ to read these 8-bit string instances as bytes objects.Usingencoding='latin1' is required for unpickling NumPy arrays andinstances ofdatetime,date andtime pickled by Python 2.

Ifbuffers isNone (the default), then all data necessary fordeserialization must be contained in the pickle stream. This meansthat thebuffer_callback argument wasNone when aPicklerwas instantiated (or whendump() ordumps() was called).

Ifbuffers is notNone, it should be an iterable of buffer-enabledobjects that is consumed each time the pickle stream referencesanout-of-band buffer view. Such buffers have beengiven in order to thebuffer_callback of a Pickler object.

Changed in version 3.8:Thebuffers argument was added.

load()¶

Read the pickled representation of an object from the open file objectgiven in the constructor, and return the reconstituted object hierarchyspecified therein. Bytes past the pickled representation of the objectare ignored.

persistent_load(pid)¶

Raise anUnpicklingError by default.

If defined,persistent_load() should return the object specified bythe persistent IDpid. If an invalid persistent ID is encountered, anUnpicklingError should be raised.

SeePersistence of External Objects for details and examples of uses.

Changed in version 3.13:Add the default implementation of this method in the C implementationofUnpickler.

find_class(module,name)¶

Importmodule if necessary and return the object calledname from it,where themodule andname arguments arestr objects. Note,unlike its name suggests,find_class() is also used for findingfunctions.

Subclasses may override this to gain control over what type of objects andhow they can be loaded, potentially reducing security risks. Refer toRestricting Globals for details.

Raises anauditing eventpickle.find_class with argumentsmodule,name.

classpickle.PickleBuffer(buffer)¶

A wrapper for a buffer representing picklable data.buffer must be abuffer-providing object, such as abytes-like object or a N-dimensional array.

PickleBuffer is itself a buffer provider, therefore it ispossible to pass it to other APIs expecting a buffer-providing object,such asmemoryview.

PickleBuffer objects can only be serialized using pickleprotocol 5 or higher. They are eligible forout-of-band serialization.

Added in version 3.8.

raw()¶

Return amemoryview of the memory area underlying this buffer.The returned object is a one-dimensional, C-contiguous memoryviewwith formatB (unsigned bytes).BufferError is raised ifthe buffer is neither C- nor Fortran-contiguous.

release()¶

Release the underlying buffer exposed by the PickleBuffer object.

What can be pickled and unpickled?¶

The following types can be pickled:

• built-in constants (None,True,False,Ellipsis, andNotImplemented);

• integers, floating-point numbers, complex numbers;

• strings, bytes, bytearrays;

• tuples, lists, sets, and dictionaries containing only picklable objects;

• functions (built-in and user-defined) accessible from the top level of amodule (usingdef, notlambda);

• classes accessible from the top level of a module;

• instances of such classes whose the result of calling__getstate__()is picklable (see sectionPickling Class Instances for details).

Attempts to pickle unpicklable objects will raise thePicklingErrorexception; when this happens, an unspecified number of bytes may have alreadybeen written to the underlying file. Trying to pickle a highly recursive datastructure may exceed the maximum recursion depth, aRecursionError will beraised in this case. You can carefully raise this limit withsys.setrecursionlimit().

Note that functions (built-in and user-defined) are pickled by fullyqualified name, not by value.[2] This means that only the function name ispickled, along with the name of the containing module and classes. Neitherthe function’s code, nor any of its function attributes are pickled. Thus thedefining module must be importable in the unpickling environment, and the modulemust contain the named object, otherwise an exception will be raised.[3]

Similarly, classes are pickled by fully qualified name, so the same restrictions inthe unpickling environment apply. Note that none of the class’s code or data ispickled, so in the following example the class attributeattr is notrestored in the unpickling environment:

classFoo:attr='A class attribute'picklestring=pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined atthe top level of a module.

Similarly, when class instances are pickled, their class’s code and data are notpickled along with them. Only the instance data are pickled. This is done onpurpose, so you can fix bugs in a class or add methods to the class and stillload objects that were created with an earlier version of the class. If youplan to have long-lived objects that will see many versions of a class, it maybe worthwhile to put a version number in the objects so that suitableconversions can be made by the class’s__setstate__() method.

Pickling Class Instances¶

In this section, we describe the general mechanisms available to you to define,customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable. Bydefault, pickle will retrieve the class and the attributes of an instance viaintrospection. When a class instance is unpickled, its__init__() methodis usuallynot invoked. The default behaviour first creates an uninitializedinstance and then restores the saved attributes. The following code shows animplementation of this behaviour:

defsave(obj):return(obj.__class__,obj.__dict__)defrestore(cls,attributes):obj=cls.__new__(cls)obj.__dict__.update(attributes)returnobj

Classes can alter the default behaviour by providing one or several specialmethods:

object.__getnewargs_ex__()¶

In protocols 2 and newer, classes that implements the__getnewargs_ex__() method can dictate the values passed to the__new__() method upon unpickling. The method must return a pair(args,kwargs) whereargs is a tuple of positional argumentsandkwargs a dictionary of named arguments for constructing theobject. Those will be passed to the__new__() method uponunpickling.

You should implement this method if the__new__() method of yourclass requires keyword-only arguments. Otherwise, it is recommended forcompatibility to implement__getnewargs__().

Changed in version 3.6:__getnewargs_ex__() is now used in protocols 2 and 3.

object.__getnewargs__()¶

This method serves a similar purpose as__getnewargs_ex__(), butsupports only positional arguments. It must return a tuple of argumentsargs which will be passed to the__new__() method upon unpickling.

__getnewargs__() will not be called if__getnewargs_ex__() isdefined.

Changed in version 3.6:Before Python 3.6,__getnewargs__() was called instead of__getnewargs_ex__() in protocols 2 and 3.

object.__getstate__()¶

Classes can further influence how their instances are pickled by overridingthe method__getstate__(). It is called and the returned objectis pickled as the contents for the instance, instead of a default state.There are several cases:

• For a class that has no instance__dict__ and no__slots__, the default state isNone.

• For a class that has an instance__dict__ and no__slots__, the default state isself.__dict__.

• For a class that has an instance__dict__ and__slots__, the default state is a tuple consisting of twodictionaries:self.__dict__, and a dictionary mapping slotnames to slot values. Only slots that have a value areincluded in the latter.

• For a class that has__slots__ and no instance__dict__, the default state is a tuple whose first itemisNone and whose second item is a dictionary mapping slot namesto slot values described in the previous bullet.

Changed in version 3.11:Added the default implementation of the__getstate__() method in theobject class.

object.__setstate__(state)¶

Upon unpickling, if the class defines__setstate__(), it is called withthe unpickled state. In that case, there is no requirement for the stateobject to be a dictionary. Otherwise, the pickled state must be a dictionaryand its items are assigned to the new instance’s dictionary.

Note

If__reduce__() returns a state with valueNone at pickling,the__setstate__() method will not be called upon unpickling.

Refer to the sectionHandling Stateful Objects for more information about how to usethe methods__getstate__() and__setstate__().

As we shall see, pickle does not use directly the methods described above. Infact, these methods are part of the copy protocol which implements the__reduce__() special method. The copy protocol provides a unifiedinterface for retrieving the data necessary for pickling and copyingobjects.[4]

Although powerful, implementing__reduce__() directly in your classes iserror prone. For this reason, class designers should use the high-levelinterface (i.e.,__getnewargs_ex__(),__getstate__() and__setstate__()) whenever possible. We will show, however, cases whereusing__reduce__() is the only option or leads to more efficient picklingor both.

object.__reduce__()¶

The interface is currently defined as follows. The__reduce__() methodtakes no argument and shall return either a string or preferably a tuple (thereturned object is often referred to as the “reduce value”).

If a string is returned, the string should be interpreted as the name of aglobal variable. It should be the object’s local name relative to itsmodule; the pickle module searches the module namespace to determine theobject’s module. This behaviour is typically useful for singletons.

When a tuple is returned, it must be between two and six items long.Optional items can either be omitted, orNone can be provided as theirvalue. The semantics of each item are in order:

• A callable object that will be called to create the initial version of theobject.

• A tuple of arguments for the callable object. An empty tuple must be givenif the callable does not accept any argument.

• Optionally, the object’s state, which will be passed to the object’s__setstate__() method as previously described. If the object has nosuch method then, the value must be a dictionary and it will be added tothe object’s__dict__ attribute.

• Optionally, an iterator (and not a sequence) yielding successive items.These items will be appended to the object either usingobj.append(item) or, in batch, usingobj.extend(list_of_items).This is primarily used for list subclasses, but may be used by otherclasses as long as they haveappend and extend methods withthe appropriate signature. (Whetherappend() orextend() isused depends on which pickle protocol version is used as well as the numberof items to append, so both must be supported.)

• Optionally, an iterator (not a sequence) yielding successive key-valuepairs. These items will be stored to the object usingobj[key]=value. This is primarily used for dictionary subclasses, but may be usedby other classes as long as they implement__setitem__().

• Optionally, a callable with a(obj,state) signature. Thiscallable allows the user to programmatically control the state-updatingbehavior of a specific object, instead of usingobj’s static__setstate__() method. If notNone, this callable will havepriority overobj’s__setstate__().

  Added in version 3.8:The optional sixth tuple item,(obj,state), was added.

object.__reduce_ex__(protocol)¶

Alternatively, a__reduce_ex__() method may be defined. The onlydifference is this method should take a single integer argument, the protocolversion. When defined, pickle will prefer it over the__reduce__()method. In addition,__reduce__() automatically becomes a synonym forthe extended version. The main use for this method is to providebackwards-compatible reduce values for older Python releases.

# Simple example presenting how persistent ID can be used to pickle# external objects by reference.importpickleimportsqlite3fromcollectionsimportnamedtuple# Simple class representing a record in our database.MemoRecord=namedtuple("MemoRecord","key, task")classDBPickler(pickle.Pickler):defpersistent_id(self,obj):# Instead of pickling MemoRecord as a regular class instance, we emit a# persistent ID.ifisinstance(obj,MemoRecord):# Here, our persistent ID is simply a tuple, containing a tag and a# key, which refers to a specific record in the database.return("MemoRecord",obj.key)else:# If obj does not have a persistent ID, return None. This means obj# needs to be pickled as usual.returnNoneclassDBUnpickler(pickle.Unpickler):def__init__(self,file,connection):super().__init__(file)self.connection=connectiondefpersistent_load(self,pid):# This method is invoked whenever a persistent ID is encountered.# Here, pid is the tuple returned by DBPickler.cursor=self.connection.cursor()type_tag,key_id=pidiftype_tag=="MemoRecord":# Fetch the referenced record from the database and return it.cursor.execute("SELECT * FROM memos WHERE key=?",(str(key_id),))key,task=cursor.fetchone()returnMemoRecord(key,task)else:# Always raises an error if you cannot return the correct object.# Otherwise, the unpickler will think None is the object referenced# by the persistent ID.raisepickle.UnpicklingError("unsupported persistent object")defmain():importioimportpprint# Initialize and populate our database.conn=sqlite3.connect(":memory:")cursor=conn.cursor()cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)")tasks=('give food to fish','prepare group meeting','fight with a zebra',)fortaskintasks:cursor.execute("INSERT INTO memos VALUES(NULL, ?)",(task,))# Fetch the records to be pickled.cursor.execute("SELECT * FROM memos")memos=[MemoRecord(key,task)forkey,taskincursor]# Save the records using our custom DBPickler.file=io.BytesIO()DBPickler(file).dump(memos)print("Pickled records:")pprint.pprint(memos)# Update a record, just for good measure.cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1")# Load the records from the pickle data stream.file.seek(0)memos=DBUnpickler(file,conn).load()print("Unpickled records:")pprint.pprint(memos)if__name__=='__main__':main()

f=io.BytesIO()p=pickle.Pickler(f)p.dispatch_table=copyreg.dispatch_table.copy()p.dispatch_table[SomeClass]=reduce_SomeClass

classMyPickler(pickle.Pickler):dispatch_table=copyreg.dispatch_table.copy()dispatch_table[SomeClass]=reduce_SomeClassf=io.BytesIO()p=MyPickler(f)

copyreg.pickle(SomeClass,reduce_SomeClass)f=io.BytesIO()p=pickle.Pickler(f)

classTextReader:"""Print and number lines in a text file."""def__init__(self,filename):self.filename=filenameself.file=open(filename)self.lineno=0defreadline(self):self.lineno+=1line=self.file.readline()ifnotline:returnNoneifline.endswith('\n'):line=line[:-1]return"%i:%s"%(self.lineno,line)def__getstate__(self):# Copy the object's state from self.__dict__ which contains# all our instance attributes. Always use the dict.copy()# method to avoid modifying the original state.state=self.__dict__.copy()# Remove the unpicklable entries.delstate['file']returnstatedef__setstate__(self,state):# Restore instance attributes (i.e., filename and lineno).self.__dict__.update(state)# Restore the previously opened file's state. To do so, we need to# reopen it and read from it until the line count is restored.file=open(self.filename)for_inrange(self.lineno):file.readline()# Finally, save the file.self.file=file

>>>reader=TextReader("hello.txt")>>>reader.readline()'1: Hello world!'>>>reader.readline()'2: I am line number two.'>>>new_reader=pickle.loads(pickle.dumps(reader))>>>new_reader.readline()'3: Goodbye!'

Custom Reduction for Types, Functions, and Other Objects¶

Sometimes,dispatch_table may not be flexible enough.In particular we may want to customize pickling based on another criterionthan the object’s type, or we may want to customize the pickling offunctions and classes.

For those cases, it is possible to subclass from thePickler class andimplement areducer_override() method. This method can return anarbitrary reduction tuple (see__reduce__()). It can alternatively returnNotImplemented to fallback to the traditional behavior.

If both thedispatch_table andreducer_override() are defined, thenreducer_override() method takes priority.

Here is a simple example where we allow pickling and reconstructinga given class:

importioimportpickleclassMyClass:my_attribute=1classMyPickler(pickle.Pickler):defreducer_override(self,obj):"""Custom reducer for MyClass."""ifgetattr(obj,"__name__",None)=="MyClass":returntype,(obj.__name__,obj.__bases__,{'my_attribute':obj.my_attribute})else:# For any other object, fallback to usual reductionreturnNotImplementedf=io.BytesIO()p=MyPickler(f)p.dump(MyClass)delMyClassunpickled_class=pickle.loads(f.getvalue())assertisinstance(unpickled_class,type)assertunpickled_class.__name__=="MyClass"assertunpickled_class.my_attribute==1

Out-of-band Buffers¶

In some contexts, thepickle module is used to transfer massive amountsof data. Therefore, it can be important to minimize the number of memorycopies, to preserve performance and resource consumption. However, normaloperation of thepickle module, as it transforms a graph-like structureof objects into a sequential stream of bytes, intrinsically involves copyingdata to and from the pickle stream.

This constraint can be eschewed if both theprovider (the implementationof the object types to be transferred) and theconsumer (the implementationof the communications system) support the out-of-band transfer facilitiesprovided by pickle protocol 5 and higher.

classZeroCopyByteArray(bytearray):def__reduce_ex__(self,protocol):ifprotocol>=5:returntype(self)._reconstruct,(PickleBuffer(self),),Noneelse:# PickleBuffer is forbidden with pickle protocols <= 4.returntype(self)._reconstruct,(bytearray(self),)@classmethoddef_reconstruct(cls,obj):withmemoryview(obj)asm:# Get a handle over the original buffer objectobj=m.objiftype(obj)iscls:# Original buffer object is a ZeroCopyByteArray, return it# as-is.returnobjelse:returncls(obj)

b=ZeroCopyByteArray(b"abc")data=pickle.dumps(b,protocol=5)new_b=pickle.loads(data)print(b==new_b)# Trueprint(bisnew_b)# False: a copy was made

b=ZeroCopyByteArray(b"abc")buffers=[]data=pickle.dumps(b,protocol=5,buffer_callback=buffers.append)new_b=pickle.loads(data,buffers=buffers)print(b==new_b)# Trueprint(bisnew_b)# True: no copy was made

Restricting Globals¶

By default, unpickling will import any class or function that it finds in thepickle data. For many applications, this behaviour is unacceptable as itpermits the unpickler to import and invoke arbitrary code. Just consider whatthis hand-crafted pickle data stream does when loaded:

>>>importpickle>>>pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")hello world0

In this example, the unpickler imports theos.system() function and thenapply the string argument “echo hello world”. Although this example isinoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizingUnpickler.find_class(). Unlike its name suggests,Unpickler.find_class() is called whenever a global (i.e., a class ora function) is requested. Thus it is possible to either completely forbidglobals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from thebuiltins module to be loaded:

importbuiltinsimportioimportpicklesafe_builtins={'range','complex','set','frozenset','slice',}classRestrictedUnpickler(pickle.Unpickler):deffind_class(self,module,name):# Only allow safe classes from builtins.ifmodule=="builtins"andnameinsafe_builtins:returngetattr(builtins,name)# Forbid everything else.raisepickle.UnpicklingError("global '%s.%s' is forbidden"%(module,name))defrestricted_loads(s):"""Helper function analogous to pickle.loads()."""returnRestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working as intended:

>>>restricted_loads(pickle.dumps([1,2,range(15)]))[1, 2, range(0, 15)]>>>restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")Traceback (most recent call last):...pickle.UnpicklingError:global 'os.system' is forbidden>>>restricted_loads(b'cbuiltins\neval\n'...b'(S\'getattr(__import__("os"), "system")'...b'("echo hello world")\'\ntR.')Traceback (most recent call last):...pickle.UnpicklingError:global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to beunpickled. Therefore if security is a concern, you may want to consideralternatives such as the marshalling API inxmlrpc.client orthird-party solutions.

Performance¶

Recent versions of the pickle protocol (from protocol 2 and upwards) featureefficient binary encodings for several common features and built-in types.Also, thepickle module has a transparent optimizer written in C.

Examples¶

For the simplest code, use thedump() andload() functions.

importpickle# An arbitrary collection of objects supported by pickle.data={'a':[1,2.0,3+4j],'b':("character string",b"byte string"),'c':{None,True,False}}withopen('data.pickle','wb')asf:# Pickle the 'data' dictionary using the highest protocol available.pickle.dump(data,f,pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data.

importpicklewithopen('data.pickle','rb')asf:# The protocol version used is detected automatically, so we do not# have to specify it.data=pickle.load(f)

Footnotes

Don’t confuse this with themarshal module

This is whylambda functions cannot be pickled: alllambda functions share the same name:<lambda>.

The exception raised will likely be anImportError or anAttributeError but it could be something else.

Thecopy module uses this protocol for shallow and deep copyingoperations.

The limitation on alphanumeric characters is due to the factthat persistent IDs in protocol 0 are delimited by the newlinecharacter. Therefore if any kind of newline characters occurs inpersistent IDs, the resulting pickled data will become unreadable.