New in version 2.3.

9.7.1.Itertool functions¶

The following module functions all construct and return iterators. Some providestreams of infinite length, so they should only be accessed by functions orloops that truncate the stream.

itertools.chain(*iterables)¶

Make an iterator that returns elements from the first iterable until it isexhausted, then proceeds to the next iterable, until all of the iterables areexhausted. Used for treating consecutive sequences as a single sequence.Roughly equivalent to:

defchain(*iterables):# chain('ABC', 'DEF') --> A B C D E Fforitiniterables:forelementinit:yieldelement

classmethodchain.from_iterable(iterable)¶

Alternate constructor forchain(). Gets chained inputs from asingle iterable argument that is evaluated lazily. Roughly equivalent to:

deffrom_iterable(iterables):# chain.from_iterable(['ABC', 'DEF']) --> A B C D E Fforitiniterables:forelementinit:yieldelement

New in version 2.6.

itertools.combinations(iterable,r)¶

Returnr length subsequences of elements from the inputiterable.

Combinations are emitted in lexicographic sort order. So, if theinputiterable is sorted, the combination tuples will be producedin sorted order.

Elements are treated as unique based on their position, not on theirvalue. So if the input elements are unique, there will be no repeatvalues in each combination.

Roughly equivalent to:

defcombinations(iterable,r):# combinations('ABCD', 2) --> AB AC AD BC BD CD# combinations(range(4), 3) --> 012 013 023 123pool=tuple(iterable)n=len(pool)ifr>n:returnindices=range(r)yieldtuple(pool[i]foriinindices)whileTrue:foriinreversed(range(r)):ifindices[i]!=i+n-r:breakelse:returnindices[i]+=1forjinrange(i+1,r):indices[j]=indices[j-1]+1yieldtuple(pool[i]foriinindices)

The code forcombinations() can be also expressed as a subsequenceofpermutations() after filtering entries where the elements are notin sorted order (according to their position in the input pool):

defcombinations(iterable,r):pool=tuple(iterable)n=len(pool)forindicesinpermutations(range(n),r):ifsorted(indices)==list(indices):yieldtuple(pool[i]foriinindices)

The number of items returned isn!/r!/(n-r)! when0<=r<=nor zero whenr>n.

New in version 2.6.

itertools.combinations_with_replacement(iterable,r)¶

Returnr length subsequences of elements from the inputiterableallowing individual elements to be repeated more than once.

Combinations are emitted in lexicographic sort order. So, if theinputiterable is sorted, the combination tuples will be producedin sorted order.

Elements are treated as unique based on their position, not on theirvalue. So if the input elements are unique, the generated combinationswill also be unique.

Roughly equivalent to:

defcombinations_with_replacement(iterable,r):# combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CCpool=tuple(iterable)n=len(pool)ifnotnandr:returnindices=[0]*ryieldtuple(pool[i]foriinindices)whileTrue:foriinreversed(range(r)):ifindices[i]!=n-1:breakelse:returnindices[i:]=[indices[i]+1]*(r-i)yieldtuple(pool[i]foriinindices)

The code forcombinations_with_replacement() can be also expressed asa subsequence ofproduct() after filtering entries where the elementsare not in sorted order (according to their position in the input pool):

defcombinations_with_replacement(iterable,r):pool=tuple(iterable)n=len(pool)forindicesinproduct(range(n),repeat=r):ifsorted(indices)==list(indices):yieldtuple(pool[i]foriinindices)

The number of items returned is(n+r-1)!/r!/(n-1)! whenn>0.

New in version 2.7.

itertools.compress(data,selectors)¶

Make an iterator that filters elements fromdata returning only those thathave a corresponding element inselectors that evaluates toTrue.Stops when either thedata orselectors iterables has been exhausted.Roughly equivalent to:

defcompress(data,selectors):# compress('ABCDEF', [1,0,1,0,1,1]) --> A C E Freturn(dford,sinizip(data,selectors)ifs)

New in version 2.7.

itertools.count(start=0,step=1)¶

Make an iterator that returns evenly spaced values starting withn. Oftenused as an argument toimap() to generate consecutive data points.Also, used withizip() to add sequence numbers. Equivalent to:

defcount(start=0,step=1):# count(10) --> 10 11 12 13 14 ...# count(2.5, 0.5) -> 2.5 3.0 3.5 ...n=startwhileTrue:yieldnn+=step

When counting with floating point numbers, better accuracy can sometimes beachieved by substituting multiplicative code such as:(start+step*iforiincount()).

Changed in version 2.7:addedstep argument and allowed non-integer arguments.

itertools.cycle(iterable)¶

Make an iterator returning elements from the iterable and saving a copy of each.When the iterable is exhausted, return elements from the saved copy. Repeatsindefinitely. Roughly equivalent to:

defcycle(iterable):# cycle('ABCD') --> A B C D A B C D A B C D ...saved=[]forelementiniterable:yieldelementsaved.append(element)whilesaved:forelementinsaved:yieldelement

Note, this member of the toolkit may require significant auxiliary storage(depending on the length of the iterable).

itertools.dropwhile(predicate,iterable)¶

Make an iterator that drops elements from the iterable as long as the predicateis true; afterwards, returns every element. Note, the iterator does not produceany output until the predicate first becomes false, so it may have a lengthystart-up time. Roughly equivalent to:

defdropwhile(predicate,iterable):# dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1iterable=iter(iterable)forxiniterable:ifnotpredicate(x):yieldxbreakforxiniterable:yieldx

itertools.groupby(iterable[,key])¶

Make an iterator that returns consecutive keys and groups from theiterable.Thekey is a function computing a key value for each element. If notspecified or isNone,key defaults to an identity function and returnsthe element unchanged. Generally, the iterable needs to already be sorted onthe same key function.

The operation ofgroupby() is similar to theuniq filter in Unix. Itgenerates a break or new group every time the value of the key function changes(which is why it is usually necessary to have sorted the data using the same keyfunction). That behavior differs from SQL’s GROUP BY which aggregates commonelements regardless of their input order.

The returned group is itself an iterator that shares the underlying iterablewithgroupby(). Because the source is shared, when thegroupby()object is advanced, the previous group is no longer visible. So, if that datais needed later, it should be stored as a list:

groups=[]uniquekeys=[]data=sorted(data,key=keyfunc)fork,gingroupby(data,keyfunc):groups.append(list(g))# Store group iterator as a listuniquekeys.append(k)

groupby() is roughly equivalent to:

classgroupby(object):# [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B# [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC Ddef__init__(self,iterable,key=None):ifkeyisNone:key=lambdax:xself.keyfunc=keyself.it=iter(iterable)self.tgtkey=self.currkey=self.currvalue=object()def__iter__(self):returnselfdefnext(self):whileself.currkey==self.tgtkey:self.currvalue=next(self.it)# Exit on StopIterationself.currkey=self.keyfunc(self.currvalue)self.tgtkey=self.currkeyreturn(self.currkey,self._grouper(self.tgtkey))def_grouper(self,tgtkey):whileself.currkey==tgtkey:yieldself.currvalueself.currvalue=next(self.it)# Exit on StopIterationself.currkey=self.keyfunc(self.currvalue)

New in version 2.4.

itertools.ifilter(predicate,iterable)¶

Make an iterator that filters elements from iterable returning only those forwhich the predicate isTrue. Ifpredicate isNone, return the itemsthat are true. Roughly equivalent to:

defifilter(predicate,iterable):# ifilter(lambda x: x%2, range(10)) --> 1 3 5 7 9ifpredicateisNone:predicate=boolforxiniterable:ifpredicate(x):yieldx

itertools.ifilterfalse(predicate,iterable)¶

Make an iterator that filters elements from iterable returning only those forwhich the predicate isFalse. Ifpredicate isNone, return the itemsthat are false. Roughly equivalent to:

defifilterfalse(predicate,iterable):# ifilterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8ifpredicateisNone:predicate=boolforxiniterable:ifnotpredicate(x):yieldx

itertools.imap(function,*iterables)¶

Make an iterator that computes the function using arguments from each of theiterables. Iffunction is set toNone, thenimap() returns thearguments as a tuple. Likemap() but stops when the shortest iterable isexhausted instead of filling inNone for shorter iterables. The reason forthe difference is that infinite iterator arguments are typically an error formap() (because the output is fully evaluated) but represent a common anduseful way of supplying arguments toimap(). Roughly equivalent to:

defimap(function,*iterables):# imap(pow, (2,3,10), (5,2,3)) --> 32 9 1000iterables=map(iter,iterables)whileTrue:args=[next(it)foritiniterables]iffunctionisNone:yieldtuple(args)else:yieldfunction(*args)

itertools.islice(iterable,stop)¶

itertools.islice(iterable,start,stop[,step])

Make an iterator that returns selected elements from the iterable. Ifstart isnon-zero, then elements from the iterable are skipped until start is reached.Afterward, elements are returned consecutively unlessstep is set higher thanone which results in items being skipped. Ifstop isNone, then iterationcontinues until the iterator is exhausted, if at all; otherwise, it stops at thespecified position. Unlike regular slicing,islice() does not supportnegative values forstart,stop, orstep. Can be used to extract relatedfields from data where the internal structure has been flattened (for example, amulti-line report may list a name field on every third line). Roughly equivalent to:

defislice(iterable,*args):# islice('ABCDEFG', 2) --> A B# islice('ABCDEFG', 2, 4) --> C D# islice('ABCDEFG', 2, None) --> C D E F G# islice('ABCDEFG', 0, None, 2) --> A C E Gs=slice(*args)start,stop,step=s.startor0,s.stoporsys.maxint,s.stepor1it=iter(xrange(start,stop,step)))try:nexti=next(it)exceptStopIteration:# Consume *iterable* up to the *start* position.fori,elementinizip(xrange(start),iterable):passreturntry:fori,elementinenumerate(iterable):ifi==nexti:yieldelementnexti=next(it)exceptStopIteration:# Consume to *stop*.fori,elementinizip(xrange(i+1,stop),iterable):pass

Ifstart isNone, then iteration starts at zero. Ifstep isNone,then the step defaults to one.

Changed in version 2.5:acceptNone values for defaultstart andstep.

itertools.izip(*iterables)¶

Make an iterator that aggregates elements from each of the iterables. Likezip() except that it returns an iterator instead of a list. Used forlock-step iteration over several iterables at a time. Roughly equivalent to:

defizip(*iterables):# izip('ABCD', 'xy') --> Ax Byiterators=map(iter,iterables)whileiterators:yieldtuple(map(next,iterators))

Changed in version 2.4:When no iterables are specified, returns a zero length iterator instead ofraising aTypeError exception.

The left-to-right evaluation order of the iterables is guaranteed. Thismakes possible an idiom for clustering a data series into n-length groupsusingizip(*[iter(s)]*n).

izip() should only be used with unequal length inputs when you don’tcare about trailing, unmatched values from the longer iterables. If thosevalues are important, useizip_longest() instead.

itertools.izip_longest(*iterables[,fillvalue])¶

Make an iterator that aggregates elements from each of the iterables. If theiterables are of uneven length, missing values are filled-in withfillvalue.Iteration continues until the longest iterable is exhausted. Roughly equivalent to:

classZipExhausted(Exception):passdefizip_longest(*args,**kwds):# izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-fillvalue=kwds.get('fillvalue')counter=[len(args)-1]defsentinel():ifnotcounter[0]:raiseZipExhaustedcounter[0]-=1yieldfillvaluefillers=repeat(fillvalue)iterators=[chain(it,sentinel(),fillers)foritinargs]try:whileiterators:yieldtuple(map(next,iterators))exceptZipExhausted:pass

If one of the iterables is potentially infinite, then theizip_longest() function should be wrapped with something that limitsthe number of calls (for exampleislice() ortakewhile()). Ifnot specified,fillvalue defaults toNone.

New in version 2.6.

itertools.permutations(iterable[,r])¶

Return successiver length permutations of elements in theiterable.

Ifr is not specified or isNone, thenr defaults to the lengthof theiterable and all possible full-length permutationsare generated.

Permutations are emitted in lexicographic sort order. So, if theinputiterable is sorted, the permutation tuples will be producedin sorted order.

Elements are treated as unique based on their position, not on theirvalue. So if the input elements are unique, there will be no repeatvalues in each permutation.

Roughly equivalent to:

defpermutations(iterable,r=None):# permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC# permutations(range(3)) --> 012 021 102 120 201 210pool=tuple(iterable)n=len(pool)r=nifrisNoneelserifr>n:returnindices=range(n)cycles=range(n,n-r,-1)yieldtuple(pool[i]foriinindices[:r])whilen:foriinreversed(range(r)):cycles[i]-=1ifcycles[i]==0:indices[i:]=indices[i+1:]+indices[i:i+1]cycles[i]=n-ielse:j=cycles[i]indices[i],indices[-j]=indices[-j],indices[i]yieldtuple(pool[i]foriinindices[:r])breakelse:return

The code forpermutations() can be also expressed as a subsequence ofproduct(), filtered to exclude entries with repeated elements (thosefrom the same position in the input pool):

defpermutations(iterable,r=None):pool=tuple(iterable)n=len(pool)r=nifrisNoneelserforindicesinproduct(range(n),repeat=r):iflen(set(indices))==r:yieldtuple(pool[i]foriinindices)

The number of items returned isn!/(n-r)! when0<=r<=nor zero whenr>n.

New in version 2.6.

itertools.product(*iterables[,repeat])¶

Cartesian product of input iterables.

Roughly equivalent to nested for-loops in a generator expression. For example,product(A,B) returns the same as((x,y)forxinAforyinB).

The nested loops cycle like an odometer with the rightmost element advancingon every iteration. This pattern creates a lexicographic ordering so that ifthe input’s iterables are sorted, the product tuples are emitted in sortedorder.

To compute the product of an iterable with itself, specify the number ofrepetitions with the optionalrepeat keyword argument. For example,product(A,repeat=4) means the same asproduct(A,A,A,A).

This function is roughly equivalent to the following code, except that theactual implementation does not build up intermediate results in memory:

defproduct(*args,**kwds):# product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy# product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111pools=map(tuple,args)*kwds.get('repeat',1)result=[[]]forpoolinpools:result=[x+[y]forxinresultforyinpool]forprodinresult:yieldtuple(prod)

New in version 2.6.

itertools.repeat(object[,times])¶

Make an iterator that returnsobject over and over again. Runs indefinitelyunless thetimes argument is specified. Used as argument toimap() forinvariant function parameters. Also used withizip() to create constantfields in a tuple record. Roughly equivalent to:

defrepeat(object,times=None):# repeat(10, 3) --> 10 10 10iftimesisNone:whileTrue:yieldobjectelse:foriinxrange(times):yieldobject

A common use forrepeat is to supply a stream of constant values toimaporzip:

>>>list(imap(pow,xrange(10),repeat(2)))[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

itertools.starmap(function,iterable)¶

Make an iterator that computes the function using arguments obtained fromthe iterable. Used instead ofimap() when argument parameters are alreadygrouped in tuples from a single iterable (the data has been “pre-zipped”). Thedifference betweenimap() andstarmap() parallels the distinctionbetweenfunction(a,b) andfunction(*c). Roughly equivalent to:

defstarmap(function,iterable):# starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000forargsiniterable:yieldfunction(*args)

Changed in version 2.6:Previously,starmap() required the function arguments to be tuples.Now, any iterable is allowed.

itertools.takewhile(predicate,iterable)¶

Make an iterator that returns elements from the iterable as long as thepredicate is true. Roughly equivalent to:

deftakewhile(predicate,iterable):# takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4forxiniterable:ifpredicate(x):yieldxelse:break

itertools.tee(iterable[,n=2])¶

Returnn independent iterators from a single iterable. Roughly equivalent to:

deftee(iterable,n=2):it=iter(iterable)deques=[collections.deque()foriinrange(n)]defgen(mydeque):whileTrue:ifnotmydeque:# when the local deque is emptynewval=next(it)# fetch a new value andfordindeques:# load it to all the dequesd.append(newval)yieldmydeque.popleft()returntuple(gen(d)fordindeques)

Oncetee() has made a split, the originaliterable should not beused anywhere else; otherwise, theiterable could get advanced withoutthe tee objects being informed.

tee iterators are not threadsafe. ARuntimeError may beraised when using simultaneously iterators returned by the sametee()call, even if the originaliterable is threadsafe.

This itertool may require significant auxiliary storage (depending on howmuch temporary data needs to be stored). In general, if one iterator usesmost or all of the data before another iterator starts, it is faster to uselist() instead oftee().

New in version 2.4.

9.7.2.Recipes¶

This section shows recipes for creating an extended toolset using the existingitertools as building blocks.

The extended tools offer the same high performance as the underlying toolset.The superior memory performance is kept by processing elements one at a timerather than bringing the whole iterable into memory all at once. Code volume iskept small by linking the tools together in a functional style which helpseliminate temporary variables. High speed is retained by preferring“vectorized” building blocks over the use of for-loops andgeneratorswhich incur interpreter overhead.

deftake(n,iterable):"Return first n items of the iterable as a list"returnlist(islice(iterable,n))deftabulate(function,start=0):"Return function(0), function(1), ..."returnimap(function,count(start))defconsume(iterator,n=None):"Advance the iterator n-steps ahead. If n is None, consume entirely."# Use functions that consume iterators at C speed.ifnisNone:# feed the entire iterator into a zero-length dequecollections.deque(iterator,maxlen=0)else:# advance to the empty slice starting at position nnext(islice(iterator,n,n),None)defnth(iterable,n,default=None):"Returns the nth item or a default value"returnnext(islice(iterable,n,None),default)defall_equal(iterable):"Returns True if all the elements are equal to each other"g=groupby(iterable)returnnext(g,True)andnotnext(g,False)defquantify(iterable,pred=bool):"Count how many times the predicate is true"returnsum(imap(pred,iterable))defpadnone(iterable):"""Returns the sequence elements and then returns None indefinitely.    Useful for emulating the behavior of the built-in map() function.    """returnchain(iterable,repeat(None))defncycles(iterable,n):"Returns the sequence elements n times"returnchain.from_iterable(repeat(tuple(iterable),n))defdotproduct(vec1,vec2):returnsum(imap(operator.mul,vec1,vec2))defflatten(listOfLists):"Flatten one level of nesting"returnchain.from_iterable(listOfLists)defrepeatfunc(func,times=None,*args):"""Repeat calls to func with specified arguments.    Example:  repeatfunc(random.random)    """iftimesisNone:returnstarmap(func,repeat(args))returnstarmap(func,repeat(args,times))defpairwise(iterable):"s -> (s0,s1), (s1,s2), (s2, s3), ..."a,b=tee(iterable)next(b,None)returnizip(a,b)defgrouper(iterable,n,fillvalue=None):"Collect data into fixed-length chunks or blocks"# grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxxargs=[iter(iterable)]*nreturnizip_longest(fillvalue=fillvalue,*args)defroundrobin(*iterables):"roundrobin('ABC', 'D', 'EF') --> A D E B F C"# Recipe credited to George Sakkispending=len(iterables)nexts=cycle(iter(it).nextforitiniterables)whilepending:try:fornextinnexts:yieldnext()exceptStopIteration:pending-=1nexts=cycle(islice(nexts,pending))defpowerset(iterable):"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"s=list(iterable)returnchain.from_iterable(combinations(s,r)forrinrange(len(s)+1))defunique_everseen(iterable,key=None):"List unique elements, preserving order. Remember all elements ever seen."# unique_everseen('AAAABBBCCDAABBB') --> A B C D# unique_everseen('ABBCcAD', str.lower) --> A B C Dseen=set()seen_add=seen.addifkeyisNone:forelementinifilterfalse(seen.__contains__,iterable):seen_add(element)yieldelementelse:forelementiniterable:k=key(element)ifknotinseen:seen_add(k)yieldelementdefunique_justseen(iterable,key=None):"List unique elements, preserving order. Remember only the element just seen."# unique_justseen('AAAABBBCCDAABBB') --> A B C D A B# unique_justseen('ABBCcAD', str.lower) --> A B C A Dreturnimap(next,imap(itemgetter(1),groupby(iterable,key)))defiter_except(func,exception,first=None):""" Call a function repeatedly until an exception is raised.    Converts a call-until-exception interface to an iterator interface.    Like __builtin__.iter(func, sentinel) but uses an exception instead    of a sentinel to end the loop.    Examples:        bsddbiter = iter_except(db.next, bsddb.error, db.first)        heapiter = iter_except(functools.partial(heappop, h), IndexError)        dictiter = iter_except(d.popitem, KeyError)        dequeiter = iter_except(d.popleft, IndexError)        queueiter = iter_except(q.get_nowait, Queue.Empty)        setiter = iter_except(s.pop, KeyError)    """try:iffirstisnotNone:yieldfirst()while1:yieldfunc()exceptexception:passdefrandom_product(*args,**kwds):"Random selection from itertools.product(*args, **kwds)"pools=map(tuple,args)*kwds.get('repeat',1)returntuple(random.choice(pool)forpoolinpools)defrandom_permutation(iterable,r=None):"Random selection from itertools.permutations(iterable, r)"pool=tuple(iterable)r=len(pool)ifrisNoneelserreturntuple(random.sample(pool,r))defrandom_combination(iterable,r):"Random selection from itertools.combinations(iterable, r)"pool=tuple(iterable)n=len(pool)indices=sorted(random.sample(xrange(n),r))returntuple(pool[i]foriinindices)defrandom_combination_with_replacement(iterable,r):"Random selection from itertools.combinations_with_replacement(iterable, r)"pool=tuple(iterable)n=len(pool)indices=sorted(random.randrange(n)foriinxrange(r))returntuple(pool[i]foriinindices)deftee_lookahead(t,i):"""Inspect the i-th upcomping value from a tee object       while leaving the tee object at its current position.       Raise an IndexError if the underlying iterator doesn't       have enough values.    """forvalueinislice(t.__copy__(),i,None):returnvalueraiseIndexError(i)

Note, many of the above recipes can be optimized by replacing global lookupswith local variables defined as default values. For example, thedotproduct recipe can be written as:

defdotproduct(vec1,vec2,sum=sum,imap=imap,mul=operator.mul):returnsum(imap(mul,vec1,vec2))