Bitwise boolean¶

Bitwise boolean operators like== and!= will return a booleanSeries,which is almost always what you want anyways.

>>>s=pd.Series(range(5))>>>s==40    False1    False2    False3    False4     Truedtype: bool

Seeboolean comparisons for more examples.

Using thein operator¶

Using the Pythonin operator on a Series tests for membership in theindex, not membership among the values.

If this behavior is surprising, keep in mind that usingin on a Pythondictionary tests keys, not values, and Series are dict-like.To test for membership in the values, use the methodisin():

For DataFrames, likewise,in applies to the column axis,testing for membership in the list of column names.

Choice ofNA representation¶

For lack ofNA (missing) support from the ground up in NumPy and Python ingeneral, we were given the difficult choice between either

• Amasked array solution: an array of data and an array of boolean valuesindicating whether a value
• Using a special sentinel value, bit pattern, or set of sentinel values todenoteNA across the dtypes

For many reasons we chose the latter. After years of production use it hasproven, at least in my opinion, to be the best decision given the state ofaffairs in NumPy and Python in general. The special valueNaN(Not-A-Number) is used everywhere as theNA value, and there are APIfunctionsisnull andnotnull which can be used across the dtypes todetect NA values.

However, it comes with it a couple of trade-offs which I most certainly havenot ignored.

Support for integerNA¶

In the absence of high performanceNA support being built into NumPy fromthe ground up, the primary casualty is the ability to represent NAs in integerarrays. For example:

In [5]:s=pd.Series([1,2,3,4,5],index=list('abcde'))In [6]:sOut[6]:a    1b    2c    3d    4e    5dtype: int64In [7]:s.dtypeOut[7]:dtype('int64')In [8]:s2=s.reindex(['a','b','c','f','u'])In [9]:s2Out[9]:a    1.0b    2.0c    3.0f    NaNu    NaNdtype: float64In [10]:s2.dtypeOut[10]:dtype('float64')

This trade-off is made largely for memory and performance reasons, and also sothat the resulting Series continues to be “numeric”. One possibility is to usedtype=object arrays instead.

NA type promotions¶

When introducing NAs into an existing Series or DataFrame viareindex orsome other means, boolean and integer types will be promoted to a differentdtype in order to store the NAs. These are summarized by this table:

While this may seem like a heavy trade-off, I have found very fewcases where this is an issue in practice. Some explanation for the motivationhere in the next section.

Why not make NumPy like R?¶

Many people have suggested that NumPy should simply emulate theNA supportpresent in the more domain-specific statistical programming languageR. Part of the reason is the NumPy type hierarchy:

The R language, by contrast, only has a handful of built-in data types:integer,numeric (floating-point),character, andboolean.NA types are implemented by reserving special bit patterns foreach type to be used as the missing value. While doing this with the full NumPytype hierarchy would be possible, it would be a more substantial trade-off(especially for the 8- and 16-bit data types) and implementation undertaking.

An alternate approach is that of using masked arrays. A masked array is anarray of data with an associated booleanmask denoting whether each valueshould be consideredNA or not. I am personally not in love with thisapproach as I feel that overall it places a fairly heavy burden on the user andthe library implementer. Additionally, it exacts a fairly high performance costwhen working with numerical data compared with the simple approach of usingNaN. Thus, I have chosen the Pythonic “practicality beats purity” approachand traded integerNA capability for a much simpler approach of using aspecial value in float and object arrays to denoteNA, and promotinginteger arrays to floating when NAs must be introduced.

Non-monotonic indexes require exact matches¶

If the index of aSeries orDataFrame is monotonically increasing or decreasing, then the boundsof a label-based slice can be outside the range of the index, much like slice indexing anormal Pythonlist. Monotonicity of an index can be tested with theis_monotonic_increasing andis_monotonic_decreasing attributes.

In [11]:df=pd.DataFrame(index=[2,3,3,4,5],columns=['data'],data=range(5))In [12]:df.index.is_monotonic_increasingOut[12]:True# no rows 0 or 1, but still returns rows 2, 3 (both of them), and 4:In [13]:df.loc[0:4,:]Out[13]:   data2     03     13     24     3# slice is are outside the index, so empty DataFrame is returnedIn [14]:df.loc[13:15,:]Out[14]:Empty DataFrameColumns: [data]Index: []

On the other hand, if the index is not monotonic, then both slice bounds must beunique members of the index.

In [15]:df=pd.DataFrame(index=[2,3,1,4,3,5],columns=['data'],data=range(6))In [16]:df.index.is_monotonic_increasingOut[16]:False# OK because 2 and 4 are in the indexIn [17]:df.loc[2:4,:]Out[17]:   data2     03     11     24     3

# 0 is not in the indexIn[9]:df.loc[0:4,:]KeyError:0# 3 is not a unique labelIn[11]:df.loc[2:3,:]KeyError:'Cannot get right slice bound for non-unique label: 3'

Endpoints are inclusive¶

Compared with standard Python sequence slicing in which the slice endpoint isnot inclusive, label-based slicing in pandasis inclusive. The primaryreason for this is that it is often not possible to easily determine the“successor” or next element after a particular label in an index. For example,consider the following Series:

In [18]:s=pd.Series(np.random.randn(6),index=list('abcdef'))In [19]:sOut[19]:a    1.544821b   -1.708552c    1.545458d   -0.735738e   -0.649091f   -0.403878dtype: float64

Suppose we wished to slice fromc toe, using integers this would be

In [20]:s[2:5]Out[20]:c    1.545458d   -0.735738e   -0.649091dtype: float64

However, if you only hadc ande, determining the next element in theindex can be somewhat complicated. For example, the following does not work:

s.ix['c':'e'+1]

A very common use case is to limit a time series to start and end at twospecific dates. To enable this, we made the design design to make label-basedslicing include both endpoints:

In [21]:s.ix['c':'e']Out[21]:c    1.545458d   -0.735738e   -0.649091dtype: float64

This is most definitely a “practicality beats purity” sort of thing, but it issomething to watch out for if you expect label-based slicing to behave exactlyin the way that standard Python integer slicing works.

Reindex versus ix gotchas¶

Many users will find themselves using theix indexing capabilities as aconcise means of selecting data from a pandas object:

In [22]:df=pd.DataFrame(np.random.randn(6,4),columns=['one','two','three','four'],   ....:index=list('abcdef'))   ....:In [23]:dfOut[23]:        one       two     three      foura -2.474932  0.975891 -0.204206  0.452707b  3.478418 -0.591538 -0.508560  0.047946c -0.170009 -1.615606 -0.894382  1.334681d -0.418002 -0.690649  0.128522  0.429260e  1.207515 -1.308877 -0.548792 -1.520879f  1.153696  0.609378 -0.825763  0.218223In [24]:df.ix[['b','c','e']]Out[24]:        one       two     three      fourb  3.478418 -0.591538 -0.508560  0.047946c -0.170009 -1.615606 -0.894382  1.334681e  1.207515 -1.308877 -0.548792 -1.520879

This is, of course, completely equivalentin this case to using thereindex method:

In [25]:df.reindex(['b','c','e'])Out[25]:        one       two     three      fourb  3.478418 -0.591538 -0.508560  0.047946c -0.170009 -1.615606 -0.894382  1.334681e  1.207515 -1.308877 -0.548792 -1.520879

Some might conclude thatix andreindex are 100% equivalent based onthis. This is indeed trueexcept in the case of integer indexing. Forexample, the above operation could alternately have been expressed as:

In [26]:df.ix[[1,2,4]]Out[26]:        one       two     three      fourb  3.478418 -0.591538 -0.508560  0.047946c -0.170009 -1.615606 -0.894382  1.334681e  1.207515 -1.308877 -0.548792 -1.520879

If you pass[1,2,4] toreindex you will get another thing entirely:

In [27]:df.reindex([1,2,4])Out[27]:   one  two  three  four1  NaN  NaN    NaN   NaN2  NaN  NaN    NaN   NaN4  NaN  NaN    NaN   NaN

So it’s important to remember thatreindex isstrict label indexingonly. This can lead to some potentially surprising results in pathologicalcases where an index contains, say, both integers and strings:

In [28]:s=pd.Series([1,2,3],index=['a',0,1])In [29]:sOut[29]:a    10    21    3dtype: int64In [30]:s.ix[[0,1]]Out[30]:0    21    3dtype: int64In [31]:s.reindex([0,1])Out[31]:0    21    3dtype: int64

Because the index in this case does not contain solely integers,ix fallsback on integer indexing. By contrast,reindex only looks for the valuespassed in the index, thus finding the integers0 and1. While it wouldbe possible to insert some logic to check whether a passed sequence is allcontained in the index, that logic would exact a very high cost in large datasets.

Reindex potentially changes underlying Series dtype¶

The use ofreindex_like can potentially change the dtype of aSeries.

In [32]:series=pd.Series([1,2,3])In [33]:x=pd.Series([True])In [34]:x.dtypeOut[34]:dtype('bool')In [35]:x=pd.Series([True]).reindex_like(series)In [36]:x.dtypeOut[36]:dtype('O')

This is becausereindex_like silently insertsNaNs and thedtypechanges accordingly. This can cause some issues when usingnumpyufuncssuch asnumpy.logical_and.

See thethis old issue for a moredetailed discussion.