Search

Enter search terms or a module, class or function name.

Enhancing Performance¶

Cython (Writing C extensions for pandas)¶

For many use cases writing pandas in pure python and numpy is sufficient. In somecomputationally heavy applications however, it can be possible to achieve sizeablespeed-ups by offloading work tocython.

This tutorial assumes you have refactored as much as possible in python, for exampletrying to remove for loops and making use of numpy vectorization, it’s always worthoptimising in python first.

This tutorial walks through a “typical” process of cythonizing a slow computation.We use anexample from the cython documentationbut in the context of pandas. Our final cythonized solution is around 100 timesfaster than the pure python.

Pure python¶

We have a DataFrame to which we want to apply a function row-wise.

In [1]:df=pd.DataFrame({'a':np.random.randn(1000),   ...:'b':np.random.randn(1000),   ...:'N':np.random.randint(100,1000,(1000)),   ...:'x':'x'})   ...:In [2]:dfOut[2]:       N         a         b  x0    585  0.469112 -0.218470  x1    841 -0.282863 -0.061645  x2    251 -1.509059 -0.723780  x3    972 -1.135632  0.551225  x4    181  1.212112 -0.497767  x5    458 -0.173215  0.837519  x6    159  0.119209  1.103245  x..   ...       ...       ... ..993  190  0.131892  0.290162  x994  931  0.342097  0.215341  x995  374 -1.512743  0.874737  x996  246  0.933753  1.120790  x997  157 -0.308013  0.198768  x998  977 -0.079915  1.757555  x999  770 -1.010589 -1.115680  x[1000 rows x 4 columns]

Here’s the function in pure python:

In [3]:deff(x):   ...:returnx*(x-1)   ...:In [4]:defintegrate_f(a,b,N):   ...:s=0   ...:dx=(b-a)/N   ...:foriinrange(N):   ...:s+=f(a+i*dx)   ...:returns*dx   ...:

We achieve our result by usingapply (row-wise):

In [7]:%timeitdf.apply(lambdax:integrate_f(x['a'],x['b'],x['N']),axis=1)10 loops, best of 3: 174 ms per loop

But clearly this isn’t fast enough for us. Let’s take a look and see where thetime is spent during this operation (limited to the most time consumingfour calls) using theprun ipython magic function:

In [5]:%prun-l4df.apply(lambdax:integrate_f(x['a'],x['b'],x['N']),axis=1)         671915 function calls (666906 primitive calls) in 0.440 seconds   Ordered by: internal time   List reduced from 128 to 4 due to restriction <4>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)     1000    0.211    0.000    0.322    0.000 <ipython-input-4-91e33489f136>:1(integrate_f)   552423    0.097    0.000    0.097    0.000 <ipython-input-3-bc41a25943f6>:1(f)     3000    0.014    0.000    0.077    0.000 base.py:2146(get_value)     1000    0.014    0.000    0.014    0.000 {range}

By far the majority of time is spend inside eitherintegrate_f orf,hence we’ll concentrate our efforts cythonizing these two functions.

Note

In python 2 replacing therange with its generator counterpart (xrange)would mean therange line would vanish. In python 3range is already a generator.

Plain cython¶

First we’re going to need to import the cython magic function to ipython (forcython versions < 0.21 you can use%load_extcythonmagic):

In [6]:%load_extCython

Now, let’s simply copy our functions over to cython as is (the suffixis here to distinguish between function versions):

In [7]:%%cython   ...:deff_plain(x):   ...:returnx*(x-1)   ...:defintegrate_f_plain(a,b,N):   ...:s=0   ...:dx=(b-a)/N   ...:foriinrange(N):   ...:s+=f_plain(a+i*dx)   ...:returns*dx   ...:

Note

If you’re having trouble pasting the above into your ipython, you may needto be using bleeding edge ipython for paste to play well with cell magics.

In [4]:%timeitdf.apply(lambdax:integrate_f_plain(x['a'],x['b'],x['N']),axis=1)10 loops, best of 3: 85.5 ms per loop

Already this has shaved a third off, not too bad for a simple copy and paste.

Adding type¶

We get another huge improvement simply by providing type information:

In [8]:%%cython   ...:cdefdoublef_typed(doublex)except?-2:   ...:returnx*(x-1)   ...:cpdefdoubleintegrate_f_typed(doublea,doubleb,intN):   ...:cdefinti   ...:cdefdoubles,dx   ...:s=0   ...:dx=(b-a)/N   ...:foriinrange(N):   ...:s+=f_typed(a+i*dx)   ...:returns*dx   ...:

In [4]:%timeitdf.apply(lambdax:integrate_f_typed(x['a'],x['b'],x['N']),axis=1)10 loops, best of 3: 20.3 ms per loop

Now, we’re talking! It’s now over ten times faster than the original pythonimplementation, and we haven’treally modified the code. Let’s have anotherlook at what’s eating up time:

In [9]:%prun-l4df.apply(lambdax:integrate_f_typed(x['a'],x['b'],x['N']),axis=1)         118490 function calls (113481 primitive calls) in 0.149 seconds   Ordered by: internal time   List reduced from 124 to 4 due to restriction <4>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)     3000    0.018    0.000    0.100    0.000 base.py:2146(get_value)     3000    0.011    0.000    0.114    0.000 series.py:598(__getitem__)        1    0.008    0.008    0.148    0.148 {pandas.lib.reduce}     9024    0.008    0.000    0.018    0.000 {getattr}

Using ndarray¶

It’s calling series... a lot! It’s creating a Series from each row, and get-ting from boththe index and the series (three times for each row). Function calls are expensivein python, so maybe we could minimise these by cythonizing the apply part.

Note

We are now passing ndarrays into the cython function, fortunately cython playsvery nicely with numpy.

In [10]:%%cython   ....:cimportnumpyasnp   ....:importnumpyasnp   ....:cdefdoublef_typed(doublex)except?-2:   ....:returnx*(x-1)   ....:cpdefdoubleintegrate_f_typed(doublea,doubleb,intN):   ....:cdefinti   ....:cdefdoubles,dx   ....:s=0   ....:dx=(b-a)/N   ....:foriinrange(N):   ....:s+=f_typed(a+i*dx)   ....:returns*dx   ....:cpdefnp.ndarray[double]apply_integrate_f(np.ndarraycol_a,np.ndarraycol_b,np.ndarraycol_N):   ....:assert(col_a.dtype==np.floatandcol_b.dtype==np.floatandcol_N.dtype==np.int)   ....:cdefPy_ssize_ti,n=len(col_N)   ....:assert(len(col_a)==len(col_b)==n)   ....:cdefnp.ndarray[double]res=np.empty(n)   ....:foriinrange(len(col_a)):   ....:res[i]=integrate_f_typed(col_a[i],col_b[i],col_N[i])   ....:returnres   ....:

The implementation is simple, it creates an array of zeros and loops overthe rows, applying ourintegrate_f_typed, and putting this in the zeros array.

Warning

In 0.13.0 sinceSeries has internaly been refactored to no longer sub-classndarraybut instead subclassNDFrame, you cannot pass aSeries directly as andarray typed parameterto a cython function. Instead pass the actualndarray using the.values attribute of the Series.

Prior to 0.13.0

apply_integrate_f(df['a'],df['b'],df['N'])

Use.values to get the underlyingndarray

apply_integrate_f(df['a'].values,df['b'].values,df['N'].values)

Note

Loops like this would beextremely slow in python, but in Cython loopingover numpy arrays isfast.

In [4]:%timeitapply_integrate_f(df['a'].values,df['b'].values,df['N'].values)1000 loops, best of 3: 1.25 ms per loop

We’ve gotten another big improvement. Let’s check again where the time is spent:

In [11]:%prun-l4apply_integrate_f(df['a'].values,df['b'].values,df['N'].values)         240 function calls in 0.002 seconds   Ordered by: internal time   List reduced from 57 to 4 due to restriction <4>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)        1    0.002    0.002    0.002    0.002 {_cython_magic_40485b2751cb6bc085f3a7be0856f402.apply_integrate_f}        6    0.000    0.000    0.000    0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}        3    0.000    0.000    0.000    0.000 internals.py:3563(iget)        9    0.000    0.000    0.000    0.000 generic.py:2746(__setattr__)

As one might expect, the majority of the time is now spent inapply_integrate_f,so if we wanted to make anymore efficiencies we must continue to concentrate ourefforts here.

More advanced techniques¶

There is still hope for improvement. Here’s an example of using some moreadvanced cython techniques:

In [12]:%%cython   ....:cimportcython   ....:cimportnumpyasnp   ....:importnumpyasnp   ....:cdefdoublef_typed(doublex)except?-2:   ....:returnx*(x-1)   ....:cpdefdoubleintegrate_f_typed(doublea,doubleb,intN):   ....:cdefinti   ....:cdefdoubles,dx   ....:s=0   ....:dx=(b-a)/N   ....:foriinrange(N):   ....:s+=f_typed(a+i*dx)   ....:returns*dx   ....:@cython.boundscheck(False)   ....:@cython.wraparound(False)   ....:cpdefnp.ndarray[double]apply_integrate_f_wrap(np.ndarray[double]col_a,np.ndarray[double]col_b,np.ndarray[int]col_N):   ....:cdefinti,n=len(col_N)   ....:assertlen(col_a)==len(col_b)==n   ....:cdefnp.ndarray[double]res=np.empty(n)   ....:foriinrange(n):   ....:res[i]=integrate_f_typed(col_a[i],col_b[i],col_N[i])   ....:returnres   ....:

In [4]:%timeitapply_integrate_f_wrap(df['a'].values,df['b'].values,df['N'].values)1000 loops, best of 3: 987 us per loop

Even faster, with the caveat that a bug in our cython code (an off-by-one error,for example) might cause a segfault because memory access isn’t checked.

Using numba¶

A recent alternative to statically compiling cython code, is to use adynamic jit-compiler,numba.

Numba gives you the power to speed up your applications with high performance functions written directly in Python. With a few annotations, array-oriented and math-heavy Python code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran, without having to switch languages or Python interpreters.

Numba works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool). Numba supports compilation of Python to run on either CPU or GPU hardware, and is designed to integrate with the Python scientific software stack.

Note

You will need to installnumba. This is easy withconda, by using:condainstallnumba, seeinstalling using miniconda.

Note

As ofnumba version 0.20, pandas objects cannot be passed directly to numba-compiled functions. Instead, one must pass thenumpy array underlying thepandas object to the numba-compiled function as demonstrated below.

Jit¶

Usingnumba to just-in-time compile your code. We simply take the plain python code from above and annotate with the@jit decorator.

importnumba@numba.jitdeff_plain(x):returnx*(x-1)@numba.jitdefintegrate_f_numba(a,b,N):s=0dx=(b-a)/Nforiinrange(N):s+=f_plain(a+i*dx)returns*dx@numba.jitdefapply_integrate_f_numba(col_a,col_b,col_N):n=len(col_N)result=np.empty(n,dtype='float64')assertlen(col_a)==len(col_b)==nforiinrange(n):result[i]=integrate_f_numba(col_a[i],col_b[i],col_N[i])returnresultdefcompute_numba(df):result=apply_integrate_f_numba(df['a'].values,df['b'].values,df['N'].values)returnpd.Series(result,index=df.index,name='result')

Note that we directly passnumpy arrays to the numba function.compute_numba is just a wrapper that provides a nicer interface by passing/returning pandas objects.

In [4]:%timeitcompute_numba(df)1000 loops, best of 3: 798 us per loop

Vectorize¶

numba can also be used to write vectorized functions that do not require the user to explicitlyloop over the observations of a vector; a vectorized function will be applied to each row automatically.Consider the following toy example of doubling each observation:

importnumbadefdouble_every_value_nonumba(x):returnx*2@numba.vectorizedefdouble_every_value_withnumba(x):returnx*2# Custom function without numbaIn[5]:%timeitdf['col1_doubled']=df.a.apply(double_every_value_nonumba)1000loops,bestof3:797usperloop# Standard implementation (faster than a custom function)In[6]:%timeitdf['col1_doubled']=df.a*21000loops,bestof3:233usperloop# Custom function with numbaIn[7]:%timeitdf['col1_doubled']=double_every_value_withnumba(df.a.values)1000loops,bestof3:145usperloop

Caveats¶

Note

numba will execute on any function, but can only accelerate certain classes of functions.

numba is best at accelerating functions that apply numerical functions to numpy arrays. When passed a function that only uses operations it knows how to accelerate, it will execute innopython mode.

Ifnumba is passed a function that includes something it doesn’t know how to work with – a category that currently includes sets, lists, dictionaries, or string functions – it will revert toobjectmode. Inobjectmode, numba will execute but your code will not speed up significantly. If you would prefer thatnumba throw an error if it cannot compile a function in a way that speeds up your code, pass numba the argumentnopython=True (e.g.@numba.jit(nopython=True)). For more on troubleshootingnumba modes, see thenumba troubleshooting page.

Expression Evaluation via`eval()` (Experimental)¶

New in version 0.13.

The top-level functionpandas.eval() implements expression evaluation ofSeries andDataFrame objects.

Note

To benefit from usingeval() you need toinstallnumexpr. See therecommended dependencies section for more details.

The point of usingeval() for expression evaluation rather thanplain Python is two-fold: 1) largeDataFrame objects areevaluated more efficiently and 2) large arithmetic and boolean expressions areevaluated all at once by the underlying engine (by defaultnumexpr is usedfor evaluation).

Note

You should not useeval() for simpleexpressions or for expressions involving small DataFrames. In fact,eval() is many orders of magnitude slower forsmaller expressions/objects than plain ol’ Python. A good rule of thumb isto only useeval() when you have aDataFrame with more than 10,000 rows.

eval() supports all arithmetic expressions supported by theengine in addition to some extensions available only in pandas.

Note

The larger the frame and the larger the expression the more speedup you willsee from usingeval().

Supported Syntax¶

These operations are supported bypandas.eval():

Arithmetic operations except for the left shift (<<) and right shift(>>) operators, e.g.,df+2*pi/s**4%42-the_golden_ratio
Comparison operations, including chained comparisons, e.g.,2<df<df2
Boolean operations, e.g.,df<df2anddf3<df4ornotdf_bool
list andtuple literals, e.g.,[1,2] or(1,2)
Attribute access, e.g.,df.a
Subscript expressions, e.g.,df[0]
Simple variable evaluation, e.g.,pd.eval('df') (this is not very useful)
Math functions,sin,cos,exp,log,expm1,log1p,sqrt,sinh,cosh,tanh,arcsin,arccos,arctan,arccosh,arcsinh,arctanh,abs andarctan2.

This Python syntax isnot allowed:

Expressions
- Function calls other than math functions.
- is/isnot operations
- if expressions
- lambda expressions
- list/set/dict comprehensions
- Literaldict andset expressions
- yield expressions
- Generator expressions
- Boolean expressions consisting of only scalar values
Statements
- Neithersimplenorcompoundstatements are allowed. This includes things likefor,while, andif.

`eval()` Examples¶

pandas.eval() works well with expressions containing large arrays.

First let’s create a few decent-sized arrays to play with:

In [13]:nrows,ncols=20000,100In [14]:df1,df2,df3,df4=[pd.DataFrame(np.random.randn(nrows,ncols))for_inrange(4)]

Now let’s compare adding them together using plain ol’ Python versuseval():

In [15]:%timeitdf1+df2+df3+df410 loops, best of 3: 30.1 ms per loop

In [16]:%timeitpd.eval('df1 + df2 + df3 + df4')100 loops, best of 3: 11.6 ms per loop

Now let’s do the same thing but with comparisons:

In [17]:%timeit(df1>0)&(df2>0)&(df3>0)&(df4>0)10 loops, best of 3: 53.5 ms per loop

In [18]:%timeitpd.eval('(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)')10 loops, best of 3: 20.5 ms per loop

eval() also works with unaligned pandas objects:

In [19]:s=pd.Series(np.random.randn(50))In [20]:%timeitdf1+df2+df3+df4+s10 loops, best of 3: 56.9 ms per loop

In [21]:%timeitpd.eval('df1 + df2 + df3 + df4 + s')100 loops, best of 3: 12.9 ms per loop

Note

Operations such as

1and2# would parse to 1 & 2, but should evaluate to 23or4# would parse to 3 | 4, but should evaluate to 3~1# this is okay, but slower when using eval

should be performed in Python. An exception will be raised if you try toperform any boolean/bitwise operations with scalar operands that are notof typebool ornp.bool_. Again, you should perform these kinds ofoperations in plain Python.

The`DataFrame.eval` method (Experimental)¶

New in version 0.13.

In addition to the top levelpandas.eval() function you can alsoevaluate an expression in the “context” of aDataFrame.

In [22]:df=pd.DataFrame(np.random.randn(5,2),columns=['a','b'])In [23]:df.eval('a + b')Out[23]:0   -0.2467471    0.8677862   -1.6260633   -1.1349784   -1.027798dtype: float64

Any expression that is a validpandas.eval() expression is also a validDataFrame.eval() expression, with the added benefit that you don’t have toprefix the name of theDataFrame to the column(s) you’reinterested in evaluating.

In addition, you can perform assignment of columns within an expression.This allows forformulaic evaluation. The assignment target can be anew column name or an existing column name, and it must be a valid Pythonidentifier.

New in version 0.18.0.

Theinplace keyword determines whether this assignment will performedon the originalDataFrame or return a copy with the new column.

Warning

For backwards compatability,inplace defaults toTrue if notspecified. This will change in a future version of pandas - if yourcode depends on an inplace assignment you should update to explicitlysetinplace=True

In [24]:df=pd.DataFrame(dict(a=range(5),b=range(5,10)))In [25]:df.eval('c = a + b',inplace=True)In [26]:df.eval('d = a + b + c',inplace=True)In [27]:df.eval('a = 1',inplace=True)In [28]:dfOut[28]:   a  b   c   d0  1  5   5  101  1  6   7  142  1  7   9  183  1  8  11  224  1  9  13  26

Wheninplace is set toFalse, a copy of theDataFrame with thenew or modified columns is returned and the original frame is unchanged.

In [29]:dfOut[29]:   a  b   c   d0  1  5   5  101  1  6   7  142  1  7   9  183  1  8  11  224  1  9  13  26In [30]:df.eval('e = a - c',inplace=False)Out[30]:   a  b   c   d   e0  1  5   5  10  -41  1  6   7  14  -62  1  7   9  18  -83  1  8  11  22 -104  1  9  13  26 -12In [31]:dfOut[31]:   a  b   c   d0  1  5   5  101  1  6   7  142  1  7   9  183  1  8  11  224  1  9  13  26

New in version 0.18.0.

As a convenience, multiple assignments can be performed by using amulti-line string.

In [32]:df.eval("""   ....: c = a + b   ....: d = a + b + c   ....: a = 1""",inplace=False)   ....:Out[32]:   a  b   c   d0  1  5   6  121  1  6   7  142  1  7   8  163  1  8   9  184  1  9  10  20

The equivalent in standard Python would be

In [33]:df=pd.DataFrame(dict(a=range(5),b=range(5,10)))In [34]:df['c']=df.a+df.bIn [35]:df['d']=df.a+df.b+df.cIn [36]:df['a']=1In [37]:dfOut[37]:   a  b   c   d0  1  5   5  101  1  6   7  142  1  7   9  183  1  8  11  224  1  9  13  26

New in version 0.18.0.

Thequery method gained theinplace keyword which determineswhether the query modifies the original frame.

In [38]:df=pd.DataFrame(dict(a=range(5),b=range(5,10)))In [39]:df.query('a > 2')Out[39]:   a  b3  3  84  4  9In [40]:df.query('a > 2',inplace=True)In [41]:dfOut[41]:   a  b3  3  84  4  9

Warning

Unlike witheval, the default value forinplace forqueryisFalse. This is consistent with prior versions of pandas.

Local Variables¶

In pandas version 0.14 the local variable API has changed. In pandas 0.13.x,you could refer to local variables the same way you would in standard Python.For example,

df=pd.DataFrame(np.random.randn(5,2),columns=['a','b'])newcol=np.random.randn(len(df))df.eval('b + newcol')UndefinedVariableError:name'newcol'isnotdefined

As you can see from the exception generated, this syntax is no longer allowed.You mustexplicitly reference any local variable that you want to use in anexpression by placing the@ character in front of the name. For example,

In [42]:df=pd.DataFrame(np.random.randn(5,2),columns=list('ab'))In [43]:newcol=np.random.randn(len(df))In [44]:df.eval('b + @newcol')Out[44]:0   -0.1739261    2.4930832   -0.8818313   -0.6910454    1.334703dtype: float64In [45]:df.query('b < @newcol')Out[45]:          a         b0  0.863987 -0.1159982 -2.621419 -1.297879

If you don’t prefix the local variable with@, pandas will raise anexception telling you the variable is undefined.

When usingDataFrame.eval() andDataFrame.query(), this allows youto have a local variable and aDataFrame column with the samename in an expression.

In [46]:a=np.random.randn()In [47]:df.query('@a < a')Out[47]:          a         b0  0.863987 -0.115998In [48]:df.loc[a<df.a]# same as the previous expressionOut[48]:          a         b0  0.863987 -0.115998

Withpandas.eval() you cannot use the@ prefixat all, because itisn’t defined in that context.pandas will let you know this if you try touse@ in a top-level call topandas.eval(). For example,

In [49]:a,b=1,2In [50]:pd.eval('@a + b')  File "<string>", line unknownSyntaxError: The '@' prefix is not allowed in top-level eval calls,please refer to your variables by name without the '@' prefix

In this case, you should simply refer to the variables like you would instandard Python.

In [51]:pd.eval('a + b')Out[51]:3

`pandas.eval()` Parsers¶

There are two different parsers and two different engines you can use asthe backend.

The default'pandas' parser allows a more intuitive syntax for expressingquery-like operations (comparisons, conjunctions and disjunctions). Inparticular, the precedence of the& and| operators is made equal tothe precedence of the corresponding boolean operationsand andor.

For example, the above conjunction can be written without parentheses.Alternatively, you can use the'python' parser to enforce strict Pythonsemantics.

In [52]:expr='(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)'In [53]:x=pd.eval(expr,parser='python')In [54]:expr_no_parens='df1 > 0 & df2 > 0 & df3 > 0 & df4 > 0'In [55]:y=pd.eval(expr_no_parens,parser='pandas')In [56]:np.all(x==y)Out[56]:True

The same expression can be “anded” together with the wordand aswell:

In [57]:expr='(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)'In [58]:x=pd.eval(expr,parser='python')In [59]:expr_with_ands='df1 > 0 and df2 > 0 and df3 > 0 and df4 > 0'In [60]:y=pd.eval(expr_with_ands,parser='pandas')In [61]:np.all(x==y)Out[61]:True

Theand andor operators here have the same precedence that they wouldin vanilla Python.

`pandas.eval()` Backends¶

There’s also the option to makeeval() operate identical to plainol’ Python.

Note

Using the'python' engine is generallynot useful, except for testingother evaluation engines against it. You will achieveno performancebenefits usingeval() withengine='python' and in fact mayincur a performance hit.

You can see this by usingpandas.eval() with the'python' engine. Itis a bit slower (not by much) than evaluating the same expression in Python

In [62]:%timeitdf1+df2+df3+df410 loops, best of 3: 32.2 ms per loop

In [63]:%timeitpd.eval('df1 + df2 + df3 + df4',engine='python')10 loops, best of 3: 33.4 ms per loop

`pandas.eval()` Performance¶

eval() is intended to speed up certain kinds of operations. Inparticular, those operations involving complex expressions with largeDataFrame/Series objects should see asignificant performance benefit. Here is a plot showing the running time ofpandas.eval() as function of the size of the frame involved in thecomputation. The two lines are two different engines.

Note

Operations with smallish objects (around 15k-20k rows) are faster usingplain Python:

This plot was created using aDataFrame with 3 columns each containingfloating point values generated usingnumpy.random.randn().

Technical Minutia Regarding Expression Evaluation¶

Expressions that would result in an object dtype or involve datetime operations(because ofNaT) must be evaluated in Python space. The main reason forthis behavior is to maintain backwards compatibility with versions of numpy <1.7. In those versions ofnumpy a call tondarray.astype(str) willtruncate any strings that are more than 60 characters in length. Second, wecan’t passobject arrays tonumexpr thus string comparisons must beevaluated in Python space.

The upshot is that thisonly applies to object-dtype’d expressions. So, ifyou have an expression–for example

In [64]:df=pd.DataFrame({'strings':np.repeat(list('cba'),3),   ....:'nums':np.repeat(range(3),3)})   ....:In [65]:dfOut[65]:   nums strings0     0       c1     0       c2     0       c3     1       b4     1       b5     1       b6     2       a7     2       a8     2       aIn [66]:df.query('strings == "a" and nums == 1')Out[66]:Empty DataFrameColumns: [nums, strings]Index: []

the numeric part of the comparison (nums==1) will be evaluated bynumexpr.

In general,DataFrame.query()/pandas.eval() willevaluate the subexpressions thatcan be evaluated bynumexpr and thosethat must be evaluated in Python space transparently to the user. This is doneby inferring the result type of an expression from its arguments and operators.

Movatterモバイル変換

Navigation

Table Of Contents

Search

Enhancing Performance¶

Cython (Writing C extensions for pandas)¶

Pure python¶

Plain cython¶

Adding type¶

Using ndarray¶

More advanced techniques¶

Using numba¶

Jit¶

Vectorize¶

Caveats¶

Expression Evaluation via`eval()` (Experimental)¶

Supported Syntax¶

`eval()` Examples¶

The`DataFrame.eval` method (Experimental)¶

Local Variables¶

`pandas.eval()` Parsers¶

`pandas.eval()` Backends¶

`pandas.eval()` Performance¶

Technical Minutia Regarding Expression Evaluation¶

Navigation

Movatterモバイル変換

Table Of Contents

Search

Enhancing Performance¶

Cython (Writing C extensions for pandas)¶

Pure python¶

Plain cython¶

Adding type¶

Using ndarray¶

More advanced techniques¶

Using numba¶

Jit¶

Vectorize¶

Caveats¶

Expression Evaluation viaeval() (Experimental)¶

Supported Syntax¶

eval() Examples¶

TheDataFrame.eval method (Experimental)¶

Local Variables¶

pandas.eval() Parsers¶

pandas.eval() Backends¶

pandas.eval() Performance¶

Technical Minutia Regarding Expression Evaluation¶

Expression Evaluation via`eval()` (Experimental)¶

`eval()` Examples¶

The`DataFrame.eval` method (Experimental)¶

`pandas.eval()` Parsers¶

`pandas.eval()` Backends¶

`pandas.eval()` Performance¶