Language Basics¶
Note
This page uses two different syntax variants:
Cython specific
cdef
syntax, which was designed to make type declarationsconcise and easily readable from a C/C++ perspective.Pure Python syntax which allows static Cython type declarations inpure Python code,followingPEP-484 type hintsandPEP 526 variable annotations.
To make use of C data types in Python syntax, you need to import the special
cython
module in the Python module that you want to compile, e.g.importcython
If you use the pure Python syntax we strongly recommend you use a recentCython 3 release, since significant improvements have been made herecompared to the 0.29.x releases.
Declaring Data Types¶
As a dynamic language, Python encourages a programming style of consideringclasses and objects in terms of their methods and attributes, more than wherethey fit into the class hierarchy.
This can make Python a very relaxed and comfortable language for rapiddevelopment, but with a price - the ‘red tape’ of managing data types isdumped onto the interpreter. At run time, the interpreter does a lot of worksearching namespaces, fetching attributes and parsing argument and keyword tuples.This run-time ‘late binding’ is a major cause of Python’s relative slownesscompared to ‘early binding’ languages such as C++.
However with Cython it is possible to gain significant speed-ups throughthe use of ‘early binding’ programming techniques.
Note
Typing is not a necessity
Providing static typing to parameters and variables is convenience tospeed up your code, but it is not a necessity. Optimize where and when needed.In fact, typing canslow down your code in the case where thetyping does not allow optimizations but where Cython still needs tocheck that the type of some object matches the declared type.
C variable and type definitions¶
C variables can be declared by
using the Cython specific
cdef
statement,using PEP-484/526 type annotations with C data types or
using the function
cython.declare()
.
Thecdef
statement anddeclare()
can define function-local andmodule-level variables as well as attributes in classes, but type annotations onlyaffect local variables and attributes and are ignored at the module level.This is because type annotations are not Cython specific, so Cython keepsthe variables in the module dict (as Python values) instead of making themmodule internal C variables. Usedeclare()
in Python code to explicitlydefine global C variables.
a_global_variable=declare(cython.int)deffunc():i:cython.intj:cython.intk:cython.intf:cython.floatg:cython.float[42]h:cython.p_floati=j=5
cdefinta_global_variabledeffunc():cdefinti,j,kcdeffloatfcdeffloat[42]gcdeffloat *h# cdef float f, g[42], *h # mix of pointers, arrays and values in a single line is deprecatedi=j=5
As known from C, declared global variables are automatically initialised to0
,NULL
orNone
, depending on their type. However, also as knownfrom both Python and C, for a local variable, simply declaring it is not enoughto initialise it. If you use a local variable but did not assign a value, bothCython and the C compiler will issue a warning “local variable … referencedbefore assignment”. You need to assign a value at some point before firstusing the variable, but you can also assign a value directly as part ofthe declaration in most cases:
a_global_variable=declare(cython.int,42)deffunc():i:cython.int=10f:cython.float=2.5g:cython.int[4]=[1,2,3,4]h:cython.p_float=cython.address(f)c:cython.doublecomplex=2+3j
cdefinta_global_variable=42deffunc():cdefinti=10,j,kcdeffloatf=2.5cdefint[4]g=[1,2,3,4]cdeffloat *h=&fcdefdoublecomplexc=2+3j
Note
There is also support for giving names to types using thectypedef
statement or thecython.typedef()
function, e.g.
ULong=cython.typedef(cython.ulong)IntPtr=cython.typedef(cython.p_int)
ctypedefunsignedlongULongctypedefint*IntPtr
C Arrays¶
C array can be declared by adding[ARRAY_SIZE]
to the type of variable:
deffunc():g:cython.float[42]f:cython.int[5][5][5]ptr_char_array:cython.pointer[cython.char[4]]# pointer to the array of 4 charsarray_ptr_char:cython.p_char[4]# array of 4 char pointers
deffunc():cdeffloat[42]gcdefint[5][5][5]fcdefchar[4] *ptr_char_array# pointer to the array of 4 charscdef (char *)[4]array_ptr_char# array of 4 char pointers
Note
Cython syntax currently supports two ways to declare an array:
cdefintarr1[4],arr2[4] #Cstylearraydeclarationcdefint[4]arr1,arr2# Java style array declaration
Both of them generate the same C code, but the Java style is moreconsistent withTyped Memoryviews andFused Types (Templates). The C styledeclaration is soft-deprecated and it’s recommended to use Java styledeclaration instead.
The soft-deprecated C style array declaration doesn’t supportinitialization.
cdefintg[4] = [1, 2, 3, 4] #errorcdefint[4]g=[1,2,3,4]# OKcdefintg[4] #OKbutnotrecommendedg=[1,2,3,4]
Structs, Unions, Enums¶
In addition to the basic types, Cstruct
,union
andenum
are supported:
Grail=cython.struct(age=cython.int,volume=cython.float)defmain():grail:Grail=Grail(5,3.0)print(grail.age,grail.volume)
cdefstructGrail:intagefloatvolumedefmain():cdefGrailgrail=Grail(5,3.0)print(grail.age,grail.volume)
Structs can be declared ascdefpackedstruct
, which hasthe same effect as the C directive#pragmapack(1)
:
cdefpackedstructStructArray:int[4]spamsignedchar[5]eggs
Note
This declaration removes the emptyspace between members that C automatically to ensure that they’re aligned in memory(seeWikipedia article for more details).The main use is that numpy structured arrays store their data in packed form, so acdefpackedstruct
can beused in a memoryview to match that.
Pure python mode does not support packed structs.
The following example shows a declaration of unions:
Food=cython.union(spam=cython.p_char,eggs=cython.p_float)defmain():arr:cython.p_float=[1.0,2.0]spam:Food=Food(spam='b')eggs:Food=Food(eggs=arr)print(spam.spam,eggs.eggs[0])
cdefunionFood:char*spamfloat*eggsdefmain():cdeffloat *arr=[1.0,2.0]cdefFoodspam=Food(spam='b')cdefFoodeggs=Food(eggs=arr)print(spam.spam,eggs.eggs[0])
Enums are created bycdefenum
statement:
cdefenumCheeseType:cheddar,edam,camembertcdefenumCheeseState:hard=1soft=2runny=3print(CheeseType.cheddar)print(CheeseState.hard)
Note
Currently, Pure Python mode does not support enums. (GitHub issue#4252)
Declaring an enum ascpdef
will create aPEP 435-style Python wrapper:
cpdefenumCheeseState:hard=1soft=2runny=3
Up to Cython version 3.0.x, this used to copy all item names into the globalmodule namespace, so that they were available both as attributes of the Pythonenum type (CheseState
above) and as global constants.This was changed in Cython 3.1 to distinguish between anonymous cpdef enums,which only create global Python constants for their items, and named cpdef enums,where the items live only in the namespace of the enum type and do not createglobal Python constants.
To create global constants also for the items of a declared named enum type,you can copy the enum items into the global Python module namespace manually.In Cython before 3.1, this will simply overwrite the global names withtheir own value, which makes it backwards compatible.
globals().update(getattr(CheeseState,'__members__'))
There is currently no special syntax for defining a constant, but you can usean anonymousenum
declaration for this purpose, for example,:
cdefenum:tons_of_spam=3
Note
In the Cython syntax, the wordsstruct
,union
andenum
are used only whendefining a type, not when referring to it. For example, to declare a variablepointing to aGrail
struct, you would write:
cdefGrail *gp
and not:
cdefstructGrail *gp# WRONG
Types¶
The Cython language uses the normal C syntax for C types, including pointers. It providesall the standard C types, namelychar
,short
,int
,long
,longlong
as well as theirunsigned
versions,e.g.unsignedint
(cython.uint
in Python code):
Cython type | Pure Python type |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Note
Additional types are declared in thestdint pxd file.
The specialbint
type is used for C boolean values (int
with 0/non-0values for False/True) andPy_ssize_t
for (signed) sizes of Pythoncontainers.
Pointer types are constructed as in C when using Cython syntax, by appending a*
to the base typethey point to, e.g.int**
for a pointer to a pointer to a C int. In Pure python mode, simple pointer typesuse a naming scheme with “p”s instead, separated from the type name with an underscore, e.g.cython.pp_int
for a pointer toa pointer to a C int. Further pointer types can be constructed with thecython.pointer[]
type construct,e.g.cython.p_int
is equivalent tocython.pointer[cython.int]
.
Arrays use the normal C array syntax, e.g.int[10]
, and the size must be knownat compile time for stack allocated arrays. Cython doesn’t support variable length arrays from C99.Note that Cython uses array access for pointer dereferencing, as*x
is not valid Python syntax,whereasx[0]
is.
Also, the Python typeslist
,dict
,tuple
, etc. may be used forstatic typing, as well as any user definedExtension Types.For example
defmain():foo:list=[]
cdeflistfoo=[]
This requires anexact match of the class, it does not allow subclasses.This allows Cython to optimize code by accessing internals of the builtin class,which is the main reason for declaring builtin types in the first place.
Since Cython 3.1, builtinexception types generally no longer fall under the “exact type” restriction.Thus, declarations likeexc:BaseException
accept all exception objects, as they probably intend.
For declared builtin types, Cython uses internally a C variable of typePyObject*.
Note
The Python typesint
,long
, andfloat
are not available for statictyping in.pyx
files and instead interpreted as Cint
,long
, andfloat
respectively, as statically typing variables with these Pythontypes has zero advantages. On the other hand, annotating in Pure Python withint
,long
, andfloat
Python types will be interpreted asPython object types.
Cython provides an accelerated and typed equivalent of a Python tuple, thectuple
.Actuple
is assembled from any valid C types. For example
defmain():bar:tuple[cython.double,cython.int]
cdef (double,int)bar
They compile down to C-structures and can be used as efficient alternatives toPython tuples.
While these C types can be vastly faster, they have C semantics.Specifically, the integer types overflowand the Cfloat
type only has 32 bits of precision(as opposed to the 64-bit Cdouble
which Python floats wrapand is typically what one wants).If you want to use these numeric Python types simply omit thetype declaration and let them be objects.
Type qualifiers¶
Cython supportsconst
andvolatile
C type qualifiers
defuse_volatile():i:cython.volatile[cython.int]=5@cython.cfuncdefsum(a:cython.const[cython.int],b:cython.const[cython.int])->cython.const[cython.int]:returna+b@cython.cfuncdefpointer_to_const_int(value:cython.pointer[cython.const[cython.int]])->cython.void:# Declares value as pointer to const int type (alias: "cython.p_const_int").# The value can be modified but the object pointed to by value cannot be modified.new_value:cython.int=10print(value[0])value=cython.address(new_value)print(value[0])@cython.cfuncdefconst_pointer_to_int(value:cython.const[cython.pointer[cython.int]])->cython.void:# Declares value as const pointer to int type (alias: "cython.const[cython.p_int]").# Value cannot be modified but the object pointed to by value can be modified.print(value[0])value[0]=10print(value[0])@cython.cfuncdefconst_pointer_to_const_int(value:cython.const[cython.pointer[cython.const[cython.int]]])->cython.void:# Declares value as const pointer to const int type (alias: "cython.const[cython.p_const_int]").# Neither the value variable nor the int pointed to can be modified.print(value[0])
defuse_volatile():cdefvolatileinti=5cdefconstintsum(constinta,constintb):returna+bcdefvoidpointer_to_const_int(constint*value):# Declares value as pointer to const int type.# The value can be modified but the object pointed to by value cannot be modified.cdefintnew_value=10print(value[0])value=&new_valueprint(value[0])cdefvoidconst_pointer_to_int(int*constvalue):# Declares value as const pointer to int type.# Value cannot be modified but the object pointed to by value can be modified.print(value[0])value[0]=10print(value[0])cdefvoidconst_pointer_to_const_int(constint*constvalue):# Declares value as const pointer to const int type.# Neither the value variable nor the int pointed to can be modified.print(value[0])
Similar to pointers Cython supports shortcut types that can be used in pure python mode. The following table shows several examples:
Cython | Full Annotation type | Shortcut type |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
For full list of shortcut types see theShadow.pyi
file.
Theconst
qualifier supports declaration of global constants:
cdefconstinti=5# constant pointers are defined as pointer to a constant value.cdefconstchar *msg="Dummy string"msg="Another dummy string"
Note
Theconst
modifier is unusablein a lot of contexts since Cython needs to generate definitions and their assignments separately. Thereforewe suggest using it mainly for function argument and pointer types whereconst
is necessary towork with an existing C/C++ interface.
Extension Types¶
It is also possible to declareExtension Types (declared withcdefclass
or the@cclass
decorator).Those will have a behaviour very close to python classes (e.g. creating subclasses),but access to their members is faster from Cython code. Typing a variableas extension type is mostly used to accesscdef
/@cfunc
methods and attributes of the extension type.The C code uses a variable which is a pointer to a structure of thespecific type, something likestructMyExtensionTypeObject*
.
Here is a simple example:
@cython.cclassclassShrubbery:width:cython.intheight:cython.intdef__init__(self,w,h):self.width=wself.height=hdefdescribe(self):print("This shrubbery is",self.width,"by",self.height,"cubits.")
cdefclassShrubbery:cdefintwidthcdefintheightdef__init__(self,w,h):self.width=wself.height=hdefdescribe(self):print("This shrubbery is",self.width,"by",self.height,"cubits.")
You can read more about them inExtension Types.
Grouping multiple C declarations¶
If you have a series of declarations that all begin withcdef
, youcan group them into acdef
block like this:
Note
This is supported only in Cython’scdef
syntax.
cdef:structSpam:inttonsintifloataSpam*pvoidf(Spam*s)except*:print(s.tons,"Tons of spam")
Python functions vs. C functions¶
There are two kinds of function definition in Cython:
Python functions are defined using thedef
statement, as in Python. They takePython objects as parameters and return Python objects.
C functions are defined using thecdef
statement in Cython syntax or with the@cfunc
decorator. They takeeither Python objects or C values as parameters, and can return either Pythonobjects or C values.
Within a Cython module, Python functions and C functions can call each otherfreely, but only Python functions can be called from outside the module byinterpreted Python code. So, any functions that you want to “export” from yourCython module must be declared as Python functions usingdef
.There is also a hybrid function, declared withcpdef
in.pyx
files or with the@ccall
decorator. These functionscan be called from anywhere, but use the faster C calling conventionwhen being called from other Cython code. They can also be overriddenby a Python method on a subclass or an instance attribute, even when called from Cython.If this happens, most performance gains are of course lost and even if it does not,there is a tiny overhead in calling such a method from Cython compared tocalling a C method.
Parameters of either type of function can be declared to have C data types,using normal C declaration syntax. For example,
defspam(i:cython.int,s:cython.p_char):...@cython.cfuncdefeggs(l:cython.ulong,f:cython.float)->cython.int:...
defspam(inti,char*s):...cdefinteggs(unsignedlongl,floatf):...
ctuples
may also be used
@cython.cfuncdefchips(t:tuple[cython.long,cython.long,cython.double])->tuple[cython.int,cython.float]:...
cdef (int,float)chips((long,long,double)t):...
When a parameter of a Python function is declared to have a C data type, it ispassed in as a Python object and automatically converted to a C value, ifpossible. In other words, the definition ofspam
above is equivalent towriting
defspam(python_i,python_s):i:cython.int=python_is:cython.p_char=python_s...
defspam(python_i,python_s):cdefinti=python_icdefchar*s=python_s...
Automatic conversion is currently only possible for numeric types,string types and structs (composed recursively of any of these types);attempting to use any other type for the parameter of aPython function will result in a compile-time error.Care must be taken with strings to ensure a reference if the pointer is to be usedafter the call. Structs can be obtained from Python mappings, and again care must be takenwith string attributes if they are to be used after the function returns.
C functions, on the other hand, can have parameters of any type, since they’repassed in directly using a normal C function call.
C Functions declared usingcdef
or the@cfunc
decorator with aPython object return type, like Python functions, will return aNone
value when execution leaves the function body without an explicit return value. This is incontrast to C/C++, which leaves the return value undefined.In the case of non-Python object return types, the equivalent of zero is returned, for example, 0 forint
,False
forbint
andNULL
for pointer types.
A more complete comparison of the pros and cons of these different methodtypes can be found atEarly Binding for Speed.
Python objects as parameters and return values¶
If no type is specified for a parameter or return value, it is assumed to be aPython object. (Note that this is different from the C convention, where itwould default toint
.) For example, the following defines a C function thattakes two Python objects as parameters and returns a Python object
@cython.cfuncdefspamobjs(x,y):...
cdefspamobjs(x,y):...
Reference counting for these objects is performed automatically according tothe standard Python/C API rules (i.e. borrowed references are taken asparameters and a new reference is returned).
Warning
This only applies to Cython code. Other Python packages whichare implemented in C like NumPy may not follow these conventions.
The type nameobject
can also be used to explicitly declare something as a Pythonobject. This can be useful if the name being declared would otherwise be takenas the name of a type, for example,
@cython.cfuncdefftang(int:object):...
cdefftang(objectint):...
declares a parameter calledint
which is a Python object. You can also useobject
as the explicit return type of a function, e.g.
@cython.cfuncdefftang(int:object)->object:...
cdefobjectftang(objectint):...
In the interests of clarity, it is probably a good idea to always be explicitabout object parameters in C functions.
To create a borrowed reference, specify the parameter type asPyObject*.Cython won’t perform automaticPy_INCREF()
, orPy_DECREF()
, e.g.:
# Py_REFCNT and _Py_REFCNT are the same, except _Py_REFCNT takes# a raw pointer and Py_REFCNT takes a normal Python objectfromcython.cimports.cpython.refimportPyObject,_Py_REFCNT,Py_REFCNTimportsyspython_dict={"abc":123}python_dict_refcount=Py_REFCNT(python_dict)@cython.cfuncdefowned_reference(obj:object):refcount1=Py_REFCNT(obj)print(f'Inside owned_reference initially: {refcount1}')another_ref_to_object=objrefcount2=Py_REFCNT(obj)print(f'Inside owned_reference after new ref: {refcount2}')@cython.cfuncdefborrowed_reference(obj:cython.pointer[PyObject]):refcount1=_Py_REFCNT(obj)print(f'Inside borrowed_reference initially: {refcount1}')another_ptr_to_object=objrefcount2=_Py_REFCNT(obj)print(f'Inside borrowed_reference after new pointer: {refcount2}')# Casting to a managed reference to call a cdef function doesn't increase the countrefcount3=Py_REFCNT(cython.cast(object,obj))print(f'Inside borrowed_reference with temporary managed reference: {refcount3}')# However calling a Python function may depending on the Python version and the number# of arguments.print(f'Initial refcount: {python_dict_refcount}')owned_reference(python_dict)borrowed_reference(cython.cast(cython.pointer[PyObject],python_dict))
# Py_REFCNT and _Py_REFCNT are the same, except _Py_REFCNT takes# a raw pointer and Py_REFCNT takes a normal Python objectfromcpython.refcimportPyObject,_Py_REFCNT,Py_REFCNTimportsyspython_dict={"abc":123}python_dict_refcount=Py_REFCNT(python_dict)cdefowned_reference(objectobj):refcount1=Py_REFCNT(obj)print(f'Inside owned_reference initially: {refcount1}')another_ref_to_object=objrefcount2=Py_REFCNT(obj)print(f'Inside owned_reference after new ref: {refcount2}')cdefborrowed_reference(PyObject*obj):refcount1=_Py_REFCNT(obj)print(f'Inside borrowed_reference initially: {refcount1}')another_ptr_to_object=objrefcount2=_Py_REFCNT(obj)print(f'Inside borrowed_reference after new pointer: {refcount2}')# Casting to a managed reference to call a cdef function doesn't increase the countrefcount3=Py_REFCNT(<object>obj)print(f'Inside borrowed_reference with temporary managed reference: {refcount3}')# However calling a Python function may depending on the Python version and the number# of arguments.print(f'Initial refcount: {python_dict_refcount}')owned_reference(python_dict)borrowed_reference(<PyObject*>python_dict)
will display:
Initialrefcount:2Insideowned_referenceinitially:2Insideowned_referenceafternewref:3Insideborrowed_referenceinitially:2Insideborrowed_referenceafternewpointer:2Insideborrowed_referencewithtemporarymanagedreference:2
Optional Arguments¶
Unlike C, it is possible to use optional arguments in C andcpdef
/@ccall
functions.There are differences though whether you declare them in a.pyx
/.py
file or the corresponding.pxd
file.
To avoid repetition (and potential future inconsistencies), default argument values arenot visible in the declaration (in.pxd
files) but only inthe implementation (in.pyx
files).
When in a.pyx
/.py
file, the signature is the same as it is in Python itself:
@cython.cclassclassA:@cython.cfuncdeffoo(self):print("A")@cython.cclassclassB(A):@cython.cfuncdeffoo(self,x=None):print("B",x)@cython.cclassclassC(B):@cython.ccalldeffoo(self,x=True,k:cython.int=3):print("C",x,k)
cdefclassA:cdeffoo(self):print("A")cdefclassB(A):cdeffoo(self,x=None):print("B",x)cdefclassC(B):cpdeffoo(self,x=True,intk=3):print("C",x,k)
When in a.pxd
file, the signature is different like this example:cdeffoo(x=*)
.This is because the program calling the function just needs to know what signatures arepossible in C, but doesn’t need to know the value of the default arguments.:
cdefclassA:cdeffoo(self)cdefclassB(A):cdeffoo(self,x=*)cdefclassC(B):cpdeffoo(self,x=*,intk=*)
Note
The number of arguments may increase when subclassing,but the arg types and order must be the same, as shown in the example above.
There may be a slight performance penalty when the optional arg is overriddenwith one that does not have default values.
Keyword-only Arguments¶
As in Python 3,def
functions can have keyword-only argumentslisted after a"*"
parameter and before a"**"
parameter if any:
deff(a,b,*args,c,d=42,e,**kwds):...# We cannot call f with less verbosity than this.foo=f(4,"bar",c=68,e=1.0)
As shown above, thec
,d
ande
arguments can not bepassed as positional arguments and must be passed as keyword arguments.Furthermore,c
ande
arerequired keyword argumentssince they do not have a default value.
A single"*"
without argument name can be used toterminate the list of positional arguments:
defg(a,b,*,c,d):...# We cannot call g with less verbosity than this.foo=g(4.0,"something",c=68,d="other")
Shown above, the signature takes exactly two positionalparameters and has two required keyword parameters.
Function Pointers¶
Note
Pointers to functions are currently not supported by pure Python mode. (GitHub issue#4279)
The following example shows declaring aptr_add
function pointer and assigning theadd
function to it:
cdefint(*ptr_add)(int,int)cdefintadd(inta,intb):returna+bptr_add=addprint(ptr_add(1,3))
Functions declared in astruct
are automatically converted to function pointers:
cdefstructBar:intsum(inta,intb)cdefintadd(inta,intb):returna+bcdefBarbar=Bar(add)print(bar.sum(1,2))
For using error return values with function pointers, see the note at the bottomofError return values.
Error return values¶
In Python (more specifically, in the CPython runtime), exceptions that occurinside of a function are signaled to the caller and propagated up the call stackthrough defined error return values. For functions that return a Python object(and thus, a pointer to such an object), the error return value is simply theNULL
pointer, so any function returning a Python object has a well-definederror return value.
While this is always the case for Python functions, functionsdefined as C functions orcpdef
/@ccall
functions can return arbitrary C types,which do not have such a well-defined error return value.By default Cython uses a dedicated return value to signal that an exception has been raised from non-externalcpdef
/@ccall
functions. However, how Cython handles exceptions from these functions can be changed if needed.
Acdef
function may be declared with an exception return value for itas a contract with the caller. Here is an example:
@cython.cfunc@cython.exceptval(-1)defspam()->cython.int:...
cdefintspam()except-1:...
With this declaration, whenever an exception occurs insidespam
, it willimmediately return with the value-1
. From the caller’s side, whenevera call to spam returns-1
, the caller will assume that an exception hasoccurred and can now process or propagate it. Callingspam()
is roughly translated to the following C code:
ret_val=spam();if(ret_val==-1)gotoerror_handler;
When you declare an exception value for a function, you should never explicitlyor implicitly return that value. This includes emptyreturn
statements, without a return value, for which Cython inserts the default returnvalue (e.g.0
for C number types). In general, exception return valuesare best chosen from invalid or very unlikely return values of the function,such as a negative value for functions that return only non-negative results,or a very large value likeINT_MAX
for a function that “usually” onlyreturns small results.
If all possible return values are legal and youcan’t reserve one entirely for signalling errors, you can use an alternativeform of exception value declaration
@cython.cfunc@cython.exceptval(-1,check=True)defspam()->cython.int:...
The keyword argumentcheck=True
indicates that the value-1
may signal an error.
cdefintspam()except?-1:...
The?
indicates that the value-1
may signal an error.
In this case, Cython generates a call toPyErr_Occurred()
if the exception valueis returned, to make sure it really received an exception and not just a normalresult. Callingspam()
is roughly translated to the following C code:
ret_val=spam();if(ret_val==-1&&PyErr_Occurred())gotoerror_handler;
There is also a third form of exception value declaration
@cython.cfunc@cython.exceptval(check=True)defspam()->cython.void:...
cdefvoidspam()except*:...
This form causes Cython to generate a call toPyErr_Occurred()
afterevery call to spam, regardless of what value it returns. Callingspam()
is roughly translated to the following C code:
spam()if(PyErr_Occurred())gotoerror_handler;
If you have afunction returningvoid
that needs to propagate errors, you will have touse this form, since there isn’t any error return value to test.Otherwise, an explicit error return value allows the C compiler to generatemore efficient code and is thus generally preferable.
An external C++ function that may raise an exception can be declared with:
cdefintspam()except+
Note
These declarations are not used in Python code, only in.pxd
and.pyx
files.
SeeUsing C++ in Cython for more details.
Finally, if you are certain that your function should not raise an exception, (e.g., itdoes not use Python objects at all, or you plan to use it as a callback in C code thatis unaware of Python exceptions), you can declare it as such usingnoexcept
or by@cython.exceptval(check=False)
:
@cython.cfunc@cython.exceptval(check=False)defspam()->cython.int:...
cdefintspam()noexcept:...
If anoexcept
functiondoes finish with an exception then it will print a warning message but not allow the exception to propagate further.On the other hand, calling anoexcept
function has zero overhead related to managing exceptions, unlike the previous declarations.
Some things to note:
cdef
functions that are alsoextern
are implicitly declarednoexcept
or@cython.exceptval(check=False)
.In the uncommon case of external C/C++ functions thatcan raise Python exceptions,e.g., external functions that use the Python C API, you should explicitly declarethem with an exception value.cdef
functions that arenotextern
are implicitly declared with a suitableexception specification for the return type (e.g.except*
or@cython.exceptval(check=True)
for avoid
returntype,except?-1
or@cython.exceptval(-1,check=True)
for anint
return type).Exception values can only be declared for functions returning a C integer,enum, float or pointer type, and the value must be a constant expression.Functions that return
void
, or a struct/union by value, can only usetheexcept*
orexceptval(check=True)
form.The exception value specification is part of the signature of the function.If you’re passing a pointer to a function as a parameter or assigning itto a variable, the declared type of the parameter or variable must havethe same exception value specification (or lack thereof). Here is anexample of a pointer-to-function declaration with an exception value:
int(*grail)(int,char*)except-1
Note
Pointers to functions are currently not supported by pure Python mode. (GitHub issue#4279)
If the returning type of a
cdef
function withexcept*
or@cython.exceptval(check=True)
is C integer,enum, float or pointer type, Cython callsPyErr_Occurred()
only whendedicated value is returned instead of checking after every call of the function.You don’t need to (and shouldn’t) declare exception values for functionswhich return Python objects. Remember that a function with no declaredreturn type implicitly returns a Python object. (Exceptions on suchfunctions are implicitly propagated by returning
NULL
.)There’s a known performance pitfall when combining
nogil
andexcept*
@cython.exceptval(check=True)
.In this case Cython must always briefly re-acquire the GIL after a functioncall to check if an exception has been raised. This can commonly happen with afunction returning nothing (Cvoid
). Simple workarounds are to mark thefunction asnoexcept
if you’re certain that exceptions cannot be thrown, orto change the return type toint
and just let Cython use the return valueas an error flag (by default,-1
triggers the exception check).
Checking return values of non-Cython functions¶
It’s important to understand that the except clause does not cause an error tobe raised when the specified value is returned. For example, you can’t writesomething like:
cdefexternFILE *fopen(char*filename,char*mode)exceptNULL# WRONG!
and expect an exception to be automatically raised if a call tofopen()
returnsNULL
. The except clause doesn’t work that way; its only purpose isfor propagating Python exceptions that have already been raised, either by a Cythonfunction or a C function that calls Python/C API routines. To get an exceptionfrom a non-Python-aware function such asfopen()
, you will have to check thereturn value and raise it yourself, for example:
fromcython.cimports.libc.stdioimportFILE,fopenfromcython.cimports.libc.stdlibimportmalloc,freefromcython.cimports.cpython.excimportPyErr_SetFromErrnoWithFilenameObjectdefopen_file():p=fopen("spam.txt","r")# The type of "p" is "FILE*", as returned by fopen().ifpiscython.NULL:PyErr_SetFromErrnoWithFilenameObject(OSError,"spam.txt")...defallocating_memory(number=10):# Note that the type of the variable "my_array" is automatically inferred from the assignment.my_array=cython.cast(cython.p_double,malloc(number*cython.sizeof(double)))ifnotmy_array:# same as 'is NULL' aboveraiseMemoryError()...free(my_array)
fromlibc.stdiocimportFILE,fopenfromlibc.stdlibcimportmalloc,freefromcpython.exccimportPyErr_SetFromErrnoWithFilenameObjectdefopen_file():cdefFILE*pp=fopen("spam.txt","r")ifpisNULL:PyErr_SetFromErrnoWithFilenameObject(OSError,"spam.txt")...defallocating_memory(number=10):cdefdouble *my_array=<double*>malloc(number*sizeof(double))ifnotmy_array:# same as 'is NULL' aboveraiseMemoryError()...free(my_array)
Overriding in extension types¶
cpdef
/@ccall
methods can override C methods:
@cython.cclassclassA:@cython.cfuncdeffoo(self):print("A")@cython.cclassclassB(A):@cython.cfuncdeffoo(self,x=None):print("B",x)@cython.cclassclassC(B):@cython.ccalldeffoo(self,x=True,k:cython.int=3):print("C",x,k)
cdefclassA:cdeffoo(self):print("A")cdefclassB(A):cdeffoo(self,x=None):print("B",x)cdefclassC(B):cpdeffoo(self,x=True,intk=3):print("C",x,k)
When subclassing an extension type with a Python class,Python methods can overridecpdef
/@ccall
methods but not plain C methods:
@cython.cclassclassA:@cython.cfuncdeffoo(self):print("A")@cython.cclassclassB(A):@cython.ccalldeffoo(self):print("B")classC(B):# NOTE: no cclass decoratordeffoo(self):print("C")
cdefclassA:cdeffoo(self):print("A")cdefclassB(A):cpdeffoo(self):print("B")classC(B):# NOTE: not cdef classdeffoo(self):print("C")
IfC
above would be an extension type (cdefclass
),this would not work correctly.The Cython compiler will give a warning in that case.
Automatic type conversions¶
In most situations, automatic conversions will be performed for the basicnumeric and string types when a Python object is used in a context requiring aC value, or vice versa. The following table summarises the conversionpossibilities.
C types | From Python types | To Python types |
---|---|---|
[unsigned] char,[unsigned] short,int, long | int, long | int |
unsigned int,unsigned long,[unsigned] long long | int, long | long |
float, double, long double | int, long, float | float |
char* | str/bytes | str/bytes[3] |
C array | iterable | list[6] |
struct,union |
The conversion is to/from str for Python 2.x, and bytes for Python 3.x.
[5]The conversion from a C union type to a Python dict will adda value for each of the union fields. Cython 0.23 and later, however,will refuse to automatically convert a union with unsafe typecombinations. An example is a union of anint
and achar*
,in which case the pointer value may or may not be a valid pointer.
Other than signed/unsigned char[].The conversion will fail if the length of C array is not known at compile time,and when using a slice of a C array.
[7]The automatic conversion of a struct to adict
(and viceversa) does have some potential pitfalls detailedelsewhere in the documentation.
Caveats when using a Python string in a C context¶
You need to be careful when using a Python string in a context expecting achar*
. In this situation, a pointer to the contents of the Python string isused, which is only valid as long as the Python string exists. So you need tomake sure that a reference to the original Python string is held for as longas the C string is needed. If you can’t guarantee that the Python string willlive long enough, you will need to copy the C string.
Cython detects and prevents some mistakes of this kind. For instance, if youattempt something like
defmain():s:cython.p_chars=pystring1+pystring2
cdefchar *ss=pystring1+pystring2
then Cython will produce the error messageStoringunsafeCderivativeoftemporaryPythonreference
. The reason is that concatenating the two Python stringsproduces a new Python string object that is referenced only by a temporaryinternal variable that Cython generates. As soon as the statement has finished,the temporary variable will be decrefed and the Python string deallocated,leavings
dangling. Since this code could not possibly work, Cython refuses tocompile it.
The solution is to assign the result of the concatenation to a Pythonvariable, and then obtain thechar*
from that, i.e.
defmain():s:cython.p_charp=pystring1+pystring2s=p
cdefchar *sp=pystring1+pystring2s=p
It is then your responsibility to hold the reference p for as long asnecessary.
Keep in mind that the rules used to detect such errors are only heuristics.Sometimes Cython will complain unnecessarily, and sometimes it will fail todetect a problem that exists. Ultimately, you need to understand the issue andbe careful what you do.
Type Casting¶
The Cython language supports type casting in a similar way as C. Where C uses"("
and")"
,Cython uses"<"
and">"
. In pure python mode, thecython.cast()
function is used. For example:
defmain():p:cython.p_charq:cython.p_floatp=cython.cast(cython.p_char,q)
When casting a C value to a Python object type or vice versa,Cython will attempt a coercion. Simple examples are casts likecast(int,pyobj_value)
,which convert a Python number to a plain Cint
value, or the statementcast(bytes,charptr_value)
,which copies a Cchar*
string into a new Python bytes object.
Note
Cython will not prevent a redundant cast, but emits a warning for it.
To get the address of some Python object, use a cast to a pointer typelikecast(p_void,...)
orcast(pointer[PyObject],...)
.You can also cast a C pointer back to a Python object referencewithcast(object,...)
, or to a more specific builtin or extension type(e.g.cast(MyExtType,ptr)
). This will increase the reference count ofthe object by one, i.e. the cast returns an owned reference.Here is an example:
cdefchar *pcdeffloat *qp=<char*>q
When casting a C value to a Python object type or vice versa,Cython will attempt a coercion. Simple examples are casts like<int>pyobj_value
,which convert a Python number to a plain Cint
value, or the statement<bytes>charptr_value
,which copies a Cchar*
string into a new Python bytes object.
Note
Cython will not prevent a redundant cast, but emits a warning for it.
To get the address of some Python object, use a cast to a pointer typelike<void*>
or<PyObject*>
.You can also cast a C pointer back to a Python object referencewith<object>
, or to a more specific builtin or extension type(e.g.<MyExtType>ptr
). This will increase the reference count ofthe object by one, i.e. the cast returns an owned reference.Here is an example:
cdefexternfrom*:ctypedefPy_ssize_tPy_intptr_t
fromcython.cimports.cpython.refimportPyObjectdefmain():python_string="foo"# Note that the variables below are automatically inferred# as the correct pointer type that is assigned to them.# They do not need to be typed explicitly.ptr=cython.cast(cython.p_void,python_string)adress_in_c=cython.cast(Py_intptr_t,ptr)address_from_void=adress_in_c# address_from_void is a python intptr2=cython.cast(cython.pointer[PyObject],python_string)address_in_c2=cython.cast(Py_intptr_t,ptr2)address_from_PyObject=address_in_c2# address_from_PyObject is a python intassertaddress_from_void==address_from_PyObject==id(python_string)print(cython.cast(object,ptr))# Prints "foo"print(cython.cast(object,ptr2))# prints "foo"
Casting withcast(object,...)
creates an owned reference. Cython will automaticallyperform aPy_INCREF()
andPy_DECREF()
operation. Casting tocast(pointer[PyObject],...)
creates a borrowed reference, leaving the refcount unchanged.
cdefexternfrom*:ctypedefPy_ssize_tPy_intptr_t
fromcpython.refcimportPyObjectpython_string="foo"cdefvoid*ptr=<void*>python_stringcdefPy_intptr_tadress_in_c=<Py_intptr_t>ptraddress_from_void=adress_in_c# address_from_void is a python intcdefPyObject*ptr2=<PyObject*>python_stringcdefPy_intptr_taddress_in_c2=<Py_intptr_t>ptr2address_from_PyObject=address_in_c2# address_from_PyObject is a python intassertaddress_from_void==address_from_PyObject==id(python_string)print(<object>ptr)# Prints "foo"print(<object>ptr2)# prints "foo"
The precedence of<...>
is such that<type>a.b.c
is interpreted as<type>(a.b.c)
.
Casting to<object>
creates an owned reference. Cython will automaticallyperform aPy_INCREF()
andPy_DECREF()
operation. Casting to<PyObject*>
creates a borrowed reference, leaving the refcount unchanged.
Checked Type Casts¶
A cast like<MyExtensionType>x
orcast(MyExtensionType,x)
will castx
to the classMyExtensionType
without any checking at all.
To have a cast checked, use<MyExtensionType?>x
in Cython syntaxorcast(MyExtensionType,x,typecheck=True)
.In this case, Cython will apply a runtime check that raises aTypeError
ifx
is not an instance ofMyExtensionType
.This tests for the exact class for builtin types,but allows subclasses forExtension Types.
Statements and expressions¶
Control structures and expressions follow Python syntax for the most part.When applied to Python objects, they have the same semantics as in Python(unless otherwise noted). Most of the Python operators can also be applied toC values, with the obvious semantics.
If Python objects and C values are mixed in an expression, conversions areperformed automatically between Python objects and C numeric or string types.
Reference counts are maintained automatically for all Python objects, and allPython operations are automatically checked for errors, with appropriateaction taken.
Differences between C and Cython expressions¶
There are some differences in syntax and semantics between C expressions andCython expressions, particularly in the area of C constructs which have nodirect equivalent in Python.
An integer literal is treated as a C constant, and willbe truncated to whatever size your C compiler thinks appropriate.To get a Python integer (of arbitrary precision), cast immediately toan object (e.g.
<object>100000000000000000000
orcast(object,100000000000000000000)
). TheL
,LL
,andU
suffixes have the same meaning in Cython syntax as in C.There is no
->
operator in Cython. Instead ofp->x
, usep.x
There is no unary
*
operator in Cython. Instead of*p
, usep[0]
There is an
&
operator in Cython, with the same semantics as in C.In pure python mode, use thecython.address()
function instead.The null C pointer is called
NULL
, not0
.NULL
is a reserved word in Cythonandcython.NULL
is a special object in pure python mode.Type casts are written
<type>value
orcast(type,value)
, for example,defmain():p:cython.p_charq:cython.p_floatp=cython.cast(cython.p_char,q)
cdefchar*pcdeffloat*qp=<char*>q
Scope rules¶
Cython determines whether a variable belongs to a local scope, the modulescope, or the built-in scope completely statically. As with Python, assigningto a variable which is not otherwise declared implicitly declares it to be avariable residing in the scope where it is assigned. The type of the variabledepends on type inference, except for the global module scope, where it isalways a Python object.
Built-in Functions¶
Cython compiles calls to most built-in functions into direct calls tothe corresponding Python/C API routines, making them particularly fast.
Only direct function calls using these names are optimised. If you dosomething else with one of these names that assumes it’s a Python object,such as assign it to a Python variable, and later call it, the call willbe made as a Python function call.
Function and arguments | Return type | Python/C API Equivalent |
---|---|---|
abs(obj) | object,double, … | PyNumber_Absolute, fabs,fabsf, … |
callable(obj) | bint | PyObject_Callable |
delattr(obj, name) | None | PyObject_DelAttr |
exec(code, [glob, [loc]]) | object | |
dir(obj) | list | PyObject_Dir |
divmod(a, b) | tuple | PyNumber_Divmod |
getattr(obj, name, [default])(Note 1) | object | PyObject_GetAttr |
hasattr(obj, name) | bint | PyObject_HasAttr |
hash(obj) | int / long | PyObject_Hash |
intern(obj) | object | Py*_InternFromString |
isinstance(obj, type) | bint | PyObject_IsInstance |
issubclass(obj, type) | bint | PyObject_IsSubclass |
iter(obj, [sentinel]) | object | PyObject_GetIter |
len(obj) | Py_ssize_t | PyObject_Length |
pow(x, y, [z]) | object | PyNumber_Power |
reload(obj) | object | PyImport_ReloadModule |
repr(obj) | object | PyObject_Repr |
setattr(obj, name) | void | PyObject_SetAttr |
Note 1: Pyrex originally provided a functiongetattr3(obj,name,default)()
corresponding to the three-argument form of the Python builtingetattr()
.Cython still supports this function, but the usage is deprecated in favour ofthe normal builtin, which Cython can optimise in both forms.
Operator Precedence¶
Keep in mind that there are some differences in operator precedence betweenPython and C, and that Cython uses the Python precedences, not the C ones.
Integer for-loops¶
Note
This syntax is supported only in Cython files. Use a normalfor-in-range() loop instead.
Cython recognises the usual Python for-in-range integer loop pattern:
foriinrange(n):...
Ifi
is declared as acdef
integer type, it willoptimise this into a pure C loop. This restriction is required asotherwise the generated code wouldn’t be correct due to potentialinteger overflows on the target architecture. If you are worried thatthe loop is not being converted correctly, use the annotate feature ofthe cython commandline (-a
) to easily see the generated C code.SeeAutomatic range conversion
For backwards compatibility to Pyrex, Cython also supports a more verboseform of for-loop which you might find in legacy code:
forifrom0<=i<n:...
or:
forifrom0<=i<nbys:...
wheres
is some integer step size.
Note
This syntax is deprecated and should not be used in new code.Use the normal Python for-loop instead.
Some things to note about the for-from loop:
The target expression must be a plain variable name.
The name between the lower and upper bounds must be the same as the targetname.
The direction of iteration is determined by the relations. If they are bothfrom the set {
<
,<=
} then it is upwards; if they are both from the set{>
,>=
} then it is downwards. (Any other combination is disallowed.)
Like other Python looping statements, break and continue may be used in thebody, and the loop may have an else clause.
Cython file types¶
There are three file types in Cython:
The implementation files, carrying a
.py
or.pyx
suffix.The definition files, carrying a
.pxd
suffix.The include files, carrying a
.pxi
suffix.
The implementation file¶
The implementation file, as the name suggest, contains the implementationof your functions, classes, extension types, etc. Nearly all thepython syntax is supported in this file. Most of the time, a.py
file can be renamed into a.pyx
file without changingany code, and Cython will retain the python behavior.
It is possible for Cython to compile both.py
and.pyx
files.The name of the file isn’t important if one wants to use only the Python syntax,and Cython won’t change the generated code depending on the suffix used.Though, if one want to use the Cython syntax, using a.pyx
file is necessary.
In addition to the Python syntax, the user can alsoleverage Cython syntax (such ascdef
) to use C variables, candeclare functions ascdef
orcpdef
and can import C definitionswithcimport
. Many other Cython features usable in implementation filescan be found throughout this page and the rest of the Cython documentation.
There are some restrictions on the implementation part of someExtension Typesif the corresponding definition file also defines that type.
Note
When a.pyx
file is compiled, Cython first checks to see if a corresponding.pxd
file exists and processes it first. It acts like a header file fora Cython.pyx
file. You can put inside functions that will be used byother Cython modules. This allows different Cython modules to use functionsand classes from each other without the Python overhead. To read more aboutwhat how to do that, you can seepxd files.
The definition file¶
A definition file is used to declare various things.
Any C declaration can be made, and it can be also a declaration of a C variable orfunction implemented in a C/C++ file. This can be done withcdefexternfrom
.Sometimes,.pxd
files are used as a translation of C/C++ header filesinto a syntax that Cython can understand. This allows then the C/C++ variable andfunctions to be used directly in implementation files withcimport
.You can read more about it inInterfacing with External C Code andUsing C++ in Cython.
It can also contain the definition part of an extension type and the declarationsof functions for an external library.
It cannot contain the implementations of any C or Python functions, or anyPython class definitions, or any executable statements. It is needed when onewants to accesscdef
attributes and methods, or to inherit fromcdef
classes defined in this module.
Note
You don’t need to (and shouldn’t) declare anything in a declaration filepublic
in order to make it available to other Cython modules; its merepresence in a definition file does that. You only need a publicdeclaration if you want to make something available to external C code.
The include statement and include files¶
Warning
Historically theinclude
statement was used for sharing declarations.UseSharing Declarations Between Cython Modules instead.
A Cython source file can include material from other files using the includestatement, for example,:
include"spamstuff.pxi"
The contents of the named file are textually included at that point. Theincluded file can contain any complete statements or declarations that arevalid in the context where the include statement appears, including otherinclude statements. The contents of the included file should begin at anindentation level of zero, and will be treated as though they were indented tothe level of the include statement that is including the file. The includestatement cannot, however, be used outside of the module scope, such as insideof functions or class bodies.
Note
There are other mechanisms available for splitting Cython code intoseparate parts that may be more appropriate in many cases. SeeSharing Declarations Between Cython Modules.
Conditional Compilation¶
Some language features are available for conditional compilation and compile-timeconstants within a Cython source file.
Note
This feature has been deprecated and should not be used in new code.It is very foreign to the Python language and also behavesdifferently from the C preprocessor. It is often misunderstood by users.For the current deprecation status, seehttps://github.com/cython/cython/issues/4310.For alternatives, seeDeprecation of DEF / IF.
Note
This feature has very little use cases. Specifically, it is not a goodway to adapt code to platform and environment. Use runtime conditions,conditional Python imports, or C compile time adaptation for this.See, for example,Including verbatim C code orResolving naming conflicts - C name specifications.
Note
Cython currently does not support conditional compilation and compile-timedefinitions in Pure Python mode. As it stands, this is unlikely to change.
Compile-Time Definitions¶
A compile-time constant can be defined using the DEF statement:
DEFFavouriteFood=u"spam"DEFArraySize=42DEFOtherArraySize=2*ArraySize+17
The right-hand side of theDEF
must be a valid compile-time expression.Such expressions are made up of literal values and names defined usingDEF
statements, combined using any of the Python expression syntax.
Note
Cython does not intend to copy literal compile-time values 1:1 into the generated code.Instead, these values are internally represented and calculated as plain Pythonvalues and use Python’srepr()
when a serialisation is needed. This meansthat values defined usingDEF
may lose precision or change their typedepending on the calculation rules of the Python environment where Cython parses andtranslates the source code. Specifically, usingDEF
to define high-precisionfloating point constants may not give the intended result and may generate differentC values in different Python versions.
The following compile-time names are predefined, corresponding to the valuesreturned byos.uname()
. As noted above, they are not considered good waysto adapt code to different platforms and are mostly provided for legacy reasons.
UNAME_SYSNAME, UNAME_NODENAME, UNAME_RELEASE,UNAME_VERSION, UNAME_MACHINE
The following selection of builtin constants and functions are also available:
None, True, False,abs, all, any, ascii, bin, bool, bytearray, bytes, chr, cmp, complex, dict,divmod, enumerate, filter, float, format, frozenset, hash, hex, int, len,list, long, map, max, min, oct, ord, pow, range, reduce, repr, reversed,round, set, slice, sorted, str, sum, tuple, xrange, zip
Note that some of these builtins may not be available when compiling underPython 2.x or 3.x, or may behave differently in both.
A name defined usingDEF
can be used anywhere an identifier can appear,and it is replaced with its compile-time value as though it were written intothe source at that point as a literal. For this to work, the compile-timeexpression must evaluate to a Python value of typeint
,long
,float
,bytes
orunicode
(str
in Py3).
DEFFavouriteFood=u"spam"DEFArraySize=42DEFOtherArraySize=2*ArraySize+17cdefint[ArraySize]a1cdefint[OtherArraySize]a2print("I like",FavouriteFood)
Note
Compile-time constants are deprecated. The preferred way for declaring globalconstants is using globalconst
variables. SeeType qualifiers.
Conditional Statements¶
TheIF
statement can be used to conditionally include or exclude sectionsof code at compile time. It works in a similar way to the#if
preprocessordirective in C.
IFARRAY_SIZE>64:include"large_arrays.pxi"ELIFARRAY_SIZE>16:include"medium_arrays.pxi"ELSE:include"small_arrays.pxi"
TheELIF
andELSE
clauses are optional. AnIF
statement can appearanywhere that a normal statement or declaration can appear, and it can containany statements or declarations that would be valid in that context, includingDEF
statements and otherIF
statements.
The expressions in theIF
andELIF
clauses must be valid compile-timeexpressions as for theDEF
statement, although they can evaluate to anyPython value, and the truth of the result is determined in the usual Pythonway.