pandas docstring guide#

About docstrings and standards#

A Python docstring is a string used to document a Python module, class,function or method, so programmers can understand what it does without havingto read the details of the implementation.

Also, it is a common practice to generate online (html) documentationautomatically from docstrings.Sphinx servesthis purpose.

The next example gives an idea of what a docstring looks like:

defadd(num1,num2):"""    Add up two integer numbers.    This function simply wraps the ``+`` operator, and does not    do anything interesting, except for illustrating what    the docstring of a very simple function looks like.    Parameters    ----------    num1 : int        First number to add.    num2 : int        Second number to add.    Returns    -------    int        The sum of ``num1`` and ``num2``.    See Also    --------    subtract : Subtract one integer from another.    Examples    --------    >>> add(2, 2)    4    >>> add(25, 0)    25    >>> add(10, -10)    0    """returnnum1+num2

Some standards regarding docstrings exist, which make them easier to read, and allow thembe easily exported to other formats such as html or pdf.

The first conventions every Python docstring should follow are defined inPEP-257.

As PEP-257 is quite broad, other more specific standards also exist. In thecase of pandas, the NumPy docstring convention is followed. These conventions areexplained in this document:

numpydoc is a Sphinx extension to support the NumPy docstring convention.

The standard uses reStructuredText (reST). reStructuredText is a markuplanguage that allows encoding styles in plain text files. Documentationabout reStructuredText can be found in:

pandas has some helpers for sharing docstrings between related classes, seeSharing docstrings.

The rest of this document will summarize all the above guidelines, and willprovide additional conventions specific to the pandas project.

Writing a docstring#

General rules#

Docstrings must be defined with three double-quotes. No blank lines should beleft before or after the docstring. The text starts in the next line after theopening quotes. The closing quotes have their own line(meaning that they are not at the end of the last sentence).

On rare occasions reST styles like bold text or italics will be used indocstrings, but is it common to have inline code, which is presented betweenbackticks. The following are considered inline code:

  • The name of a parameter

  • Python code, a module, function, built-in, type, literal… (e.g.os,list,numpy.abs,datetime.date,True)

  • A pandas class (in the form:class:`pandas.Series`)

  • A pandas method (in the form:meth:`pandas.Series.sum`)

  • A pandas function (in the form:func:`pandas.to_datetime`)

Note

To display only the last component of the linked class, method orfunction, prefix it with~. For example,:class:`~pandas.Series`will link topandas.Series but only display the last part,Seriesas the link text. SeeSphinx cross-referencing syntaxfor details.

Good:

defadd_values(arr):"""    Add the values in ``arr``.    This is equivalent to Python ``sum`` of :meth:`pandas.Series.sum`.    Some sections are omitted here for simplicity.    """returnsum(arr)

Bad:

deffunc():"""Some function.    With several mistakes in the docstring.    It has a blank line after the signature ``def func():``.    The text 'Some function' should go in the line after the    opening quotes of the docstring, not in the same line.    There is a blank line between the docstring and the first line    of code ``foo = 1``.    The closing quotes should be in the next line, not in this one."""foo=1bar=2returnfoo+bar

Section 1: short summary#

The short summary is a single sentence that expresses what the function does ina concise way.

The short summary must start with a capital letter, end with a dot, and fit ina single line. It needs to express what the object does without providingdetails. For functions and methods, the short summary must start with aninfinitive verb.

Good:

defastype(dtype):"""    Cast Series type.    This section will provide further details.    """pass

Bad:

defastype(dtype):"""    Casts Series type.    Verb in third-person of the present simple, should be infinitive.    """pass
defastype(dtype):"""    Method to cast Series type.    Does not start with verb.    """pass
defastype(dtype):"""    Cast Series type    Missing dot at the end.    """pass
defastype(dtype):"""    Cast Series type from its current type to the new type defined in    the parameter dtype.    Summary is too verbose and doesn't fit in a single line.    """pass

Section 2: extended summary#

The extended summary provides details on what the function does. It should notgo into the details of the parameters, or discuss implementation notes, whichgo in other sections.

A blank line is left between the short summary and the extended summary.Every paragraph in the extended summary ends with a dot.

The extended summary should provide details on why the function is useful andtheir use cases, if it is not too generic.

defunstack():"""    Pivot a row index to columns.    When using a MultiIndex, a level can be pivoted so each value in    the index becomes a column. This is especially useful when a subindex    is repeated for the main index, and data is easier to visualize as a    pivot table.    The index level will be automatically removed from the index when added    as columns.    """pass

Section 3: parameters#

The details of the parameters will be added in this section. This section hasthe title “Parameters”, followed by a line with a hyphen under each letter ofthe word “Parameters”. A blank line is left before the section title, but notafter, and not between the line with the word “Parameters” and the one withthe hyphens.

After the title, each parameter in the signature must be documented, including*args and**kwargs, but notself.

The parameters are defined by their name, followed by a space, a colon, anotherspace, and the type (or types). Note that the space between the name and thecolon is important. Types are not defined for*args and**kwargs, but mustbe defined for all other parameters. After the parameter definition, it isrequired to have a line with the parameter description, which is indented, andcan have multiple lines. The description must start with a capital letter, andfinish with a dot.

For keyword arguments with a default value, the default will be listed after acomma at the end of the type. The exact form of the type in this case will be“int, default 0”. In some cases it may be useful to explain what the defaultargument means, which can be added after a comma “int, default -1, meaning allcpus”.

In cases where the default value isNone, meaning that the value will not beused. Instead of"str,defaultNone", it is preferred to write"str,optional".WhenNone is a value being used, we will keep the form “str, default None”.For example, indf.to_csv(compression=None),None is not a value being used,but means that compression is optional, and no compression is being used if notprovided. In this case we will use"str,optional". Only in cases likefunc(value=None) andNone is being used in the same way as0 orfoowould be used, then we will specify “str, int or None, default None”.

Good:

classSeries:defplot(self,kind,color='blue',**kwargs):"""        Generate a plot.        Render the data in the Series as a matplotlib plot of the        specified kind.        Parameters        ----------        kind : str            Kind of matplotlib plot.        color : str, default 'blue'            Color name or rgb code.        **kwargs            These parameters will be passed to the matplotlib plotting            function.        """pass

Bad:

classSeries:defplot(self,kind,**kwargs):"""        Generate a plot.        Render the data in the Series as a matplotlib plot of the        specified kind.        Note the blank line between the parameters title and the first        parameter. Also, note that after the name of the parameter ``kind``        and before the colon, a space is missing.        Also, note that the parameter descriptions do not start with a        capital letter, and do not finish with a dot.        Finally, the ``**kwargs`` parameter is missing.        Parameters        ----------        kind: str            kind of matplotlib plot        """pass

Parameter types#

When specifying the parameter types, Python built-in data types can be useddirectly (the Python type is preferred to the more verbose string, integer,boolean, etc):

  • int

  • float

  • str

  • bool

For complex types, define the subtypes. Fordict andtuple, as more thanone type is present, we use the brackets to help read the type (curly bracketsfordict and normal brackets fortuple):

  • list of int

  • dict of {str : int}

  • tuple of (str, int, int)

  • tuple of (str,)

  • set of str

In case where there are just a set of values allowed, list them in curlybrackets and separated by commas (followed by a space). If the values areordinal and they have an order, list them in this order. Otherwise, listthe default value first, if there is one:

  • {0, 10, 25}

  • {‘simple’, ‘advanced’}

  • {‘low’, ‘medium’, ‘high’}

  • {‘cat’, ‘dog’, ‘bird’}

If the type is defined in a Python module, the module must be specified:

  • datetime.date

  • datetime.datetime

  • decimal.Decimal

If the type is in a package, the module must be also specified:

  • numpy.ndarray

  • scipy.sparse.coo_matrix

If the type is a pandas type, also specify pandas except for Series andDataFrame:

  • Series

  • DataFrame

  • pandas.Index

  • pandas.Categorical

  • pandas.arrays.SparseArray

If the exact type is not relevant, but must be compatible with a NumPyarray, array-like can be specified. If Any type that can be iterated isaccepted, iterable can be used:

  • array-like

  • iterable

If more than one type is accepted, separate them by commas, except thelast two types, that need to be separated by the word ‘or’:

  • int or float

  • float, decimal.Decimal or None

  • str or list of str

IfNone is one of the accepted values, it always needs to be the last inthe list.

For axis, the convention is to use something like:

  • axis : {0 or ‘index’, 1 or ‘columns’, None}, default None

Section 4: returns or yields#

If the method returns a value, it will be documented in this section. Alsoif the method yields its output.

The title of the section will be defined in the same way as the “Parameters”.With the names “Returns” or “Yields” followed by a line with as many hyphensas the letters in the preceding word.

The documentation of the return is also similar to the parameters. But in thiscase, no name will be provided, unless the method returns or yields more thanone value (a tuple of values).

The types for “Returns” and “Yields” are the same as the ones for the“Parameters”. Also, the description must finish with a dot.

For example, with a single value:

defsample():"""    Generate and return a random number.    The value is sampled from a continuous uniform distribution between    0 and 1.    Returns    -------    float        Random number generated.    """returnnp.random.random()

With more than one value:

importstringdefrandom_letters():"""    Generate and return a sequence of random letters.    The length of the returned string is also random, and is also    returned.    Returns    -------    length : int        Length of the returned string.    letters : str        String of random letters.    """length=np.random.randint(1,10)letters=''.join(np.random.choice(string.ascii_lowercase)foriinrange(length))returnlength,letters

If the method yields its value:

defsample_values():"""    Generate an infinite sequence of random numbers.    The values are sampled from a continuous uniform distribution between    0 and 1.    Yields    ------    float        Random number generated.    """whileTrue:yieldnp.random.random()

Section 5: see also#

This section is used to let users know about pandas functionalityrelated to the one being documented. In rare cases, if no related methodsor functions can be found at all, this section can be skipped.

An obvious example would be thehead() andtail() methods. Astail() doesthe equivalent ashead() but at the end of theSeries orDataFrameinstead of at the beginning, it is good to let the users know about it.

To give an intuition on what can be considered related, here there are someexamples:

  • loc andiloc, as they do the same, but in one case providing indicesand in the other positions

  • max andmin, as they do the opposite

  • iterrows,itertuples anditems, as it is easy that a userlooking for the method to iterate over columns ends up in the method toiterate over rows, and vice-versa

  • fillna anddropna, as both methods are used to handle missing values

  • read_csv andto_csv, as they are complementary

  • merge andjoin, as one is a generalization of the other

  • astype andpandas.to_datetime, as users may be reading thedocumentation ofastype to know how to cast as a date, and the way to doit is withpandas.to_datetime

  • where is related tonumpy.where, as its functionality is based on it

When deciding what is related, you should mainly use your common sense andthink about what can be useful for the users reading the documentation,especially the less experienced ones.

When relating to other libraries (mainlynumpy), use the name of the modulefirst (not an alias likenp). If the function is in a module which is notthe main one, likescipy.sparse, list the full module (e.g.scipy.sparse.coo_matrix).

This section has a header, “See Also” (note the capitalS and A), followed by the line with hyphens and preceded by a blank line.

After the header, we will add a line for each related method or function,followed by a space, a colon, another space, and a short description thatillustrates what this method or function does, why is it relevant in thiscontext, and what the key differences are between the documented function andthe one being referenced. The description must also end with a dot.

Note that in “Returns” and “Yields”, the description is located on the lineafter the type. In this section, however, it is located on the sameline, with a colon in between. If the description does not fit on the sameline, it can continue onto other lines which must be further indented.

For example:

classSeries:defhead(self):"""        Return the first 5 elements of the Series.        This function is mainly useful to preview the values of the        Series without displaying the whole of it.        Returns        -------        Series            Subset of the original series with the 5 first values.        See Also        --------        Series.tail : Return the last 5 elements of the Series.        Series.iloc : Return a slice of the elements in the Series,            which can also be used to return the first or last n.        """returnself.iloc[:5]

Section 6: notes#

This is an optional section used for notes about the implementation of thealgorithm, or to document technical aspects of the function behavior.

Feel free to skip it, unless you are familiar with the implementation of thealgorithm, or you discover some counter-intuitive behavior while writing theexamples for the function.

This section follows the same format as the extended summary section.

Section 7: examples#

This is one of the most important sections of a docstring, despite beingplaced in the last position, as often people understand concepts betterby example than through accurate explanations.

Examples in docstrings, besides illustrating the usage of the function ormethod, must be valid Python code, that returns the given output in adeterministic way, and that can be copied and run by users.

Examples are presented as a session in the Python terminal.>>> is used topresent code.... is used for code continuing from the previous line.Output is presented immediately after the last line of code generating theoutput (no blank lines in between). Comments describing the examples canbe added with blank lines before and after them.

The way to present examples is as follows:

  1. Import required libraries (exceptnumpy andpandas)

  2. Create the data required for the example

  3. Show a very basic example that gives an idea of the most common use case

  4. Add examples with explanations that illustrate how the parameters can beused for extended functionality

A simple example could be:

classSeries:defhead(self,n=5):"""        Return the first elements of the Series.        This function is mainly useful to preview the values of the        Series without displaying all of it.        Parameters        ----------        n : int            Number of values to return.        Return        ------        pandas.Series            Subset of the original series with the n first values.        See Also        --------        tail : Return the last n elements of the Series.        Examples        --------        >>> ser = pd.Series(['Ant', 'Bear', 'Cow', 'Dog', 'Falcon',        ...                'Lion', 'Monkey', 'Rabbit', 'Zebra'])        >>> ser.head()        0   Ant        1   Bear        2   Cow        3   Dog        4   Falcon        dtype: object        With the ``n`` parameter, we can change the number of returned rows:        >>> ser.head(n=3)        0   Ant        1   Bear        2   Cow        dtype: object        """returnself.iloc[:n]

The examples should be as concise as possible. In cases where the complexity ofthe function requires long examples, is recommended to use blocks with headersin bold. Use double star** to make a text bold, like in**thisexample**.

Conventions for the examples#

Code in examples is assumed to always start with these two lines which are notshown:

importnumpyasnpimportpandasaspd

Any other module used in the examples must be explicitly imported, one per line (asrecommended inPEP 8#imports)and avoiding aliases. Avoid excessive imports, but if needed, imports fromthe standard library go first, followed by third-party libraries (likematplotlib).

When illustrating examples with a singleSeries use the nameser, and ifillustrating with a singleDataFrame use the namedf. For indices,idx is the preferred name. If a set of homogeneousSeries orDataFrame is used, name themser1,ser2,ser3… ordf1,df2,df3… If the data is not homogeneous, and more than one structureis needed, name them with something meaningful, for exampledf_main anddf_to_join.

Data used in the example should be as compact as possible. The number of rowsis recommended to be around 4, but make it a number that makes sense for thespecific example. For example in thehead method, it requires to be higherthan 5, to show the example with the default values. If doing themean, wecould use something like[1,2,3], so it is easy to see that the valuereturned is the mean.

For more complex examples (grouping for example), avoid using data withoutinterpretation, like a matrix of random numbers with columns A, B, C, D…And instead use a meaningful example, which makes it easier to understand theconcept. Unless required by the example, use names of animals, to keep examplesconsistent. And numerical properties of them.

When calling the method, keywords argumentshead(n=3) are preferred topositional argumentshead(3).

Good:

classSeries:defmean(self):"""        Compute the mean of the input.        Examples        --------        >>> ser = pd.Series([1, 2, 3])        >>> ser.mean()        2        """passdeffillna(self,value):"""        Replace missing values by ``value``.        Examples        --------        >>> ser = pd.Series([1, np.nan, 3])        >>> ser.fillna(0)        [1, 0, 3]        """passdefgroupby_mean(self):"""        Group by index and return mean.        Examples        --------        >>> ser = pd.Series([380., 370., 24., 26],        ...               name='max_speed',        ...               index=['falcon', 'falcon', 'parrot', 'parrot'])        >>> ser.groupby_mean()        index        falcon    375.0        parrot     25.0        Name: max_speed, dtype: float64        """passdefcontains(self,pattern,case_sensitive=True,na=numpy.nan):"""        Return whether each value contains ``pattern``.        In this case, we are illustrating how to use sections, even        if the example is simple enough and does not require them.        Examples        --------        >>> ser = pd.Series('Antelope', 'Lion', 'Zebra', np.nan)        >>> ser.contains(pattern='a')        0    False        1    False        2     True        3      NaN        dtype: bool        **Case sensitivity**        With ``case_sensitive`` set to ``False`` we can match ``a`` with both        ``a`` and ``A``:        >>> s.contains(pattern='a', case_sensitive=False)        0     True        1    False        2     True        3      NaN        dtype: bool        **Missing values**        We can fill missing values in the output using the ``na`` parameter:        >>> ser.contains(pattern='a', na=False)        0    False        1    False        2     True        3    False        dtype: bool        """pass

Bad:

defmethod(foo=None,bar=None):"""    A sample DataFrame method.    Do not import NumPy and pandas.    Try to use meaningful data, when it makes the example easier    to understand.    Try to avoid positional arguments like in ``df.method(1)``. They    can be all right if previously defined with a meaningful name,    like in ``present_value(interest_rate)``, but avoid them otherwise.    When presenting the behavior with different parameters, do not place    all the calls one next to the other. Instead, add a short sentence    explaining what the example shows.    Examples    --------    >>> import numpy as np    >>> import pandas as pd    >>> df = pd.DataFrame(np.random.randn(3, 3),    ...                   columns=('a', 'b', 'c'))    >>> df.method(1)    21    >>> df.method(bar=14)    123    """pass

Tips for getting your examples pass the doctests#

Getting the examples pass the doctests in the validation script can sometimesbe tricky. Here are some attention points:

  • Import all needed libraries (except for pandas and NumPy, those are alreadyimported asimportpandasaspd andimportnumpyasnp) and defineall variables you use in the example.

  • Try to avoid using random data. However random data might be OK in somecases, like if the function you are documenting deals with probabilitydistributions, or if the amount of data needed to make the function resultmeaningful is too much, such that creating it manually is very cumbersome.In those cases, always use a fixed random seed to make the generated examplespredictable. Example:

    >>>np.random.seed(42)>>>df=pd.DataFrame({'normal':np.random.normal(100,5,20)})
  • If you have a code snippet that wraps multiple lines, you need to use ‘…’on the continued lines:

    >>>df=pd.DataFrame([[1,2,3],[4,5,6]],index=['a','b','c'],...columns=['A','B'])
  • If you want to show a case where an exception is raised, you can do:

    >>>pd.to_datetime(["712-01-01"])Traceback (most recent call last):OutOfBoundsDatetime:Out of bounds nanosecond timestamp: 712-01-01 00:00:00

    It is essential to include the “Traceback (most recent call last):”, but forthe actual error only the error name is sufficient.

  • If there is a small part of the result that can vary (e.g. a hash in an objectrepresentation), you can use... to represent this part.

    If you want to show thats.plot() returns a matplotlib AxesSubplot object,this will fail the doctest

    >>>s.plot()<matplotlib.axes._subplots.AxesSubplot at 0x7efd0c0b0690>

    However, you can do (notice the comment that needs to be added)

    >>>s.plot()<matplotlib.axes._subplots.AxesSubplot at ...>

Plots in examples#

There are some methods in pandas returning plots. To render the plots generatedby the examples in the documentation, the..plot:: directive exists.

To use it, place the next code after the “Examples” header as shown below. Theplot will be generated automatically when building the documentation.

classSeries:defplot(self):"""        Generate a plot with the ``Series`` data.        Examples        --------        .. plot::            :context: close-figs            >>> ser = pd.Series([1, 2, 3])            >>> ser.plot()        """pass

Sharing docstrings#

pandas has a system for sharing docstrings, with slight variations, betweenclasses. This helps us keep docstrings consistent, while keeping things clearfor the user reading. It comes at the cost of some complexity when writing.

Each shared docstring will have a base template with variables, like{klass}. The variables filled in later on using thedoc decorator.Finally, docstrings can also be appended to with thedoc decorator.

In this example, we’ll create a parent docstring normally (this is likepandas.core.generic.NDFrame). Then we’ll have two children (likepandas.Series andpandas.DataFrame). We’llsubstitute the class names in this docstring.

classParent:@doc(klass="Parent")defmy_function(self):"""Apply my function to {klass}."""...classChildA(Parent):@doc(Parent.my_function,klass="ChildA")defmy_function(self):...classChildB(Parent):@doc(Parent.my_function,klass="ChildB")defmy_function(self):...

The resulting docstrings are

>>>print(Parent.my_function.__doc__)Apply my function to Parent.>>>print(ChildA.my_function.__doc__)Apply my function to ChildA.>>>print(ChildB.my_function.__doc__)Apply my function to ChildB.

Notice:

  1. We “append” the parent docstring to the children docstrings, which areinitially empty.

Our files will often contain a module-level_shared_doc_kwargs with somecommon substitution values (things likeklass,axes, etc).

You can substitute and append in one shot with something like

@doc(template,**_shared_doc_kwargs)defmy_function(self):...

wheretemplate may come from a module-level_shared_docs dictionarymapping function names to docstrings. Wherever possible, we prefer usingdoc, since the docstring-writing processes is slightly closer to normal.

Seepandas.core.generic.NDFrame.fillna for an example template, andpandas.Series.fillna andpandas.core.generic.frame.fillnafor the filled versions.