Code standards#

Writing good code is not just about what you write. It is also abouthow youwrite it. DuringContinuous Integration testing, severaltools will be run to check your code for stylistic errors.Generating any warnings will cause the test to fail.Thus, good style is a requirement for submitting code to pandas.

There are a couple of tools in pandas to help contributors verify their changesbefore contributing to the project

• ./ci/code_checks.sh: a script validates the doctests, formatting in docstrings,and imported modules. It is possible to run the checks independently by using theparametersdocstrings,code, anddoctests(e.g../ci/code_checks.shdoctests);

• pre-commit, which we go into detail on in the next section.

In addition, because a lot of people use our library, it is important that wedo not make sudden changes to the code that could have the potential to breaka lot of user code as a result, that is, we need it to be asbackwards compatibleas possible to avoid mass breakages.

Pre-commit#

Additionally,Continuous Integration will run code formatting checkslikeruff,isort, andclang-format and more usingpre-commit hooks.Any warnings from these checks will cause theContinuous Integration to fail; therefore,it is helpful to run the check yourself before submitting code. Thiscan be done by installingpre-commit (which should already have happened if you followed the instructionsinSetting up your development environment) and then running:

pre-commitinstall

from the root of the pandas repository. Now all of the styling checks will berun each time you commit changes without your needing to run each one manually.In addition, usingpre-commit will also allow you to more easilyremain up-to-date with our code checks as they change.

Note that if needed, you can skip these checks withgitcommit--no-verify.

If you don’t want to usepre-commit as part of your workflow, you can still use itto run its checks with one of the following:

pre-commitrun--files<filesyouhavemodified>pre-commitrun--from-ref=upstream/main--to-ref=HEAD--all-files

without needing to have donepre-commitinstall beforehand.

Finally, we also have some slow pre-commit checks, which don’t run on each commitbut which do run during continuous integration. You can trigger them manually with:

pre-commitrun--hook-stagemanual--all-files

Optional dependencies#

Optional dependencies (e.g. matplotlib) should be imported with the private helperpandas.compat._optional.import_optional_dependency. This ensures aconsistent error message when the dependency is not met.

All methods using an optional dependency should include a test asserting that anImportError is raised when the optional dependency is not found. This testshould be skipped if the library is present.

All optional dependencies should be documented inOptional dependencies and the minimum required version should beset in thepandas.compat._optional.VERSIONS dict.

Backwards compatibility#

Please try to maintain backward compatibility. pandas has lots of users with lots ofexisting code, so don’t break it if at all possible. If you think breakage is required,clearly state why as part of the pull request. Also, be careful when changing methodsignatures and add deprecation warnings where needed. Also, add the deprecated sphinxdirective to the deprecated functions or methods.

If a function with the same arguments as the one being deprecated exist, you can usethepandas.util._decorators.deprecate:

frompandas.util._decoratorsimportdeprecatedeprecate('old_func','new_func','1.1.0')

Otherwise, you need to do it manually:

importwarningsfrompandas.util._exceptionsimportfind_stack_leveldefold_func():"""Summary of the function.    .. deprecated:: 1.1.0       Use new_func instead.    """warnings.warn('Use new_func instead.',FutureWarning,stacklevel=find_stack_level(),)new_func()defnew_func():pass

You’ll also need to

1. Write a new test that asserts a warning is issued when calling with the deprecated argument

2. Update all of pandas existing tests and code to use the new argument

SeeTesting a warning for more.

Type hints#

pandas strongly encourages the use ofPEP 484 style type hints. New development should contain type hints and pull requests to annotate existing code are accepted as well!

classSomeClass1:str=None

str_type=strclassSomeClass2:str:str_type=None

fromtypingimportcastfrompandas.core.dtypes.commonimportis_numberdefcannot_infer_bad(obj:Union[str,int,float]):ifis_number(obj):...else:# Reasonably only str objects would reach this but...obj=cast(str,obj)# Mypy complains without this!returnobj.upper()

defcannot_infer_good(obj:Union[str,int,float]):ifisinstance(obj,str):returnobj.upper()else:...

frompandas._typingimportDtypedefas_type(dtype:Dtype)->...:...

pre-commitrun--hook-stagemanual--all-filesmypypre-commitrun--hook-stagemanual--all-filespyrightpre-commitrun--hook-stagemanual--all-filespyright_reportGeneralTypeIssues# the following might fail if the installed pandas version does not correspond to your local git versionpre-commitrun--hook-stagemanual--all-filesstubtest

python -c "import pandas; import pathlib; (pathlib.Path(pandas.__path__[0]) / 'py.typed').touch()"

Testing with continuous integration#

The pandas test suite will run automatically onGitHub Actionscontinuous integration services, once your pull request is submitted.However, if you wish to run the test suite on a branch prior to submitting the pull request,then the continuous integration services need to be hooked to your GitHub repository. Instructions are hereforGitHub Actions.

A pull-request will be considered for merging when you have an all ‘green’ build. If any tests are failing,then you will get a red ‘X’, where you can click through to see the individual failed tests.This is an example of a green build.

Test-driven development#

pandas is serious about testing and strongly encourages contributors to embracetest-driven development (TDD).This development process “relies on the repetition of a very short development cycle:first the developer writes an (initially failing) automated test case that defines a desiredimprovement or new function, then produces the minimum amount of code to pass that test.”So, before actually writing any code, you should write your tests. Often the test can betaken from the original GitHub issue. However, it is always worth considering additionaluse cases and writing corresponding tests.

We usecode coverage to help understandthe amount of code which is covered by a test. We recommend striving to ensure codeyou add or change within Pandas is covered by a test. Please see ourcode coverage dashboard through Codecovfor more information.

Adding tests is one of the most common requests after code is pushed to pandas. Therefore,it is worth getting in the habit of writing tests ahead of time so this is never an issue.

classTestReallyCoolFeature:deftest_cool_feature_aspect(self):pass

deftest_really_cool_feature():pass

deftest_xfail(request):mark=pytest.mark.xfail(raises=TypeError,reason="Indicate why here")request.applymarker(mark)

withtm.assert_produces_warning(DeprecationWarning,match="the warning message"):pd.deprecated_function()

withtm.assert_produces_warning(False):pd.no_warning_function()

@pytest.mark.filterwarnings("ignore:msg:category")deftest_thing(self):pass

withpytest.raises(ValueError,match="an error"):raiseValueError("an error")

deftest_something(temp_file):pd.DataFrame([1]).to_csv(str(temp_file))

@pytest.mark.network@pytest.mark.single_cpudeftest_network(httpserver):httpserver.serve_content(content="content")result=pd.read_html(httpserver.url)

importpytestimportnumpyasnpimportpandasaspd@pytest.mark.parametrize('dtype',['int8','int16','int32','int64'])deftest_dtypes(dtype):assertstr(np.dtype(dtype))==dtype@pytest.mark.parametrize('dtype',['float32',pytest.param('int16',marks=pytest.mark.skip),pytest.param('int32',marks=pytest.mark.xfail(reason='to show how it works'))])deftest_mark(dtype):assertstr(np.dtype(dtype))=='float32'@pytest.fixturedefseries():returnpd.Series([1,2,3])@pytest.fixture(params=['int8','int16','int32','int64'])defdtype(request):returnrequest.paramdeftest_series(series,dtype):# GH <issue_number>result=series.astype(dtype)assertresult.dtype==dtypeexpected=pd.Series([1,2,3],dtype=dtype)tm.assert_series_equal(result,expected)

((pandas)bash-3.2$pytesttest_cool_feature.py-v===========================testsessionstarts===========================platformdarwin--Python3.6.2,pytest-3.6.0,py-1.4.31,pluggy-0.4.0collected11itemstester.py::test_dtypes[int8]PASSEDtester.py::test_dtypes[int16]PASSEDtester.py::test_dtypes[int32]PASSEDtester.py::test_dtypes[int64]PASSEDtester.py::test_mark[float32]PASSEDtester.py::test_mark[int16]SKIPPEDtester.py::test_mark[int32]xfailtester.py::test_series[int8]PASSEDtester.py::test_series[int16]PASSEDtester.py::test_series[int32]PASSEDtester.py::test_series[int64]PASSED

((pandas)bash-3.2$pytesttest_cool_feature.py-v-kint8===========================testsessionstarts===========================platformdarwin--Python3.6.2,pytest-3.6.0,py-1.4.31,pluggy-0.4.0collected11itemstest_cool_feature.py::test_dtypes[int8]PASSEDtest_cool_feature.py::test_series[int8]PASSED

importjsonfromhypothesisimportgiven,strategiesasstany_json_value=st.deferred(lambda:st.one_of(st.none(),st.booleans(),st.floats(allow_nan=False),st.text(),st.lists(any_json_value),st.dictionaries(st.text(),any_json_value)))@given(value=any_json_value)deftest_json_roundtrip(value):result=json.loads(json.dumps(value))assertvalue==result

Running the test suite#

The tests can then be run directly inside your Git clone (without having toinstall pandas) by typing:

pytestpandas

Often it is worth running only a subset of tests first around your changes before running theentire suite.

The easiest way to do this is with:

pytestpandas/path/to/test.py-kregex_matching_test_name

Or with one of the following constructs:

pytestpandas/tests/[test-module].pypytestpandas/tests/[test-module].py::[TestClass]pytestpandas/tests/[test-module].py::[TestClass]::[test_method]

Usingpytest-xdist, which isincluded in our ‘pandas-dev’ environment, one can speed up local testing onmulticore machines. The-n number flag then can be specified when runningpytest to parallelize a test run across the number of specified cores or auto toutilize all the available cores on your machine.

# Utilize 4 corespytest-n4pandas# Utilizes all available corespytest-nautopandas

If you’d like to speed things along further a more advanced use of thiscommand would look like this

pytestpandas-n4-m"not slow and not network and not db and not single_cpu"-rsxX

In addition to the multithreaded performance increase this improves testspeed by skipping some tests using the-m mark flag:

• slow: any test taking long (think seconds rather than milliseconds)

• network: tests requiring network connectivity

• db: tests requiring a database (mysql or postgres)

• single_cpu: tests that should run on a single cpu only

You might want to enable the following option if it’s relevant for you:

• arm_slow: any test taking long on arm64 architecture

These markers are definedin this toml file, under[tool.pytest.ini_options] in a list calledmarkers, in caseyou want to check if new ones have been created which are of interest to you.

The-r report flag will display a short summary info (seepytestdocumentation). Here we are displaying the number of:

• s: skipped tests

• x: xfailed tests

• X: xpassed tests

The summary is optional and can be removed if you don’t need the addedinformation. Using the parallelization option can significantly reduce thetime it takes to locally run tests before submitting a pull request.

If you require assistance with the results,which has happened in the past, please set a seed before running the commandand opening a bug report, that way we can reproduce it. Here’s an examplefor setting a seed on windows

setPYTHONHASHSEED=314159265pytestpandas-n4-m"not slow and not network and not db and not single_cpu"-rsxX

On Unix use

exportPYTHONHASHSEED=314159265pytestpandas-n4-m"not slow and not network and not db and not single_cpu"-rsxX

For more, see thepytest documentation.

Furthermore one can run

pd.test()

with an imported pandas to run tests similarly.

Running the performance test suite#

Performance matters and it is worth considering whether your code has introducedperformance regressions. pandas is in the process of migrating toasv benchmarksto enable easy monitoring of the performance of critical pandas operations.These benchmarks are all found in thepandas/asv_bench directory, and thetest results can be foundhere.

To use all features of asv, you will need eitherconda orvirtualenv. For more details please check theasv installationwebpage.

To install asv:

pipinstallgit+https://github.com/airspeed-velocity/asv

If you need to run a benchmark, change your directory toasv_bench/ and run:

asvcontinuous-f1.1upstream/mainHEAD

You can replaceHEAD with the name of the branch you are working on,and report benchmarks that changed by more than 10%.The command usesconda by default for creating the benchmarkenvironments. If you want to use virtualenv instead, write:

asvcontinuous-f1.1-Evirtualenvupstream/mainHEAD

The-Evirtualenv option should be added to allasv commandsthat run benchmarks. The default value is defined inasv.conf.json.

Running the full benchmark suite can be an all-day process, depending on yourhardware and its resource utilization. However, usually it is sufficient to pasteonly a subset of the results into the pull request to show that the committed changesdo not cause unexpected performance regressions. You can run specific benchmarksusing the-b flag, which takes a regular expression. For example, this willonly run benchmarks from apandas/asv_bench/benchmarks/groupby.py file:

asvcontinuous-f1.1upstream/mainHEAD-b^groupby

If you want to only run a specific group of benchmarks from a file, you can do itusing. as a separator. For example:

asvcontinuous-f1.1upstream/mainHEAD-bgroupby.GroupByMethods

will only run theGroupByMethods benchmark defined ingroupby.py.

You can also run the benchmark suite using the version ofpandasalready installed in your current Python environment. This can beuseful if you do not have virtualenv or conda, or are using thesetup.pydevelop approach discussed above; for the in-place buildyou need to setPYTHONPATH, e.g.PYTHONPATH="$PWD/.."asv[remainingarguments].You can run benchmarks using an existing Pythonenvironment by:

asvrun-e-Eexisting

or, to use a specific Python interpreter,:

asvrun-e-Eexisting:python3.6

This will display stderr from the benchmarks, and use your localpython that comes from your$PATH.

Information on how to write a benchmark and how to use asv can be found in theasv documentation.

Documenting your code#

Changes should be reflected in the release notes located indoc/source/whatsnew/vx.y.z.rst.This file contains an ongoing change log for each release. Add an entry to this file todocument your fix, enhancement or (unavoidable) breaking change. Make sure to include theGitHub issue number when adding your entry (using:issue:`1234` where1234 is theissue/pull request number). Your entry should be written using full sentences and propergrammar.

When mentioning parts of the API, use a Sphinx:func:,:meth:, or:class:directive as appropriate. Not all public API functions and methods have adocumentation page; ideally links would only be added if they resolve. You canusually find similar examples by checking the release notes for one of the previousversions.

If your code is a bugfix, add your entry to the relevant bugfix section. Avoidadding to theOther section; only in rare cases should entries go there.Being as concise as possible, the description of the bug should include how theuser may encounter it and an indication of the bug itself, e.g.“produces incorrect results” or “incorrectly raises”. It may be necessary to alsoindicate the new behavior.

If your code is an enhancement, it is most likely necessary to add usageexamples to the existing documentation. This can be done following the sectionregardingdocumentation.Further, to let users know when this feature was added, theversionaddeddirective is used. The sphinx syntax for that is:

..versionadded:: 2.1.0

This will put the textNew in version 2.1.0 wherever you put the sphinxdirective. This should also be put in the docstring when adding a new functionor method (example)or a new keyword argument (example).