Repository files navigation

A collection of the Apache Sparkstubfiles. Thesefiles were generated bystubgenand manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to theTypeshed project. Please referto itscontributorslist andlicense fordetails.

This projecthas been merged with the main Apache Spark repository (SPARK-32714). All further development for Spark 3.1 and onwards will be continued there.

For Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.

• If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.
• Otherwise, please checkthe official Spark JIRA andcontributing guidelines. If you create a JIRA ticket or Spark PR related to type hints, please ping me with[~zero323] or@zero323 respectively. Thanks in advance.

• Static error detection (seeSPARK-20631)

  

• Improved autocompletion.

  

Please note that the guidelines for distribution of type information isstill work in progress (PEP 561 - Distributing and Packaging TypeInformation). Currentlyinstallation script overlays existing Spark installations (pyi stubfiles are copied next to theirpy counterparts in the PySparkinstallation directory). If this approach is not acceptable you can add stubfiles to the search path manually.

According toPEP484:

Third-party stub packages can use any location for stub storage.Type checkers should search for them using PYTHONPATH.

Moreover:

Third-party stub packages can use any location for stub storage.Type checkers should search for them using PYTHONPATH. A defaultfallback directory that is always checked isshared/typehints/python3.5/ (or 3.6, etc.)

Please check usage before proceeding.

The package is available onPYPI:

pip install pyspark-stubs

andconda-forge:

conda install -c conda-forge pyspark-stubs

Depending on your environment you might also need a type checker, likeMypyorPytype[1], and autocompletion tool, likeJedi.

This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.

Package versions follow PySpark versions with exception to maintenance releases - i.e. pyspark-stubs==2.3.0 should be compatible with pyspark>=2.3.0,<2.4.0.Maintenance releases (post1, post2, ..., postN) are reserved for internal annotations updates.

As of release 2.4.0 most of the public API is covered. For details please checkAPI coverage document.

• SPARK-17333 -Make pyspark interface friendly with static analysis.
• PySpark typing hints andRevisiting PySpark type annotations onApache Spark Developers List.

Apache Spark, Spark, PySpark, Apache, and the Spark logo aretrademarks ofTheApache Software Foundation. This project is not owned, endorsed, orsponsored by The Apache Software Foundation.