Python Versions Supported#

Python 3.9 and above.

Using PyPI#

PySpark installation usingPyPI (pyspark) is as follows:

pipinstallpyspark

If you want to install extra dependencies for a specific component, you can install it as below:

# Spark SQLpipinstallpyspark[sql]# pandas API on Sparkpipinstallpyspark[pandas_on_spark]plotly# to plot your data, you can install plotly together.# Spark Connectpipinstallpyspark[connect]

SeeOptional dependencies for more detail about extra dependencies.

For PySpark with/without a specific Hadoop version, you can install it by usingPYSPARK_HADOOP_VERSION environment variables as below:

PYSPARK_HADOOP_VERSION=3pipinstallpyspark

The default distribution uses Hadoop 3.3 and Hive 2.3. If users specify different versions of Hadoop, the pip installation automaticallydownloads a different version and uses it in PySpark. Downloading it can take a while depending onthe network and the mirror chosen.PYSPARK_RELEASE_MIRROR can be set to manually choose the mirror for faster downloading.

PYSPARK_RELEASE_MIRROR=http://mirror.apache-kr.orgPYSPARK_HADOOP_VERSION=3pipinstall

It is recommended to use-v option inpip to track the installation and download status.

PYSPARK_HADOOP_VERSION=3pipinstallpyspark-v

Supported values inPYSPARK_HADOOP_VERSION are:

• without: Spark pre-built with user-provided Apache Hadoop

• 3: Spark pre-built for Apache Hadoop 3.3 and later (default)

Note that this installation of PySpark with/without a specific Hadoop version is experimental. It can change or be removed between minor releases.

pipinstallpyspark-connect

pipinstallpyspark-client

Using Conda#

Conda is an open-source package management and environment management system (developed byAnaconda), which is best installed throughMiniconda orMiniforge.The tool is both cross-platform and language agnostic, and in practice, conda can replace bothpip andvirtualenv.

Conda uses so-called channels to distribute packages, and together with the default channels byAnaconda itself, the most important channel isconda-forge, whichis the community-driven packaging effort that is the most extensive & the most current (and alsoserves as the upstream for the Anaconda channels in most cases).

To create a new conda environment from your terminal and activate it, proceed as shown below:

condacreate-npyspark_envcondaactivatepyspark_env

After activating the environment, use the following command to install pyspark,a python version of your choice, as well as other packages you want to use inthe same session as pyspark (you can install in several steps too).

condainstall-cconda-forgepyspark# can also add "python=3.9 some_package [etc.]" here

Note thatPySpark for conda is maintainedseparately by the community; while new versions generally get packaged quickly, theavailability through conda(-forge) is not directly in sync with the PySpark release cycle.

While using pip in a conda environment is technically feasible (with the same command asabove), this approach isdiscouraged,because pip does not interoperate with conda.

For a short summary about useful conda commands, see theircheat sheet.

Manually Downloading#

PySpark is included in the distributions available at theApache Spark website.You can download a distribution you want from the site. After that, uncompress the tar file into the directory where you wantto install Spark, for example, as below:

tarxzvfspark-\|release|\-bin-hadoop3.tgz

Ensure theSPARK_HOME environment variable points to the directory where the tar file has been extracted.UpdatePYTHONPATH environment variable such that it can find the PySpark and Py4J underSPARK_HOME/python/lib.One example of doing this is shown below:

cdspark-\|release|\-bin-hadoop3exportSPARK_HOME=`pwd`exportPYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip);IFS=:;echo"${ZIPS[*]}"):$PYTHONPATH

Installing from Source#

To install PySpark from source, refer toBuilding Spark.

Dependencies#