SparkSession(sparkContext[, jsparkSession, ...])

The entry point to programming Spark with the Dataset and DataFrame API.

Catalog(sparkSession)

User-facing catalog API, accessible throughSparkSession.catalog.

DataFrame(jdf, sql_ctx)

A distributed collection of data grouped into named columns.

Column(*args, **kwargs)

A column in a DataFrame.

Observation(*args, **kwargs)

Class to observe (named) metrics on aDataFrame.

Row(*args, **kwargs)

A row inDataFrame.

GroupedData(jgd, df)

A set of methods for aggregations on aDataFrame, created byDataFrame.groupBy().

PandasCogroupedOps(gd1, gd2)

A logical grouping of twoGroupedData, created byGroupedData.cogroup().

DataFrameNaFunctions(df)

Functionality for working with missing data inDataFrame.

DataFrameStatFunctions(df)

Functionality for statistic functions withDataFrame.

Window()

Utility functions for defining window in DataFrames.

DataFrameReader(spark)

Interface used to load aDataFrame from external storage systems (e.g.

DataFrameWriter(df)

Interface used to write aDataFrame to external storage systems (e.g.

DataFrameWriterV2(df, table)

Interface used to write a class:pyspark.sql.dataframe.DataFrame to external storage using the v2 API.

UDFRegistration(sparkSession)

Wrapper for user-defined function registration.

UDTFRegistration(sparkSession)

Wrapper for user-defined table function registration.

udf.UserDefinedFunction(func[, returnType, ...])

User defined function in Python

udtf.UserDefinedTableFunction(func, returnType)

User-defined table function in Python

datasource.DataSource(options)

A base class for data sources.

datasource.DataSourceReader()

A base class for data source readers.

datasource.DataSourceStreamReader()

A base class for streaming data source readers.

datasource.DataSourceWriter()

A base class for data source writers.

datasource.DataSourceRegistration(sparkSession)

Wrapper for data source registration.

datasource.InputPartition(value)

A base class representing an input partition returned by thepartitions() method ofDataSourceReader.

datasource.WriterCommitMessage()

A commit message returned by theDataSourceWriter.write() and will be sent back to the driver side as input parameter ofDataSourceWriter.commit() orDataSourceWriter.abort() method.

tvf.TableValuedFunction(sparkSession)

Interface for invoking table-valued functions in Spark SQL.

VariantVal(value, metadata)

A class to represent a Variant value in Python.

table_arg.TableArg()

Represents a table argument in PySpark.