Google Cloud Serverless for Apache Spark lets you run Spark workloads without requiring youto provision and manage your own Dataproc cluster.There are two ways to run Serverless for Apache Spark workloads:batch workloads and interactive sessions.

Batch workloads

Submit a batch workload to the Serverless for Apache Spark service using theGoogle Cloud console, Google Cloud CLI, or Dataproc API. The serviceruns the workload on a managed compute infrastructure, autoscaling resourcesas needed.Serverless for Apache Spark charges applyonly to the time when the workload is executing.

Batch workload capabilities

You can run the following Serverless for Apache Spark batch workload types:

• PySpark
• Spark SQL
• Spark R
• Spark (Java or Scala)

You can specifySpark propertieswhen you submit a Serverless for Apache Spark batch workload.

Schedule batch workloads

You can schedule a Spark batch workload as part of anAirflow orCloud Composerworkflow using anAirflow batch operator.For more information, seeRun Serverless for Apache Spark workloads with Cloud Composer.

Get started

To get started, seeRun an Apache Spark batch workload.

Interactive sessions

Write and run code in Jupyter notebooks during a Serverless for Apache Sparkinteractive session. You can create a notebook session in the followingways:

• Run PySpark code in BigQuery Studio notebooks.Open a BigQuery Python notebook to create aSpark-Connect-basedServerless for Apache Spark interactive session. Each BigQuerynotebook can have only one active Serverless for Apache Spark session associatedwith it.

• Use the Dataproc JupyterLab pluginto create multiple Jupyter notebook sessions from templates that you createand manage. When you install the plugin on a local machine or Compute EngineVM, different cards that correspond to different Spark kernel configurationsappear on the JupyterLab launcher page. Click a card to create a Serverless for Apache Sparknotebook session, then start writing and testing your code in the notebook.

  The Dataproc JupyterLab plugin also lets youuse the JupyterLab launcher page to take the following actions:

  • Create Dataproc on Compute Engine clusters.
  • Submit jobs to Dataproc on Compute Engine clusters.
  • View Google Cloud and Spark logs.

Security compliance

Serverless for Apache Spark adheres to alldata residency,CMEK,VPC-SC,and other security requirements that Dataproc is compliant with.