This document describes how to install and use the JupyterLabextension on a machine or self-managed VM that has access to Google services. Italso describes how to develop and deploy serverless Spark notebook code.

Install the extension within minutes to take advantage of the following features:

• Launch serverless Spark & BigQuery notebooks to develop code quickly
• Browse and preview BigQuery datasets in JupyterLab
• Edit Cloud Storage files in JupyterLab
• Schedule a notebook on Composer

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Enable the Dataproc API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the API

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Enable the Dataproc API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the API

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Install the JupyterLab extension

You can install and use the JupyterLab extension on amachine or VM that has access to Google services, such as your local machine oraCompute Engine VM instance.

To install the extension, follow these steps:

1. Download and install Python version 3.11 or higher frompython.org/downloads.

   • Verify the Python 3.11+ installation.

     python3 --version

2. Virtualize the Python environment.

   pip3 install pipenv

   • Create a installation folder.

     mkdir jupyter

   • Change to the installation folder.

     cd jupyter

   • Create a virtual environment.

     pipenv shell

3. Install JupyterLab in the virtual environment.

   pipenv install jupyterlab

4. Install the JupyterLab extension.

   pipenv install bigquery-jupyter-plugin

5. Start JupyterLab.

   jupyter lab

   1. The JupyterLabLauncher page opens in your browser. It containsaDataproc Jobs and Sessions section. It can also containServerless for Apache Spark Notebooks andDataproc Cluster Notebooks sectionsif you have access to Dataproc serverless notebooks or Dataproc clusterswith theJupyter optional componentrunning in your project.

      

      On macOS, if you receive aSSL: CERTIFICATE_VERIFY_FAILED errorin your terminal when you launch Jupyterlab, update your Python SSL certificateby executingInstall Certificates.command from the Python installation path.This file is located in the Python home directory.

   2. By default, your Serverless for Apache Spark Interactive session runsin the project and region you set when you rangcloud init inBefore you begin. You can change the project andregion settings for your sessions fromJupyterLabSettings > Google Cloud Settings > Google Cloud Project Settings.

      You must restart the extension for the changes to take effect.

Create a Serverless for Apache Spark runtime template

Serverless for Apache Spark runtime templates (also called session templates)contain configuration settings for executing Spark code in a session. You cancreate and manage runtime templates using Jupyterlab or the gcloud CLI.

Launch and manage notebooks

Afterinstalling the Dataproc JupyterLab extension,you can click template cards on the JupyterLabLauncher page to:

• Launch a Jupyter notebook on Serverless for Apache Spark.

• Launch a Jupyter notebook on a Dataproc on Compute Engine cluster.

Launch a Jupyter notebook on Serverless for Apache Spark

TheServerless for Apache Spark Notebooks sectionon the JupyterLab Launcher page displays notebook templatecards that map to Serverless for Apache Spark runtime templates (seeCreate a Serverless for Apache Spark runtime template).

1. Click a card to create a Serverless for Apache Spark session andlaunch a notebook. When session creation is complete and the notebookkernel is ready to use, the kernel status changesfromStarting toIdle (Ready).

2. Write and test notebook code.

   1. Copy and paste the following PySparkPi estimation codein the PySpark notebook cell, then pressShift+Return torun the code.

      importrandom    definside(p):    x,y=random.random(),random.random()    returnx*x+y*y <1    count=sc.parallelize(range(0,10000)).filter(inside).count()print("Pi is roughly%f"%(4.0*count/10000))

      Notebook result:

      

3. After creating and using a notebook, you can terminate the notebook sessionby clickingShut Down Kernel from theKernel tab.

   • To reuse the session, create a new notebook by choosingNotebook from theFile>>New menu. After the new notebook is created, choose the existing session from the kernel selection dialog. The new notebook will reuse the session and retain the session context from the previous notebook.

4. If you don't terminate the session, Dataproc terminates the sessionwhen the session idle timer expires. You can configure the sessionidle time in theruntime template configuration.The default session idle time is one hour.

Launch a notebook on a Dataproc on Compute Engine cluster

If you created aDataproc on Compute Engine Jupyter cluster,the JupyterLabLauncher page contains aDataproc Cluster Notebook section with pre-installed kernel cards.

To launch a Jupyter notebook on your Dataproc onCompute Engine cluster:

1. Click a card in theDataproc Cluster Notebook section.

2. When the kernel status changes fromStarting toIdle (Ready), you canstart writing and executing notebook code.

3. After creating and using a notebook, you can terminate the notebook sessionby clickingShut Down Kernel from theKernel tab.

Manage input and output files in Cloud Storage

Analyzing exploratory data and building ML models often involvesfile-based inputs and outputs. Serverless for Apache Spark accesses these fileson Cloud Storage.

• To access the Cloud Storage browser, click the Cloud Storage browsericon in the JupyterLabLauncher page sidebar, then double-click afolder to view its contents.

  

• You can click Jupyter-supported file types to open and edit them. When yousave changes to the files, they are written to Cloud Storage.

• To create a new Cloud Storage folder, click the new folder icon,then enter the name of the folder.

• To upload files into a Cloud Storage bucket or a folder, clickthe upload icon, then select the files to upload.

Develop Spark notebook code

Afterinstalling the Dataproc JupyterLab extension,you can launch Jupyter notebooks from the JupyterLabLauncher page to developapplication code.

PySpark and Python code development

Serverless for Apache Spark and Dataproc on Compute Engineclusters support PySpark kernels. Dataproc on Compute Enginealso supports Python kernels.

• Click a PySpark card in theServerless for Apache Spark NotebooksorDataproc Cluster Notebooksection on the JupyterLabLauncher page to open a PySpark notebook.

• Click a Python kernel card in theDataproc Cluster Notebook sectionon the JupyterLabLauncher page to open a Python notebook.

SQL code development

To open a PySpark notebook to write and execute SQL code, on the JupyterLabLauncher page, in theServerless for Apache Spark Notebooks orDataproc Cluster Notebook section, click the PySpark kernel card.

Spark SQL magic: Since the PySpark kernel that launchesServerless for Apache Spark Notebooksis preloaded with Spark SQL magic, instead of usingspark.sql('SQL STATEMENT').show()to wrap your SQL statement, you can type%%sparksql magic at the top of a cell, then type your SQL statement in the cell.

BigQuery SQL: The BigQuery Spark connector allows yournotebook code to load data from BigQuery tables, perform analysisin Spark, and then write the results to a BigQuery table.

TheServerless for Apache Spark2.2and later runtimes include theBigQuery Spark connector.If you use earlier runtime to launch Serverless for Apache Spark notebooks,you can install Spark BigQuery Connector by adding the following Spark propertyto yourServerless for Apache Spark runtime template:

spark.jars: gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.25.2.jar

Scala code development

Dataproc on Compute Engine clusters created with imageversions2.0 and later includeApache Toree,a Scala kernel for the Jupyter Notebook platform that provides interactive accessto Spark.

• Click the Apache Toree card in theDataproc cluster Notebooksection on the JupyterLabLauncher page to open a notebook for Scalacode development.

  

Develop code with the Visual Studio Code extension

You can use the Google CloudVisual Studio Code (VS Code)extension lets you do the following:

• Develop and run Spark code in Serverless for Apache Spark notebooks.
• Create and manage Serverless for Apache Spark runtime (session) templates,interactive sessions, and batch workloads.

The Visual Studio Code extension is free, but you are charged for anyGoogle Cloud services, including Dataproc,Serverless for Apache Spark and Cloud Storage resources that you use.

Use VS Code with BigQuery: You can alsouse VS Code with BigQuery todo the following:

• Develop and execute BigQuery notebooks.
• Browse, inspect, and preview BigQuery datasets.

Before you begin

1. Download and install VS Code.
2. Open VS Code, and then in the activity bar, clickExtensions.

3. Using the search bar, find theJupyter extension, and then clickInstall. The Jupyter extension by Microsoft is a required dependency.

   

Install the Google Cloud extension

1. Open VS Code, and then in the activity bar, clickExtensions.

2. Using the search bar, find theGoogle Cloud Code extension, and thenclickInstall.

   

3. If prompted, restart VS Code.

TheGoogle Cloud Code icon is now visible in theVS Codeactivity bar.

Configure the extension

1. Open VS Code, and then in the activity bar, clickGoogle Cloud Code.
2. Open theDataproc section.
3. ClickLogin to Google Cloud. You are redirected to sign in with yourcredentials.
4. Use the top-level application taskbar to navigate toCode> Settings> Settings> Extensions.
5. FindGoogle Cloud Code, and click theManage icon to open the menu.
6. SelectSettings.
7. In theProject andDataproc Region fields,enter the name of the Google Cloud project and the regionto use to develop notebooks and manageServerless for Apache Spark resources.

Develop Serverless for Apache Spark notebooks

1. Open VS Code, and then in the activity bar, clickGoogle Cloud Code.
2. Open theNotebooks section, then clickNew Serverless Spark Notebook.
3. Select or create a new runtime (session) template to use for the notebook session.

4. A new.ipynb file containing sample code is created and opened in the editor.

   

   You can now write and execute code in your Serverless for Apache Spark notebook.

Create and manage Serverless for Apache Spark resources

1. Open VS Code, and then in the activity bar, clickGoogle Cloud Code.

2. Open theDataproc section, then click the followingresource names:

   • Clusters: Create and manage clusters and jobs.
   • Serverless: Create and manage batch workloads and interactive sessions.
   • Spark Runtime Templates: Create and manage session templates.

   

Dataset explorer

Use the JupyterLab Dataset explorer to viewBigLake metastore datasets.

To open the JupyterLab Dataset Explorer, click its icon in the sidebar.

You can search for a database, table, or column in the Dataset explorer.Click a database, table, or column name to view the associated metadata.

Deploy your code

Afterinstalling the Dataproc JupyterLab extension,you can use JupyterLab to:

• Execute your notebook code on the Google Cloud Serverless for Apache Spark infrastructure

• Schedule notebook execution on Cloud Composer

• Submit batch jobs to the Google Cloud Serverless for Apache Spark infrastructure or to yourDataproc on Compute Engine cluster.

Schedule notebook execution on Cloud Composer

Complete the following steps to schedule your notebook code on Cloud Composerto run as a batch job on Serverless for Apache Spark or on aDataproc on Compute Engine cluster.

1. Create a Cloud Composer environment.

2. Click theJob Scheduler button on top right of the notebook.

   

3. Fill in theCreate A Scheduled Job form to provide the following information:

   • A unique name for the notebook execution job
   • The Cloud Composer environment to use to deploy the notebook
   • Input parameters if the notebook is parameterized
   • The Dataproc cluster or serverless runtime template to userun the notebook
     • If a cluster is selected, whether to stop the cluster after the notebookfinishes executing on the cluster
   • Retry count and retry delay in minutes if notebook execution fails onthe first try
   • Execution notifications to send and the recipient list.Notifications are sent using an Airflow SMTP configuration.
   • The notebook execution schedule

4. ClickCreate.

5. After the notebook is successfully scheduled, the job name appearsin the list of scheduled jobs in the Cloud Composerenvironment.

   

Submit a batch job to Google Cloud Serverless for Apache Spark

• Click theServerless card in theDataproc Jobs and Sessions sectionon the JupyterLabLauncher page.

• Click theBatch tab, then clickCreate Batch and fill in theBatch Info fields.

• ClickSubmit to submit the job.

Submit a batch job to a Dataproc on Compute Engine cluster

• Click theClusters card in theDataproc Jobs and Sessions sectionon the JupyterLabLauncher page.

• Click theJobs tab, then clickSubmit Job.

• Select aCluster, then fill in theJob fields.

• ClickSubmit to submit the job.

View and manage resources

Afterinstalling the Dataproc JupyterLab extension,you can view and manage Google Cloud Serverless for Apache Spark and Dataproc on Compute Enginefrom theDataproc Jobs and Sessions section on the JupyterLabLauncher page.

Click theDataproc Jobs and Sessions section to show theClusters andServerless cards.

To view and manage Google Cloud Serverless for Apache Spark sessions:

1. Click theServerless card.
2. Click theSessions tab, then a session ID to open theSession details page to view session properties, view Google Cloud logs in Logs Explorer, and terminate a session. Note: A unique Google Cloud Serverless for Apache Spark session is created to launch each Google Cloud Serverless for Apache Spark notebook.

To view and manage Google Cloud Serverless for Apache Spark batches:

1. Click theBatches tab to view the list of Google Cloud Serverless for Apache Spark batches in the current project and region. Click a batch ID to view batch details.

To view and manage Dataproc on Compute Engine clusters:

1. Click theClusters card. TheClusters tab is selected to list active Dataproc on Compute Engine clusters in the current project and region. You can click the icons in theActions column to start, stop, or restart a cluster. Click a cluster name to view cluster details. You can click the icons in theActions column to clone, stop, or delete a job.

To view and manage Dataproc on Compute Engine jobs:

1. Click theJobs card to view the list of jobs in the current project. Click a job ID to view job details.