Objectives

Write a simple wordcount Spark job in Java, Scala, orPython, then run the job on a Dataproc cluster.

Costs

In this document, you use the following billable components of Google Cloud:

• Compute Engine
• Dataproc
• Cloud Storage

To generate a cost estimate based on your projected usage, use thepricing calculator.

Before you begin

Run the steps below to prepare to run the code in this tutorial.

1. Set up your project. If necessary, set up a project with the Dataproc, Compute Engine,and Cloud Storage APIs enabled and the Google Cloud CLI installed on your local machine.

   Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

   In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

   Verify that billing is enabled for your Google Cloud project.

   Enable the Dataproc, Compute Engine, and Cloud Storage APIs.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

   Enable the APIs

   Create a service account:

   1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin).Learn how to grant roles.

   2. In the Google Cloud console, go to theCreate service account page.

      Go to Create service account
   3. Select your project.

   4. In theService account name field, enter a name. The Google Cloud console fills in theService account ID field based on this name.

      In theService account description field, enter a description. For example,Service account for quickstart.

   5. ClickCreate and continue.

   6. Grant theProject > Owner role to the service account.

      To grant the role, find theSelect a role list, then selectProject > Owner.

      Note: TheRole field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant apredefined role orcustom role that meets your needs.
   7. ClickContinue.

   8. ClickDone to finish creating the service account.

      Do not close your browser window. You will use it in the next step.

   Create a service account key:

   1. In the Google Cloud console, click the email address for the service account that you created.
   2. ClickKeys.
   3. ClickAdd key, and then clickCreate new key.
   4. ClickCreate. A JSON key file is downloaded to your computer.
   5. ClickClose.

   Set the environment variableGOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your credentials. This variable applies only to your current shell session, so if you open a new session, set the variable again.

   Example: Linux or macOS

   exportGOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   For example:

   exportGOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"

   Example: Windows

   For PowerShell:

   $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   For example:

   $env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"

   For command prompt:

   setGOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   Install the Google Cloud CLI.

   If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

   Toinitialize the gcloud CLI, run the following command:

   gcloudinit

   In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

   Verify that billing is enabled for your Google Cloud project.

   Enable the Dataproc, Compute Engine, and Cloud Storage APIs.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

   Enable the APIs

   Create a service account:

   1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin).Learn how to grant roles.

   2. In the Google Cloud console, go to theCreate service account page.

      Go to Create service account
   3. Select your project.

   4. In theService account name field, enter a name. The Google Cloud console fills in theService account ID field based on this name.

      In theService account description field, enter a description. For example,Service account for quickstart.

   5. ClickCreate and continue.

   6. Grant theProject > Owner role to the service account.

      To grant the role, find theSelect a role list, then selectProject > Owner.

      Note: TheRole field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant apredefined role orcustom role that meets your needs.
   7. ClickContinue.

   8. ClickDone to finish creating the service account.

      Do not close your browser window. You will use it in the next step.

   Create a service account key:

   1. In the Google Cloud console, click the email address for the service account that you created.
   2. ClickKeys.
   3. ClickAdd key, and then clickCreate new key.
   4. ClickCreate. A JSON key file is downloaded to your computer.
   5. ClickClose.

   Set the environment variableGOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your credentials. This variable applies only to your current shell session, so if you open a new session, set the variable again.

   Example: Linux or macOS

   exportGOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   For example:

   exportGOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"

   Example: Windows

   For PowerShell:

   $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   For example:

   $env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"

   For command prompt:

   setGOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

   ReplaceKEY_PATH with the path of the JSON file that contains your credentials.

   Install the Google Cloud CLI.

   If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

   Toinitialize the gcloud CLI, run the following command:

   gcloudinit

2. Create a Cloud Storage bucket. You need a Cloud Storage to hold tutorial data. If you do not have oneready to use, create a new bucket in your project.

   1. In the Google Cloud console, go to the Cloud StorageBuckets page.

      Go to Buckets

   2. Clickadd_boxCreate.
   3. On theCreate a bucket page, enter your bucket information. To go to the next step, clickContinue.
      1. In theGet started section, do the following:
         • Enter a globally unique name that meets thebucket naming requirements.
         • To add abucket label, expand theLabels section (expand_more), clickadd_boxAdd label, and specify akey and avalue for your label.
      2. In theChoose where to store your data section, do the following:
         1. Select aLocation type.
         2. Choose a location where your bucket's data is permanently stored from theLocation type drop-down menu.
            • If you select thedual-region location type, you can also choose to enableturbo replication by using the relevant checkbox.
         3. To set upcross-bucket replication, selectAdd cross-bucket replication via Storage Transfer Service and follow these steps:

            Set up cross-bucket replication

            1. In theBucket menu, select a bucket.

            2. In theReplication settings section, clickConfigure to configure settings for the replication job.

               TheConfigure cross-bucket replication pane appears.

               • To filter objects to replicate by object name prefix, enter a prefix that you want to include or exclude objects from, then clickaddAdd a prefix.
               • To set a storage class for the replicated objects, select a storage class from theStorage class menu. If you skip this step, the replicated objects will use the destination bucket's storage class by default.
               • ClickDone.
      3. In theChoose how to store your data section, do the following:
         1. Select adefault storage class for the bucket orAutoclass for automatic storage class management of your bucket's data.
         2. To enablehierarchical namespace, in theOptimize storage for data-intensive workloads section, selectEnable hierarchical namespace on this bucket.Note: You cannot enable hierarchical namespace in existing buckets.
      4. In theChoose how to control access to objects section, select whether or not your bucket enforcespublic access prevention, and select anaccess control method for your bucket's objects.Note: You cannot change thePrevent public access setting if this setting is enforced at anorganization policy.
      5. In theChoose how to protect object data section, do the following:
         • Select any of the options underData protection that you want to set for your bucket.
           • To enablesoft delete, click theSoft delete policy (For data recovery) checkbox, and specify the number of days you want to retain objects after deletion.
           • To setObject Versioning, click theObject versioning (For version control) checkbox, and specify the maximum number of versions per object and the number of days after which the noncurrent versions expire.
           • To enable the retention policy on objects and buckets, click theRetention (For compliance) checkbox, and then do the following:
             • To enableObject Retention Lock, click theEnable object retention checkbox.
             • To enableBucket Lock, click theSet bucket retention policy checkbox, and choose a unit of time and a length of time for your retention period.
         • To choose how your object data will be encrypted, expand theData encryption section (expand_more), and select aData encryption method.
   4. ClickCreate.

3. Set local environment variables. Set environment variables on yourlocal machine. Set your Google Cloud project-id and the name of theCloud Storage bucket you will use for this tutorial. Also provide thename andregionof an existing or new Dataproc cluster.You can create a cluster to use in this tutorial in the next step.

   PROJECT=project-id

   BUCKET_NAME=bucket-name

   CLUSTER=cluster-name

   REGION=cluster-region Example: "us-central1"

4. Create a Dataproc cluster. Run the command, below, tocreate asingle-nodeDataproc cluster in the specifiedCompute Engine zone.

   gcloud dataproc clusters create ${CLUSTER} \    --project=${PROJECT} \    --region=${REGION} \    --single-node

   The above command installs the defaultcluster image version.You can use the--image-versionflag to select an image version for your cluster. Each image version installsspecific versions of Spark and Scala library components. If youprepare the Spark wordcount jobin Java or Scala, you will reference the Spark and Scala versions installedon your cluster when you prepare the job package.

5. Copy public data to your Cloud Storage bucket. Copy a public dataShakespeare text snippet into theinput folder of yourCloud Storage bucket:

   gcloud storage cp gs://pub/shakespeare/rose.txt \    gs://${BUCKET_NAME}/input/rose.txt

6. Set up aJava (Apache Maven),Scala (SBT), orPythondevelopment environment.Use Cloud Shell.Cloud Shell includestools used in this tutorial, including Apache Maven, Python,and the Google Cloud CLI.

Clean up

After you finish the tutorial, you can clean up the resources that you created so that they stop using quota and incurring charges. The following sections describe how to delete or turn off these resources.

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as anappspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to theManage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then clickDelete.
3. In the dialog, type the project ID, and then clickShut down to delete the project.

Delete the Dataproc cluster

Instead of deleting your project, you might want to onlydelete your cluster within the project.

Delete the Cloud Storage bucket

What's next

• SeeSpark job tuning tips