When you enable Dataproc cluster caching, the cluster cachesCloud Storage data frequently accessed by your Spark jobs.

Benefits

• Improved performance: Caching can improve job performance by reducing the amountof time spent retrieving data from storage.
• Reduced storage costs: Since hot data is cached on local disk,fewer API calls are made to storage to retrieve data.
• Spark job applicability: When cluster caching is enabled on a cluster,it applies to all Spark jobs run on the cluster, whether submitted to theDataproc service or run independently on the cluster.

Limitations and requirements

• Caching applies to Dataproc Spark jobs only.
• Only Cloud Storage data is cached.
• Caching only applies to clusters that meet the following requirements:
  • The cluster has one master andn workers(High Availability (HA) andsingle node clusters are not supported).
  • This feature is available in Dataproc on Compute Engineimage versions2.0.72+,2.1.20+, and2.2.0+.
  • Each cluster node must havelocal SSDsattached with theNVME (Non-Volatile Memory Express)interface (Persistent Disks (PDs) are not supported). Data is cached on NVMElocal SSDs only.
  • The cluster uses thedefault VM service accountfor authentication.Custom VM service accountsare not supported.

Enable cluster caching

You can enable cluster caching when you create a Dataproc clusterusing the Google Cloud console, Google Cloud CLI, or the Dataproc API.

gcloud dataproc clusters createCLUSTER_NAME \    --region=REGION \    --properties dataproc:dataproc.cluster.caching.enabled=true \    --num-master-local-ssds=2 \    --master-local-ssd-interface=NVME \    --num-worker-local-ssds=2 \    --worker-local-ssd-interface=NVME \    other args ...