This document describes the requirements needed for Google Cloud Serverless for Apache Sparknetwork configuration.

Virtual Private Cloud subnetwork requirements

This document explains the Virtual Private Cloud network requirements forGoogle Cloud Serverless for Apache Spark batch workloads and interactive sessions.

Private Google Access

Serverless for Apache Spark batch workloads and interactive sessionsrun on VMs with internal IP addresses only and on a regional subnet withPrivate Google Access (PGA)automatically enabled on the subnet.

If you don't specify a subnet, Serverless for Apache Spark selects thedefault subnet in the batch workload or session region as the subnet for abatch workload or session.

If your workload requires external network or internetaccess, for example to download resources such as ML models fromPyTorch Hub orHugging Face,you can set upCloud NAT to allow outbound trafficusing internal IPs on your VPC network.

Open subnet connectivity

The VPC subnet for the region selected for theServerless for Apache Spark batch workload or interactive session mustallow internal subnet communication on all ports between VM instances.

The following Google Cloud CLI command attaches a network firewall to asubnet that allows internal ingress communications among VMs using all protocolson all ports:

gcloud compute firewall-rules create allow-internal-ingress \    --network=NETWORK_NAME \    --source-ranges=SUBNET_RANGES \    --destination-ranges=SUBNET_RANGES \    --direction=ingress \    --action=allow \    --rules=all

Notes:

• SUBNET_RANGES: SeeAllow internal ingress connections between VMs.Thedefault VPC network in a project with thedefault-allow-internal firewall rule, which allows ingress communication onall ports (tcp:0-65535,udp:0-65535, andicmp protocols:ports),meets the open-subnet-connectivity requirement. However, this rule also allowsingress by any VM instance on the network.

  Use network tags to limit connectivity.In production, the recommended practice is to limitfirewall rules to the IP addresses used by your Spark workloads.

Serverless for Apache Spark and VPC-SC networks

WithVPC Service Controls,network administrators can define a security perimeter around resources ofGoogle-managed services to control communication to and between those services.

Note the following strategies when using VPC-SCnetworks with Serverless for Apache Spark:

• Set up private connectivity.

• Create acustom container imagethat pre-installs dependencies outside the VPC-SC perimeter, and thensubmit a Spark batch workloadthat uses your custom container image.

For more information, seeVPC Service Controls—Serverless for Apache Spark.