Dataproc clusters are built onCompute Engineinstances.Machine types define the virtualized hardwareresources available to an instance. Compute Engine offers bothpredefinedmachine types andcustom machine types.Dataproc clusters can use both predefined and custom types forboth master and worker nodes.

Dataproc clusters support the following Compute Engine predefinedmachine types (machine type availability varies byregion):

• General purpose machine types,which include N1, N2, N2D, E2, C3, C4, and N4 machine types (Dataprocalso supports N1, N2, N2D, E2, C3, C4, and N4custom machine types).

  Limitations:

  • the n1-standard-1 machine type isnot supported for 2.0+ images(the n1-standard-1 machine type isnot recommended for pre-2.0images—instead, use a machine type with higher memory).
  • Shared-core machine types arenot supported, whichinclude the following unsupported machine types:
    • E2: e2-micro, e2-small, and e2-medium shared-core machine types, and
    • N1: f1-micro and g1-small shared-core machine types.
  • Dataproc selectshyperdisk-balanced as the boot-disk type if machine type is C4 or N4.
• Compute-optimized machine types,which includeC2andC2D machine types.
• Memory-optimized machine types,which include M1 and M2 machine types.
• ARM machine types, which includeC4A machine types.

Custom machine types

Dataproc supportsN1 seriescustom machine types.

Custom machine types are ideal for the following workloads:

• Workloads that are not a good fit for the predefined machine types.
• Workloads that require more processing power or more memory, but don't needall of the upgrades that are provided by the next machine type level.

For example, if you have a workload that needs more processing power thanthat provided by ann1-standard-4 instance, but the next step up, ann1-standard-8instance, provides too much capacity. With custom machine types, you can create Dataproc clusters with master and/or worker nodes in the middlerange, with 6 virtual CPUs and 25 GB of memory.

Specify a custom machine type

Custom machine types use a specialmachine type specification and are subjecttolimitations. For example,the custom machine type specification for a custom VM with 6 virtual CPUs and22.5 GB of memory iscustom-6-23040.

The numbers in the machine type specification correspond to the number of virtual CPUs(vCPUs)in the machine (6) and the amount of memory (23040).The amount of memory is calculated by multiplying the amount of memory ingigabytes by1024 (seeExpressing memory in GB or MB). In this example, 22.5 (GB) is multiplied by 1024:22.5 * 1024 = 23040.

You use the above syntax to specify the custom machine type with your clusters.You can set the machine type for either master or worker nodes or both when youcreate a cluster. If you set both, the master node can use a custom machine typethat is different from the custom machine type used by workers. The machine typeused by any secondary workers follow the settings for primary workers andcannot be separately set (seeSecondary workers - preemptible and non-preemptible VMs).

Custom machine type pricing

Custom machine type pricingis based on the resources used in a custom machine.Dataproc pricing is added to the cost of compute resources, and is based on thetotal number of virtual CPUs (vCPUs) used in a cluster.

Create a Dataproc cluster with a specified machine type

gcloud dataproc clusters create test-cluster /    --master-machine-type custom-6-23040 /    --worker-machine-type custom-6-23040 /    other args

...properties:  distcp:mapreduce.map.java.opts: -Xmx1638m  distcp:mapreduce.map.memory.mb: '2048'  distcp:mapreduce.reduce.java.opts: -Xmx4915m  distcp:mapreduce.reduce.memory.mb: '6144'  mapred:mapreduce.map.cpu.vcores: '1'  mapred:mapreduce.map.java.opts: -Xmx1638m...

POST /v1/projects/my-project-id/regions/is-central1/clusters/{  "projectId": "my-project-id",  "clusterName": "test-cluster",  "config": {    "configBucket": "",    "gceClusterConfig": {      "subnetworkUri": "default",      "zoneUri": "us-central1-a"    },    "masterConfig": {      "numInstances": 1,"machineTypeUri": "n1-highmem-4",      "diskConfig": {        "bootDiskSizeGb": 500,        "numLocalSsds": 0      }    },    "workerConfig": {      "numInstances": 2,"machineTypeUri": "n1-highmem-4",      "diskConfig": {        "bootDiskSizeGb": 500,        "numLocalSsds": 0      }    }  }}

Create a Dataproc cluster with custom machine type with extended memory

Dataproc supports custom machine types withextended memorybeyond the6.5GB per vCPU limit(seeExtended Memory Pricing).

Console

ClickExtend memory when customizing Machine type memory in theMaster node and/or Worker nodes section from theConfigure nodes panelon the DataprocCreate a clusterpage in the Google Cloud console.

gcloud Command

To create a cluster from the gcloud command line with custom CPUs with extended memory, add a-ext suffix to the‑‑master-machine-type and/or‑‑worker-machine-type flags.

Example

The following gcloud command-line sample creates aDataproc cluster with 1 CPU and 50 GB memory (50 * 1024 = 51200) in eachnode:

gcloud dataproc clusters create test-cluster /    --master-machine-type custom-1-51200-ext /    --worker-machine-type custom-1-51200-ext /    other args

API

The following sampleJSON snippet from a Dataproc REST APIclusters.createrequest specifies 1 CPU and 50 GB memory (50 * 1024 = 51200) in each node:

...    "masterConfig": {      "numInstances": 1,      "machineTypeUri": "custom-1-51200-ext",    ...    },    "workerConfig": {      "numInstances": 2,      "machineTypeUri": "custom-1-51200-ext",     ......

ARM machine types

Dataproc supports creating a cluster with nodes that useARM machine types,such as theC4A machine type.

Requirements and limitations:

• The Dataproc image must be compatible with ARM chipset.The Dataproc2.1-ubuntu20-arm,2.2-ubuntu22-arm,and2.3-ubuntu22-arm(and later-arm suffix) images are compatible with the ARM chipset. ARM-compatibleimages don't support many optional and initialization-actioncomponents as noted on theimage release version pages.
• Since one image must be specified for a cluster, the master,worker, and secondary-worker nodes must use an ARM machine type thatis compatible with the selected Dataproc ARM image.
• Dataproc features that are not compatible with ARM machine typesaren't available (for example,local SSDsaren't supported by C4A machine types).
• ARM images only support pre-installed components and a limited set ofoptional components. Other optional components and all initializationactions are unsupported.

Create a Dataproc cluster with an ARM machine type

gcloud dataproc clusters createcluster-name \    --region=REGION \    --image-version=2.1-ubuntu20-arm \    --master-machine-type=c4a-standard-4 \    --worker-machine-type=c4a-standard-4

POST /v1/projects/my-project-id/regions/is-central1/clusters/{  "projectId": "my-project-id",  "clusterName": "sample-cluster",  "config": {    "configBucket": "",    "gceClusterConfig": {      "subnetworkUri": "default",      "zoneUri": "us-central1-a"    },    "masterConfig": {      "numInstances": 1,"machineTypeUri": "c4a-standard-4",      "diskConfig": {        "bootDiskSizeGb": 500,      }    },    "workerConfig": {      "numInstances": 2,"machineTypeUri": "c4a-standard-4",      "diskConfig": {        "bootDiskSizeGb": 500,        "numLocalSsds": 0      }    },    "softwareConfig": {"imageVersion": "2.1-ubuntu20-arm"    }  }}

What's next

• Learn more aboutArm VMs on Compute.
• Learn how tocreate a VM with a custom machine type.
• Learn how tocreate and start a Compute Engine instance.