Dataproc optional Docker component

You can install additional components like Docker when you create a Dataproccluster using theOptional componentsfeature. This page describes the Docker component.

The Dataproc component installs aDocker daemon on each cluster node and creates a Linux user "docker" and a Linux group"docker" on each node to run the Docker daemon. This component also createsa "docker"systemd service to run thedockerd service. You should use thesystemd service to manage thelifecycle of the Docker service.

Install the component

Install the component when you create a Dataproc cluster.The Docker component can be installed on clusters created withDataprocimageversion 1.5or later.

SeeSupported Dataproc versionsfor the component version included in each Dataproc image release.

gcloud command

To create a Dataproc cluster that includes the Docker component,use thegcloud dataproc clusters createcluster-namecommand with the--optional-components flag.

gcloud dataproc clusters createcluster-name \    --optional-components=DOCKER \    --region=region \    --image-version=1.5 \    ... other flags

REST API

The Docker component can be specified through the Dataproc API usingSoftwareConfig.Componentas part of aclusters.createrequest.

Console

  1. Enable the component.
    • In the Google Cloud console, open the DataprocCreate a cluster page. The Set up cluster panel is selected.
    • In the Components section:
      • Under Optional components, select Docker and other optional components to install on your cluster.

Enable Docker on YARN

SeeCustomize your Spark job runtime environment with Docker on YARNto use a customized Docker image with YARN.

Docker Logging

By default, the Dataproc Docker component writes logs toCloud Logging bysetting thegcplogs driver—seeViewing your logs.

Docker Registry

The Dataproc Docker component configures Docker touse Container Registry in addition to the default Docker registries.Docker will use the Docker credential helper to authenticate withContainer Registry.

Use the Docker component on a Kerberos cluster

You can install the Docker optional component on a cluster that isbeing created withKerberos security enabled.

Docker is not part of the Hadoop ecosystem, and isn'trecognized by Hadoop services. If you run a container that communicates withHadoop services directly, your container must have the required Kerberos keytabfile and credential.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.