Volumes

Kubernetesvolumes provide a way for containers in apodto access and share data via the filesystem. There are different kinds of volume that you can use for different purposes,such as:

• populating a configuration file based on aConfigMapor aSecret
• providing some temporary scratch space for a pod
• sharing a filesystem between two different containers in the same pod
• sharing a filesystem between two different pods (even if those Pods run on different nodes)
• durably storing data so that it stays available even if the Pod restarts or is replaced
• passing configuration information to an app running in a container, based on details of the Podthe container is in(for example: telling asidecar containerwhat namespace the Pod is running in)
• providing read-only access to data in a different container image

Data sharing can be between different local processes within a container, or between different containers,or between Pods.

Why volumes are important

• Data persistence: On-disk files in a container are ephemeral, which presents some problems fornon-trivial applications when running in containers. One problem occurs whena container crashes or is stopped, the container state is not saved so all of thefiles that were created or modified during the lifetime of the container are lost.After a crash, kubelet restarts the container with a clean state.

• Shared storage: Another problem occurs when multiple containers are running in aPod andneed to share files. It can be challenging to set upand access a shared filesystem across all of the containers.

The Kubernetesvolume abstractioncan help you to solve both of these problems.

Before you learn about volumes, PersistentVolumes and PersistentVolumeClaims, you should read upaboutPods and make sure that you understand howKubernetes uses Pods to run containers.

How volumes work

Kubernetes supports many types of volumes. APodcan use any number of volume types simultaneously.Ephemeral volume types have a lifetime linked to a specific Pod,butpersistent volumes exist beyondthe lifetime of any individual pod. When a pod ceases to exist, Kubernetes destroys ephemeral volumes;however, Kubernetes does not destroy persistent volumes.For any kind of volume in a given pod, data is preserved across container restarts.

At its core, a volume is a directory, possibly with some data in it, whichis accessible to the containers in a pod. How that directory comes to be, themedium that backs it, and the contents of it are determined by the particularvolume type used.

To use a volume, specify the volumes to provide for the Pod in.spec.volumesand declare where to mount those volumes into containers in.spec.containers[*].volumeMounts.

When a pod is launched, a process in the container sees a filesystem view composed from the initial contents ofthecontainer image, plus volumes(if defined) mounted inside the container.The process sees a root filesystem that initially matches the contents of the container image.Any writes to within that filesystem hierarchy, if allowed, affect what that process viewswhen it performs a subsequent filesystem access.Volumes are mounted atspecified paths within the container filesystem.For each container defined within a Pod, you must independently specify whereto mount each volume that the container uses.

Volumes cannot mount within other volumes (but seeUsing subPathfor a related mechanism). Also, a volume cannot contain a hard link to anything ina different volume.

Types of volumes

Kubernetes supports several types of volumes.

awsElasticBlockStore (deprecated)

In Kubernetes 1.33, all operations for the in-treeawsElasticBlockStore typeare redirected to theebs.csi.aws.comCSI driver.

The AWSElasticBlockStore in-tree storage driver was deprecated in the Kubernetes v1.19 releaseand then removed entirely in the v1.27 release.

The Kubernetes project suggests that you use theAWS EBSthird party storage driver instead.

azureDisk (deprecated)

In Kubernetes 1.33, all operations for the in-treeazureDisk typeare redirected to thedisk.csi.azure.comCSI driver.

The AzureDisk in-tree storage driver was deprecated in the Kubernetes v1.19 releaseand then removed entirely in the v1.27 release.

The Kubernetes project suggests that you use theAzure Diskthird party storage driver instead.

azureFile (deprecated)

In Kubernetes 1.33, all operations for the in-treeazureFile typeare redirected to thefile.csi.azure.comCSI driver.

The AzureFile in-tree storage driver was deprecated in the Kubernetes v1.21 releaseand then removed entirely in the v1.30 release.

The Kubernetes project suggests that you use theAzure Filethird party storage driver instead.

cephfs (removed)

Kubernetes 1.33 does not include acephfs volume type.

Thecephfs in-tree storage driver was deprecated in the Kubernetes v1.28release and then removed entirely in the v1.31 release.

cinder (deprecated)

In Kubernetes 1.33, all operations for the in-treecinder typeare redirected to thecinder.csi.openstack.orgCSI driver.

The OpenStack Cinder in-tree storage driver was deprecated in the Kubernetes v1.11 releaseand then removed entirely in the v1.26 release.

The Kubernetes project suggests that you use theOpenStack Cinderthird party storage driver instead.

configMap

AConfigMapprovides a way to inject configuration data into pods.The data stored in a ConfigMap can be referenced in a volume of typeconfigMap and then consumed by containerized applications running in a pod.

When referencing a ConfigMap, you provide the name of the ConfigMap in thevolume. You can customize the path to use for a specificentry in the ConfigMap. The following configuration shows how to mountthelog-config ConfigMap onto a Pod calledconfigmap-pod:

apiVersion:v1kind:Podmetadata:name:configmap-podspec:containers:-name:testimage:busybox:1.28command:['sh','-c','echo "The app is running!" && tail -f /dev/null']volumeMounts:-name:config-volmountPath:/etc/configvolumes:-name:config-volconfigMap:name:log-configitems:-key:log_levelpath:log_level.conf

Thelog-config ConfigMap is mounted as a volume, and all contents stored initslog_level entry are mounted into the Pod at path/etc/config/log_level.conf.Note that this path is derived from the volume'smountPath and thepathkeyed withlog_level.

downwardAPI

AdownwardAPI volume makesdownward APIdata available to applications. Within the volume, you can find the exposeddata as read-only files in plain text format.

SeeExpose Pod Information to Containers Through Filesto learn more.

emptyDir

For a Pod that defines anemptyDir volume, the volume is created when the Pod is assigned to a node.As the name says, theemptyDir volume is initially empty. All containers in the Pod can read and write the samefiles in theemptyDir volume, though that volume can be mounted at the sameor different paths in each container. When a Pod is removed from a node forany reason, the data in theemptyDir is deleted permanently.

Some uses for anemptyDir are:

• scratch space, such as for a disk-based merge sort
• checkpointing a long computation for recovery from crashes
• holding files that a content-manager container fetches while a webservercontainer serves the data

TheemptyDir.medium field controls whereemptyDir volumes are stored. BydefaultemptyDir volumes are stored on whatever medium that backs the nodesuch as disk, SSD, or network storage, depending on your environment. If you settheemptyDir.medium field to"Memory", Kubernetes mounts a tmpfs (RAM-backedfilesystem) for you instead. While tmpfs is very fast, be aware that, unlikedisks, files you write count against the memory limit of the container that wrote them.

A size limit can be specified for the default medium, which limits the capacityof theemptyDir volume. The storage is allocated fromnode ephemeral storage.If that is filled up from another source (for example, log files or image overlays),theemptyDir may run out of capacity before this limit.If no size is specified, memory-backed volumes are sized to node allocatable memory.

emptyDir configuration example

apiVersion:v1kind:Podmetadata:name:test-pdspec:containers:-image:registry.k8s.io/test-webservername:test-containervolumeMounts:-mountPath:/cachename:cache-volumevolumes:-name:cache-volumeemptyDir:sizeLimit:500Mi

emptyDir memory configuration example

apiVersion:v1kind:Podmetadata:name:test-pdspec:containers:-image:registry.k8s.io/test-webservername:test-containervolumeMounts:-mountPath:/cachename:cache-volumevolumes:-name:cache-volumeemptyDir:sizeLimit:500Mimedium:Memory

fc (fibre channel)

Anfc volume type allows an existing fibre channel block storage volumeto be mounted in a Pod. You can specify single or multiple target world wide names (WWNs)using the parametertargetWWNs in your Volume configuration. If multiple WWNs are specified,targetWWNs expect that those WWNs are from multi-path connections.

See thefibre channel examplefor more details.

gcePersistentDisk (deprecated)

In Kubernetes 1.33, all operations for the in-treegcePersistentDisk typeare redirected to thepd.csi.storage.gke.ioCSI driver.

ThegcePersistentDisk in-tree storage driver was deprecated in the Kubernetes v1.17 releaseand then removed entirely in the v1.28 release.

The Kubernetes project suggests that you use theGoogle Compute Engine Persistent Disk CSIthird party storage driver instead.

gitRepo (deprecated)

!has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))

You can use this deprecated storage plugin in your cluster if you explicitlyenable theGitRepoVolumeDriverfeature gate.

AgitRepo volume is an example of a volume plugin. This pluginmounts an empty directory and clones a git repository into this directoryfor your Pod to use.

Here is an example of agitRepo volume:

apiVersion:v1kind:Podmetadata:name:serverspec:containers:-image:nginxname:nginxvolumeMounts:-mountPath:/mypathname:git-volumevolumes:-name:git-volumegitRepo:repository:"git@somewhere:me/my-git-repository.git"revision:"22f1d8406d464b0c0874075539c1f2e96c253775"

glusterfs (removed)

Kubernetes 1.33 does not include aglusterfs volume type.

The GlusterFS in-tree storage driver was deprecated in the Kubernetes v1.25 releaseand then removed entirely in the v1.26 release.

hostPath

AhostPath volume mounts a file or directory from the host node's filesysteminto your Pod. This is not something that most Pods will need, but it offers apowerful escape hatch for some applications.

Some uses for ahostPath are:

• running a container that needs access to node-level system components(such as a container that transfers system logs to a central location,accessing those logs using a read-only mount of/var/log)
• making a configuration file stored on the host system available read-onlyto astatic pod;unlike normal Pods, static Pods cannot access ConfigMaps

hostPath volume types

In addition to the requiredpath property, you can optionally specify atype for ahostPath volume.

The available values fortype are:

Some files or directories created on the underlying hosts might only beaccessible by root. You then either need to run your process as root in aprivileged containeror modify the file permissions on the host to read from or write to ahostPath volume.

hostPath configuration example

• Linux node
• Windows node

---# This manifest mounts /data/foo on the host as /foo inside the# single container that runs within the hostpath-example-linux Pod.## The mount into the container is read-only.apiVersion:v1kind:Podmetadata:name:hostpath-example-linuxspec:os:{name:linux }nodeSelector:kubernetes.io/os:linuxcontainers:-name:example-containerimage:registry.k8s.io/test-webservervolumeMounts:-mountPath:/fooname:example-volumereadOnly:truevolumes:-name:example-volume# mount /data/foo, but only if that directory already existshostPath:path:/data/foo# directory location on hosttype:Directory# this field is optional

---# This manifest mounts C:\Data\foo on the host as C:\foo, inside the# single container that runs within the hostpath-example-windows Pod.## The mount into the container is read-only.apiVersion:v1kind:Podmetadata:name:hostpath-example-windowsspec:os:{name:windows }nodeSelector:kubernetes.io/os:windowscontainers:-name:example-containerimage:microsoft/windowsservercore:1709volumeMounts:-name:example-volumemountPath:"C:\\foo"readOnly:truevolumes:# mount C:\Data\foo from the host, but only if that directory already exists-name:example-volumehostPath:path:"C:\\Data\\foo"# directory location on hosttype:Directory# this field is optional

hostPath FileOrCreate configuration example

The following manifest defines a Pod that mounts/var/local/aaainside the single container in the Pod. If the node does notalready have a path/var/local/aaa, the kubelet createsit as a directory and then mounts it into the Pod.

If/var/local/aaa already exists but is not a directory,the Pod fails. Additionally, the kubelet attempts to makea file named/var/local/aaa/1.txt inside that directory(as seen from the host); if something already exists atthat path and isn't a regular file, the Pod fails.

Here's the example manifest:

apiVersion:v1kind:Podmetadata:name:test-webserverspec:os:{name:linux }nodeSelector:kubernetes.io/os:linuxcontainers:-name:test-webserverimage:registry.k8s.io/test-webserver:latestvolumeMounts:-mountPath:/var/local/aaaname:mydir-mountPath:/var/local/aaa/1.txtname:myfilevolumes:-name:mydirhostPath:# Ensure the file directory is created.path:/var/local/aaatype:DirectoryOrCreate-name:myfilehostPath:path:/var/local/aaa/1.txttype:FileOrCreate

image

Animage volume source represents an OCI object (a container image orartifact) which is available on the kubelet's host machine.

An example of using theimage volume source is:

pods/image-volumes.yaml

apiVersion:v1kind:Podmetadata:name:image-volumespec:containers:-name:shellcommand:["sleep","infinity"]image:debianvolumeMounts:-name:volumemountPath:/volumevolumes:-name:volumeimage:reference:quay.io/crio/artifact:v2pullPolicy:IfNotPresent

The volume is resolved at pod startup depending on whichpullPolicy value isprovided:

Always

the kubelet always attempts to pull the reference. If the pull fails,the kubelet sets the Pod toFailed.

Never

the kubelet never pulls the reference and only uses a local image or artifact.The Pod becomesFailed if any layers of the image aren't already present locally,or if the manifest for that image isn't already cached.

IfNotPresent

the kubelet pulls if the reference isn't already present on disk. The Pod becomesFailed if the reference isn't present and the pull fails.

The volume gets re-resolved if the pod gets deleted and recreated, which meansthat new remote content will become available on pod recreation. A failure toresolve or pull the image during pod startup will block containers from startingand may add significant latency. Failures will be retried using normal volumebackoff and will be reported on the pod reason and message.

The types of objects that may be mounted by this volume are defined by thecontainer runtime implementation on a host machine. At a minimum, they must includeall valid types supported by the container image field. The OCI object getsmounted in a single directory (spec.containers[*].volumeMounts.mountPath)and will be mounted read-only. On Linux, the container runtime typically also mounts thevolume with file execution blocked (noexec).

Besides that:

• subPath orsubPathExprmounts for containers (spec.containers[*].volumeMounts.[subPath,subPathExpr])are only supported from Kubernetes v1.33.
• The fieldspec.securityContext.fsGroupChangePolicy has no effect on thisvolume type.
• TheAlwaysPullImages Admission Controllerdoes also work for this volume source like for container images.

The following fields are available for theimage type:

reference

Artifact reference to be used. For example, you could specifyregistry.k8s.io/conformance:v1.33.0 to load thefiles from the Kubernetes conformance test image. Behaves in the same way aspod.spec.containers[*].image. Pull secrets will be assembled in the same wayas for the container image by looking up node credentials, service account imagepull secrets, and pod spec image pull secrets. This field is optional to allowhigher level config management to default or override container images inworkload controllers like Deployments and StatefulSets.More info about container images

pullPolicy

Policy for pulling OCI objects. Possible values are:Always,Never orIfNotPresent. Defaults toAlways if:latest tag is specified, orIfNotPresent otherwise.

See theUse an Image Volume With a Podexample for more details on how to use the volume source.

iscsi

Aniscsi volume allows an existing iSCSI (SCSI over IP) volume to be mountedinto your Pod. UnlikeemptyDir, which is erased when a Pod is removed, thecontents of aniscsi volume are preserved and the volume is merelyunmounted. This means that an iscsi volume can be pre-populated with data, andthat data can be shared between pods.

A feature of iSCSI is that it can be mounted as read-only by multiple consumerssimultaneously. This means that you can pre-populate a volume with your datasetand then serve it in parallel from as many Pods as you need. Unfortunately,iSCSI volumes can only be mounted by a single consumer in read-write mode.Simultaneous writers are not allowed.

See theiSCSI example for more details.

local

Alocal volume represents a mounted local storage device such as a disk,partition or directory.

Local volumes can only be used as a statically created PersistentVolume. Dynamicprovisioning is not supported.

Compared tohostPath volumes,local volumes are used in a durable andportable manner without manually scheduling pods to nodes. The system is awareof the volume's node constraints by looking at the node affinity on the PersistentVolume.

However,local volumes are subject to the availability of the underlyingnode and are not suitable for all applications. If a node becomes unhealthy,then thelocal volume becomes inaccessible to the pod. The pod using this volumeis unable to run. Applications usinglocal volumes must be able to tolerate thisreduced availability, as well as potential data loss, depending on thedurability characteristics of the underlying disk.

The following example shows a PersistentVolume using alocal volume andnodeAffinity:

apiVersion:v1kind:PersistentVolumemetadata:name:example-pvspec:capacity:storage:100GivolumeMode:FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy:DeletestorageClassName:local-storagelocal:path:/mnt/disks/ssd1nodeAffinity:required:nodeSelectorTerms:-matchExpressions:-key:kubernetes.io/hostnameoperator:Invalues:- example-node

You must set a PersistentVolumenodeAffinity when usinglocal volumes.The Kubernetes scheduler uses the PersistentVolumenodeAffinity to schedulethese Pods to the correct node.

PersistentVolumevolumeMode can be set to "Block" (instead of the defaultvalue "Filesystem") to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass withvolumeBindingMode set toWaitForFirstConsumer. For more details, see thelocalStorageClass example.Delaying volume binding ensures that the PersistentVolumeClaim binding decisionwill also be evaluated with any other node constraints the Pod may have,such as node resource requirements, node selectors, Pod affinity, and Pod anti-affinity.

An external static provisioner can be run separately for improved management ofthe local volume lifecycle. Note that this provisioner does not support dynamicprovisioning yet. For an example on how to run an external local provisioner, see thelocal volume provisioner user guide.

nfs

Annfs volume allows an existing NFS (Network File System) share to bemounted into a Pod. UnlikeemptyDir, which is erased when a Pod isremoved, the contents of annfs volume are preserved and the volume is merelyunmounted. This means that an NFS volume can be pre-populated with data, andthat data can be shared between pods. NFS can be mounted by multiplewriters simultaneously.

apiVersion:v1kind:Podmetadata:name:test-pdspec:containers:-image:registry.k8s.io/test-webservername:test-containervolumeMounts:-mountPath:/my-nfs-dataname:test-volumevolumes:-name:test-volumenfs:server:my-nfs-server.example.compath:/my-nfs-volumereadOnly:true

See theNFS examplefor an example of mounting NFS volumes with PersistentVolumes.

persistentVolumeClaim

ApersistentVolumeClaim volume is used to mount aPersistentVolume into a Pod. PersistentVolumeClaimsare a way for users to "claim" durable storage (such as an iSCSI volume)without knowing the details of the particular cloud environment.

See the information aboutPersistentVolumes for moredetails.

portworxVolume (deprecated)

AportworxVolume is an elastic block storage layer that runs hyperconverged withKubernetes.Portworx fingerprints storagein a server, tiers based on capabilities, and aggregates capacity across multiple servers.Portworx runs in-guest in virtual machines or on bare metal Linux nodes.

AportworxVolume can be dynamically created through Kubernetes or it can alsobe pre-provisioned and referenced inside a Pod.Here is an example Pod referencing a pre-provisioned Portworx volume:

apiVersion:v1kind:Podmetadata:name:test-portworx-volume-podspec:containers:-image:registry.k8s.io/test-webservername:test-containervolumeMounts:-mountPath:/mntname:pxvolvolumes:-name:pxvol# This Portworx volume must already exist.portworxVolume:volumeID:"pxvol"fsType:"<fs-type>"

For more details, see thePortworx volume examples.

Portworx CSI migration

In Kubernetes 1.33, all operations for the in-treePortworx volumes are redirected to thepxd.portworx.comContainer Storage Interface (CSI) Driver by default.
Portworx CSI Drivermust be installed on the cluster.

projected

A projected volume maps several existing volume sources into the samedirectory. For more details, seeprojected volumes.

rbd (removed)

Kubernetes 1.33 does not include arbd volume type.

TheRados Block Device (RBD) in-tree storage driverand its csi migration support were deprecated in the Kubernetes v1.28 releaseand then removed entirely in the v1.31 release.

secret

Asecret volume is used to pass sensitive information, such as passwords, toPods. You can store secrets in the Kubernetes API and mount them as files foruse by pods without coupling to Kubernetes directly.secret volumes arebacked by tmpfs (a RAM-backed filesystem) so they are never written tonon-volatile storage.

For more details, seeConfiguring Secrets.

vsphereVolume (deprecated)

In Kubernetes 1.33, all operations for the in-treevsphereVolume typeare redirected to thecsi.vsphere.vmware.comCSI driver.

ThevsphereVolume in-tree storage driver was deprecated in the Kubernetes v1.19 releaseand then removed entirely in the v1.30 release.

The Kubernetes project suggests that you use thevSphere CSIthird party storage driver instead.

Using subPath

Sometimes, it is useful to share one volume for multiple uses in a single pod.ThevolumeMounts[*].subPath property specifies a sub-path inside the referenced volumeinstead of its root.

The following example shows how to configure a Pod with a LAMP stack (Linux Apache MySQL PHP)using a single, shared volume. This samplesubPath configuration is not recommendedfor production use.

The PHP application's code and assets map to the volume'shtml folder andthe MySQL database is stored in the volume'smysql folder. For example:

apiVersion:v1kind:Podmetadata:name:my-lamp-sitespec:containers:-name:mysqlimage:mysqlenv:-name:MYSQL_ROOT_PASSWORDvalue:"rootpasswd"volumeMounts:-mountPath:/var/lib/mysqlname:site-datasubPath:mysql-name:phpimage:php:7.0-apachevolumeMounts:-mountPath:/var/www/htmlname:site-datasubPath:htmlvolumes:-name:site-datapersistentVolumeClaim:claimName:my-lamp-site-data

Using subPath with expanded environment variables

Use thesubPathExpr field to constructsubPath directory names fromdownward API environment variables.ThesubPath andsubPathExpr properties are mutually exclusive.

In this example, aPod usessubPathExpr to create a directorypod1 withinthehostPath volume/var/log/pods.ThehostPath volume takes thePod name from thedownwardAPI.The host directory/var/log/pods/pod1 is mounted at/logs in the container.

apiVersion:v1kind:Podmetadata:name:pod1spec:containers:-name:container1env:-name:POD_NAMEvalueFrom:fieldRef:apiVersion:v1fieldPath:metadata.nameimage:busybox:1.28command:["sh","-c","while [ true ]; do echo 'Hello'; sleep 10; done | tee -a /logs/hello.txt"]volumeMounts:-name:workdir1mountPath:/logs# The variable expansion uses round brackets (not curly brackets).subPathExpr:$(POD_NAME)restartPolicy:Nevervolumes:-name:workdir1hostPath:path:/var/log/pods

Resources

The storage medium (such as Disk or SSD) of anemptyDir volume is determined by themedium of the filesystem holding the kubelet root dir (typically/var/lib/kubelet). There is no limit on how much space anemptyDir orhostPath volume can consume, and no isolation between containers orpods.

To learn about requesting space using a resource specification, seehow to manage resources.

Out-of-tree volume plugins

The out-of-tree volume plugins includeContainer Storage Interface (CSI), and alsoFlexVolume (which is deprecated). These plugins enable storage vendors to create custom storage pluginswithout adding their plugin source code to the Kubernetes repository.

Previously, all volume plugins were "in-tree". The "in-tree" plugins were built, linked, compiled,and shipped with the core Kubernetes binaries. This meant that adding a new storage system toKubernetes (a volume plugin) required checking code into the core Kubernetes code repository.

Both CSI and FlexVolume allow volume plugins to be developed independently ofthe Kubernetes code base, and deployed (installed) on Kubernetes clusters asextensions.

For storage vendors looking to create an out-of-tree volume plugin, please referto thevolume plugin FAQ.

csi

Container Storage Interface(CSI) defines a standard interface for container orchestration systems (likeKubernetes) to expose arbitrary storage systems to their container workloads.

Please read theCSI design proposalfor more information.

Once a CSI-compatible volume driver is deployed on a Kubernetes cluster, usersmay use thecsi volume type to attach or mount the volumes exposed by theCSI driver.

Acsi volume can be used in a Pod in three different ways:

• through a reference to aPersistentVolumeClaim
• with ageneric ephemeral volume
• with aCSI ephemeral volumeif the driver supports that

The following fields are available to storage administrators to configure a CSIpersistent volume:

• driver: A string value that specifies the name of the volume driver to use.This value must correspond to the value returned in theGetPluginInfoResponseby the CSI driver as defined in theCSI spec.It is used by Kubernetes to identify which CSI driver to call out to, and byCSI driver components to identify which PV objects belong to the CSI driver.
• volumeHandle: A string value that uniquely identifies the volume. This valuemust correspond to the value returned in thevolume.id field of theCreateVolumeResponse by the CSI driver as defined in theCSI spec.The value is passed asvolume_id in all calls to the CSI volume driver whenreferencing the volume.
• readOnly: An optional boolean value indicating whether the volume is to be"ControllerPublished" (attached) as read only. Default is false. This value is passedto the CSI driver via thereadonly field in theControllerPublishVolumeRequest.
• fsType: If the PV'sVolumeMode isFilesystem, then this field may be usedto specify the filesystem that should be used to mount the volume. If thevolume has not been formatted and formatting is supported, this value will beused to format the volume.This value is passed to the CSI driver via theVolumeCapability field ofControllerPublishVolumeRequest,NodeStageVolumeRequest, andNodePublishVolumeRequest.
• volumeAttributes: A map of string to string that specifies static propertiesof a volume. This map must correspond to the map returned in thevolume.attributes field of theCreateVolumeResponse by the CSI driver asdefined in theCSI spec.The map is passed to the CSI driver via thevolume_context field in theControllerPublishVolumeRequest,NodeStageVolumeRequest, andNodePublishVolumeRequest.
• controllerPublishSecretRef: A reference to the secret object containingsensitive information to pass to the CSI driver to complete the CSIControllerPublishVolume andControllerUnpublishVolume calls. This field isoptional, and may be empty if no secret is required. If the Secretcontains more than one secret, all secrets are passed.
• nodeExpandSecretRef: A reference to the secret containing sensitiveinformation to pass to the CSI driver to complete the CSINodeExpandVolume call. This field is optional and may be empty if nosecret is required. If the object contains more than one secret, allsecrets are passed. When you have configured secret data for node-initiatedvolume expansion, the kubelet passes that data via theNodeExpandVolume()call to the CSI driver. All supported versions of Kubernetes offer thenodeExpandSecretRef field, and have it available by default. Kubernetes releasesprior to v1.25 did not include this support.
• Enable thefeature gatenamedCSINodeExpandSecret for each kube-apiserver and for the kubelet on everynode. Since Kubernetes version 1.27, this feature has been enabled by defaultand no explicit enablement of the feature gate is required.You must also be using a CSI driver that supports or requires secret data duringnode-initiated storage resize operations.
• nodePublishSecretRef: A reference to the secret object containingsensitive information to pass to the CSI driver to complete the CSINodePublishVolume call. This field is optional and may be empty if nosecret is required. If the secret object contains more than one secret, allsecrets are passed.
• nodeStageSecretRef: A reference to the secret object containingsensitive information to pass to the CSI driver to complete the CSINodeStageVolume call. This field is optional and may be empty if no secretis required. If the Secret contains more than one secret, all secretsare passed.

CSI raw block volume support

Vendors with external CSI drivers can implement raw block volume supportin Kubernetes workloads.

You can set up yourPersistentVolume/PersistentVolumeClaim with raw block volume supportas usual, without any CSI-specific changes.

CSI ephemeral volumes

You can directly configure CSI volumes within the Podspecification. Volumes specified in this way are ephemeral and do notpersist across pod restarts. SeeEphemeral Volumesfor more information.

For more information on how to develop a CSI driver, refer to thekubernetes-csi documentation

Windows CSI proxy

CSI node plugins need to perform various privilegedoperations like scanning of disk devices and mounting of file systems. These operationsdiffer for each host operating system. For Linux worker nodes, containerized CSI nodeplugins are typically deployed as privileged containers. For Windows worker nodes,privileged operations for containerized CSI node plugins is supported usingcsi-proxy, a community-managed,stand-alone binary that needs to be pre-installed on each Windows node.

For more details, refer to the deployment guide of the CSI plugin you wish to deploy.

Migrating to CSI drivers from in-tree plugins

TheCSIMigration feature directs operations against existing in-treeplugins to corresponding CSI plugins (which are expected to be installed and configured).As a result, operators do not have to make anyconfiguration changes to existing Storage Classes, PersistentVolumes or PersistentVolumeClaims(referring to in-tree plugins) when transitioning to a CSI driver that supersedes an in-tree plugin.

The operations and features that are supported include:provisioning/delete, attach/detach, mount/unmount and resizing of volumes.

In-tree plugins that supportCSIMigration and have a corresponding CSI driver implementedare listed inTypes of Volumes.

flexVolume (deprecated)

FlexVolume is an out-of-tree plugin interface that uses an exec-based model to interfacewith storage drivers. The FlexVolume driver binaries must be installed in a pre-definedvolume plugin path on each node and in some cases the control plane nodes as well.

Pods interact with FlexVolume drivers through theflexVolume in-tree volume plugin.

The following FlexVolumeplugins,deployed as PowerShell scripts on the host, support Windows nodes:

• SMB
• iSCSI

Mount propagation

Mount propagation allows for sharing volumes mounted by a container toother containers in the same pod, or even to other pods on the same node.

Mount propagation of a volume is controlled by themountPropagation fieldincontainers[*].volumeMounts. Its values are:

• None - This volume mount will not receive any subsequent mountsthat are mounted to this volume or any of its subdirectories by the host.In similar fashion, no mounts created by the container will be visible onthe host. This is the default mode.

  This mode is equal torprivate mount propagation as described inmount(8)

  However, the CRI runtime may chooserslave mount propagation (i.e.,HostToContainer) instead, whenrprivate propagation is not applicable.cri-dockerd (Docker) is known to chooserslave mount propagation when themount source contains the Docker daemon's root directory (/var/lib/docker).

• HostToContainer - This volume mount will receive all subsequent mountsthat are mounted to this volume or any of its subdirectories.

  In other words, if the host mounts anything inside the volume mount, thecontainer will see it mounted there.

  Similarly, if any Pod withBidirectional mount propagation to the samevolume mounts anything there, the container withHostToContainer mountpropagation will see it.

  This mode is equal torslave mount propagation as described in themount(8)

• Bidirectional - This volume mount behaves the same theHostToContainer mount.In addition, all volume mounts created by the container will be propagatedback to the host and to all containers of all pods that use the same volume.

  A typical use case for this mode is a Pod with a FlexVolume or CSI driver ora Pod that needs to mount something on the host using ahostPath volume.

  This mode is equal torshared mount propagation as described in themount(8)

  Warning:

  Bidirectional mount propagation can be dangerous. It can damagethe host operating system and therefore it is allowed only in privilegedcontainers. Familiarity with Linux kernel behavior is strongly recommended.In addition, any volume mounts created by containers in pods must be destroyed(unmounted) by the containers on termination.

Read-only mounts

A mount can be made read-only by setting the.spec.containers[].volumeMounts[].readOnlyfield totrue.This does not make the volume itself read-only, but that specific container willnot be able to write to it.Other containers in the Pod may mount the same volume as read-write.

On Linux, read-only mounts are not recursively read-only by default.For example, consider a Pod which mounts the hosts/mnt as ahostPath volume. Ifthere is another filesystem mounted read-write on/mnt/<SUBMOUNT> (such as tmpfs,NFS, or USB storage), the volume mounted into the container(s) will also have a writeable/mnt/<SUBMOUNT>, even if the mount itself was specified as read-only.

Recursive read-only mounts

Recursive read-only mounts can be enabled by setting theRecursiveReadOnlyMountsfeature gatefor kubelet and kube-apiserver, and setting the.spec.containers[].volumeMounts[].recursiveReadOnlyfield for a pod.

The allowed values are:

• Disabled (default): no effect.

• Enabled: makes the mount recursively read-only.Needs all the following requirements to be satisfied:

  • readOnly is set totrue
  • mountPropagation is unset, or, set toNone
  • The host is running with Linux kernel v5.12 or later
  • TheCRI-level container runtime supports recursive read-only mounts
  • The OCI-level container runtime supports recursive read-only mounts.

  It will fail if any of these is not true.

• IfPossible: attempts to applyEnabled, and falls back toDisabledif the feature is not supported by the kernel or the runtime class.

Example:

storage/rro.yaml

apiVersion:v1kind:Podmetadata:name:rrospec:volumes:-name:mnthostPath:# tmpfs is mounted on /mnt/tmpfspath:/mntcontainers:-name:busyboximage:busyboxargs:["sleep","infinity"]volumeMounts:# /mnt-rro/tmpfs is not writable-name:mntmountPath:/mnt-rroreadOnly:truemountPropagation:NonerecursiveReadOnly:Enabled# /mnt-ro/tmpfs is writable-name:mntmountPath:/mnt-roreadOnly:true# /mnt-rw/tmpfs is writable-name:mntmountPath:/mnt-rw

When this property is recognized by kubelet and kube-apiserver,the.status.containerStatuses[].volumeMounts[].recursiveReadOnly field is set to eitherEnabled orDisabled.

Implementations

The following container runtimes are known to support recursive read-only mounts.

CRI-level:

• containerd, since v2.0
• CRI-O, since v1.30

OCI-level:

• runc, since v1.1
• crun, since v1.8.6

What's next

Follow an example ofdeploying WordPress and MySQL with Persistent Volumes.