Storage Classes

This document describes the concept of a StorageClass in Kubernetes. Familiaritywithvolumes andpersistent volumes is suggested.

A StorageClass provides a way for administrators to describe theclasses ofstorage they offer. Different classes might map to quality-of-service levels,or to backup policies, or to arbitrary policies determined by the clusteradministrators. Kubernetes itself is unopinionated about what classesrepresent.

The Kubernetes concept of a storage class is similar to “profiles” in some otherstorage system designs.

StorageClass objects

Each StorageClass contains the fieldsprovisioner,parameters, andreclaimPolicy, which are used when a PersistentVolume belonging to theclass needs to be dynamically provisioned to satisfy a PersistentVolumeClaim (PVC).

The name of a StorageClass object is significant, and is how users canrequest a particular class. Administrators set the name and other parametersof a class when first creating StorageClass objects.

As an administrator, you can specify a default StorageClass that applies to any PVCs thatdon't request a specific class. For more details, see thePersistentVolumeClaim concept.

Here's an example of a StorageClass:

storage/storageclass-low-latency.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:low-latencyannotations:storageclass.kubernetes.io/is-default-class:"false"provisioner:csi-driver.example-vendor.examplereclaimPolicy:Retain# default value is DeleteallowVolumeExpansion:truemountOptions:- discard# this might enable UNMAP / TRIM at the block storage layervolumeBindingMode:WaitForFirstConsumerparameters:guaranteedReadWriteLatency:"true"# provider-specific

Default StorageClass

You can mark a StorageClass as the default for your cluster.For instructions on setting the default StorageClass, seeChange the default StorageClass.

When a PVC does not specify astorageClassName, the default StorageClass isused.

If you set thestorageclass.kubernetes.io/is-default-classannotation to true on more than one StorageClass in your cluster, and you thencreate a PersistentVolumeClaim with nostorageClassName set, Kubernetesuses the most recently created default StorageClass.

You can create a PersistentVolumeClaim without specifying astorageClassNamefor the new PVC, and you can do so even when no default StorageClass existsin your cluster. In this case, the new PVC creates as you defined it, and thestorageClassName of that PVC remains unset until a default becomes available.

You can have a cluster without any default StorageClass. If you don't mark anyStorageClass as default (and one hasn't been set for you by, for example, a cloud provider),then Kubernetes cannot apply that defaulting for PersistentVolumeClaims that needit.

If or when a default StorageClass becomes available, the control plane identifies anyexisting PVCs withoutstorageClassName. For the PVCs that either have an emptyvalue forstorageClassName or do not have this key, the control plane thenupdates those PVCs to setstorageClassName to match the new default StorageClass.If you have an existing PVC where thestorageClassName is"", and you configurea default StorageClass, then this PVC will not get updated.

In order to keep binding to PVs withstorageClassName set to""(while a default StorageClass is present), you need to set thestorageClassNameof the associated PVC to"".

Provisioner

Each StorageClass has a provisioner that determines what volume plugin is usedfor provisioning PVs. This field must be specified.

You are not restricted to specifying the "internal" provisionerslisted here (whose names are prefixed with "kubernetes.io" and shippedalongside Kubernetes). You can also run and specify external provisioners,which are independent programs that follow aspecificationdefined by Kubernetes. Authors of external provisioners have full discretionover where their code lives, how the provisioner is shipped, how it needs to berun, what volume plugin it uses (including Flex), etc. The repositorykubernetes-sigs/sig-storage-lib-external-provisionerhouses a library for writing external provisioners that implements the bulk ofthe specification. Some external provisioners are listed under the repositorykubernetes-sigs/sig-storage-lib-external-provisioner.

For example, NFS doesn't provide an internal provisioner, but an externalprovisioner can be used. There are also cases when 3rd party storagevendors provide their own external provisioner.

Reclaim policy

PersistentVolumes that are dynamically created by a StorageClass will have thereclaim policyspecified in thereclaimPolicy field of the class, which can beeitherDelete orRetain. If noreclaimPolicy is specified when aStorageClass object is created, it will default toDelete.

PersistentVolumes that are created manually and managed via a StorageClass will havewhatever reclaim policy they were assigned at creation.

Volume expansion

PersistentVolumes can be configured to be expandable. This allows you to resize thevolume by editing the corresponding PVC object, requesting a new larger amount ofstorage.

The following types of volumes support volume expansion, when the underlyingStorageClass has the fieldallowVolumeExpansion set to true.

Mount options

PersistentVolumes that are dynamically created by a StorageClass will have themount options specified in themountOptions field of the class.

If the volume plugin does not support mount options but mount options arespecified, provisioning will fail. Mount options arenot validated on eitherthe class or PV. If a mount option is invalid, the PV mount fails.

Volume binding mode

ThevolumeBindingMode field controls whenvolume binding and dynamic provisioningshould occur. When unset,Immediate mode is used by default.

TheImmediate mode indicates that volume binding and dynamicprovisioning occurs once the PersistentVolumeClaim is created. For storagebackends that are topology-constrained and not globally accessible from all Nodesin the cluster, PersistentVolumes will be bound or provisioned without knowledge of the Pod's schedulingrequirements. This may result in unschedulable Pods.

A cluster administrator can address this issue by specifying theWaitForFirstConsumer mode whichwill delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.PersistentVolumes will be selected or provisioned conforming to the topology that isspecified by the Pod's scheduling constraints. These include, but are not limited to,resourcerequirements,node selectors,pod affinity andanti-affinity,andtaints and tolerations.

The following plugins supportWaitForFirstConsumer with dynamic provisioning:

• CSI volumes, provided that the specific CSI driver supports this

The following plugins supportWaitForFirstConsumer with pre-created PersistentVolume binding:

• CSI volumes, provided that the specific CSI driver supports this
• local

storage/storageclass/pod-volume-binding.yaml

apiVersion:v1kind:Podmetadata:name:task-pv-podspec:nodeSelector:kubernetes.io/hostname:kube-01volumes:-name:task-pv-storagepersistentVolumeClaim:claimName:task-pv-claimcontainers:-name:task-pv-containerimage:nginxports:-containerPort:80name:"http-server"volumeMounts:-mountPath:"/usr/share/nginx/html"name:task-pv-storage

Allowed topologies

When a cluster operator specifies theWaitForFirstConsumer volume binding mode, it is no longer necessaryto restrict provisioning to specific topologies in most situations. However,if still required,allowedTopologies can be specified.

This example demonstrates how to restrict the topology of provisioned volumes to specificzones and should be used as a replacement for thezone andzones parameters for thesupported plugins.

storage/storageclass/storageclass-topology.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:standardprovisioner:example.com/exampleparameters:type:pd-standardvolumeBindingMode:WaitForFirstConsumerallowedTopologies:-matchLabelExpressions:-key:topology.kubernetes.io/zonevalues:- us-central-1a- us-central-1b

Parameters

StorageClasses have parameters that describe volumes belonging to the storageclass. Different parameters may be accepted depending on theprovisioner.When a parameter is omitted, some default is used.

There can be at most 512 parameters defined for a StorageClass.The total length of the parameters object including its keys and values cannotexceed 256 KiB.

AWS EBS

Kubernetes 1.35 does not include aawsElasticBlockStore volume type.

The AWSElasticBlockStore in-tree storage driver was deprecated in the Kubernetes v1.19 releaseand then removed entirely in the v1.27 release.

The Kubernetes project suggests that you use theAWS EBSout-of-tree storage driver instead.

Here is an example StorageClass for the AWS EBS CSI driver:

storage/storageclass/storageclass-aws-ebs.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:ebs-scprovisioner:ebs.csi.aws.comvolumeBindingMode:WaitForFirstConsumerparameters:csi.storage.k8s.io/fstype:xfstype:io1iopsPerGB:"50"encrypted:"true"tagSpecification_1:"key1=value1"tagSpecification_2:"key2=value2"allowedTopologies:-matchLabelExpressions:-key:topology.ebs.csi.aws.com/zonevalues:- us-east-2c

tagSpecification: Tags with this prefix are applied to dynamically provisioned EBS volumes.

AWS EFS

To configure AWS EFS storage, you can use the out-of-treeAWS_EFS_CSI_DRIVER.

storage/storageclass/storageclass-aws-efs.yaml

kind:StorageClassapiVersion:storage.k8s.io/v1metadata:name:efs-scprovisioner:efs.csi.aws.comparameters:provisioningMode:efs-apfileSystemId:fs-92107410directoryPerms:"700"

• provisioningMode: The type of volume to be provisioned by Amazon EFS. Currently, only access point based provisioning is supported (efs-ap).
• fileSystemId: The file system under which the access point is created.
• directoryPerms: The directory permissions of the root directory created by the access point.

For more details, refer to theAWS_EFS_CSI_Driver Dynamic Provisioning documentation.

NFS

To configure NFS storage, you can use the in-tree driver or theNFS CSI driver for Kubernetes(recommended).

storage/storageclass/storageclass-nfs.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:example-nfsprovisioner:example.com/external-nfsparameters:server:nfs-server.example.compath:/sharereadOnly:"false"

• server: Server is the hostname or IP address of the NFS server.
• path: Path that is exported by the NFS server.
• readOnly: A flag indicating whether the storage will be mounted as read only (default false).

Kubernetes doesn't include an internal NFS provisioner.You need to use an external provisioner to create a StorageClass for NFS.Here are some examples:

• NFS Ganesha server and external provisioner
• NFS subdir external provisioner

vSphere

There are two types of provisioners for vSphere storage classes:

• CSI provisioner:csi.vsphere.vmware.com
• vCP provisioner:kubernetes.io/vsphere-volume

In-tree provisioners aredeprecated.For more information on the CSI provisioner, seeKubernetes vSphere CSI Driver andvSphereVolume CSI migration.

CSI Provisioner

The vSphere CSI StorageClass provisioner works with Tanzu Kubernetes clusters.For an example, refer to thevSphere CSI repository.

vCP Provisioner

The following examples use the VMware Cloud Provider (vCP) StorageClass provisioner.

1. Create a StorageClass with a user specified disk format.

   apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:fastprovisioner:kubernetes.io/vsphere-volumeparameters:diskformat:zeroedthick

   diskformat:thin,zeroedthick andeagerzeroedthick. Default:"thin".

2. Create a StorageClass with a disk format on a user specified datastore.

   apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:fastprovisioner:kubernetes.io/vsphere-volumeparameters:diskformat:zeroedthickdatastore:VSANDatastore

   datastore: The user can also specify the datastore in the StorageClass.The volume will be created on the datastore specified in the StorageClass,which in this case isVSANDatastore. This field is optional. If thedatastore is not specified, then the volume will be created on the datastorespecified in the vSphere config file used to initialize the vSphere CloudProvider.

3. Storage Policy Management inside kubernetes

   • Using existing vCenter SPBM policy

     One of the most important features of vSphere for Storage Management ispolicy based Management. Storage Policy Based Management (SPBM) is astorage policy framework that provides a single unified control planeacross a broad range of data services and storage solutions. SPBM enablesvSphere administrators to overcome upfront storage provisioning challenges,such as capacity planning, differentiated service levels and managingcapacity headroom.

     The SPBM policies can be specified in the StorageClass using thestoragePolicyName parameter.

   • Virtual SAN policy support inside Kubernetes

     Vsphere Infrastructure (VI) Admins will have the ability to specify customVirtual SAN Storage Capabilities during dynamic volume provisioning. Youcan now define storage requirements, such as performance and availability,in the form of storage capabilities during dynamic volume provisioning.The storage capability requirements are converted into a Virtual SANpolicy which are then pushed down to the Virtual SAN layer when apersistent volume (virtual disk) is being created. The virtual disk isdistributed across the Virtual SAN datastore to meet the requirements.

     You can seeStorage Policy Based Management for dynamic provisioning of volumesfor more details on how to use storage policies for persistent volumesmanagement.

Ceph RBD (deprecated)

storage/storageclass/storageclass-ceph-rbd.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:fastprovisioner:kubernetes.io/rbd# This provisioner is deprecatedparameters:monitors:198.19.254.105:6789adminId:kubeadminSecretName:ceph-secretadminSecretNamespace:kube-systempool:kubeuserId:kubeuserSecretName:ceph-secret-useruserSecretNamespace:defaultfsType:ext4imageFormat:"2"imageFeatures:"layering"

• monitors: Ceph monitors, comma delimited. This parameter is required.

• adminId: Ceph client ID that is capable of creating images in the pool.Default is "admin".

• adminSecretName: Secret Name foradminId. This parameter is required.The provided secret must have type "kubernetes.io/rbd".

• adminSecretNamespace: The namespace foradminSecretName. Default is "default".

• pool: Ceph RBD pool. Default is "rbd".

• userId: Ceph client ID that is used to map the RBD image. Default is thesame asadminId.

• userSecretName: The name of Ceph Secret foruserId to map RBD image. Itmust exist in the same namespace as PVCs. This parameter is required.The provided secret must have type "kubernetes.io/rbd", for example created in thisway:

  kubectl create secret generic ceph-secret --type="kubernetes.io/rbd"\  --from-literal=key='QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ=='\  --namespace=kube-system

• userSecretNamespace: The namespace foruserSecretName.

• fsType: fsType that is supported by kubernetes. Default:"ext4".

• imageFormat: Ceph RBD image format, "1" or "2". Default is "2".

• imageFeatures: This parameter is optional and should only be used if yousetimageFormat to "2". Currently supported features arelayering only.Default is "", and no features are turned on.

Azure Disk

Kubernetes 1.35 does not include aazureDisk volume type.

TheazureDisk in-tree storage driver was deprecated in the Kubernetes v1.19 releaseand then removed entirely in the v1.27 release.

The Kubernetes project suggests that you use theAzure Disk third partystorage driver instead.

Azure File (deprecated)

storage/storageclass/storageclass-azure-file.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:azurefileprovisioner:kubernetes.io/azure-fileparameters:skuName:Standard_LRSlocation:eastusstorageAccount:azure_storage_account_name# example value

• skuName: Azure storage account SKU tier. Default is empty.
• location: Azure storage account location. Default is empty.
• storageAccount: Azure storage account name. Default is empty. If a storageaccount is not provided, all storage accounts associated with the resourcegroup are searched to find one that matchesskuName andlocation. If astorage account is provided, it must reside in the same resource group as thecluster, andskuName andlocation are ignored.
• secretNamespace: the namespace of the secret that contains the Azure StorageAccount Name and Key. Default is the same as the Pod.
• secretName: the name of the secret that contains the Azure Storage Account Name andKey. Default isazure-storage-account-<accountName>-secret
• readOnly: a flag indicating whether the storage will be mounted as read only.Defaults to false which means a read/write mount. This setting will impact theReadOnly setting in VolumeMounts as well.

During storage provisioning, a secret named bysecretName is created for themounting credentials. If the cluster has enabled bothRBAC andController Roles,add thecreate permission of resourcesecret for clusterrolesystem:controller:persistent-volume-binder.

In a multi-tenancy context, it is strongly recommended to set the value forsecretNamespace explicitly, otherwise the storage account credentials maybe read by other users.

Portworx volume (deprecated)

storage/storageclass/storageclass-portworx-volume.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:portworx-io-priority-highprovisioner:kubernetes.io/portworx-volume# This provisioner is deprecatedparameters:repl:"1"snap_interval:"70"priority_io:"high"

• fs: filesystem to be laid out:none/xfs/ext4 (default:ext4).
• block_size: block size in Kbytes (default:32).
• repl: number of synchronous replicas to be provided in the form ofreplication factor1..3 (default:1) A string is expected here i.e."1" and not1.
• priority_io: determines whether the volume will be created from higherperformance or a lower priority storagehigh/medium/low (default:low).
• snap_interval: clock/time interval in minutes for when to trigger snapshots.Snapshots are incremental based on difference with the prior snapshot, 0disables snaps (default:0). A string is expected here i.e."70" and not70.
• aggregation_level: specifies the number of chunks the volume would bedistributed into, 0 indicates a non-aggregated volume (default:0). A stringis expected here i.e."0" and not0
• ephemeral: specifies whether the volume should be cleaned-up after unmountor should be persistent.emptyDir use case can set this value to true andpersistent volumes use case such as for databases like Cassandra should setto false,true/false (defaultfalse). A string is expected here i.e."true" and nottrue.

Local

storage/storageclass/storageclass-local.yaml

apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:local-storageprovisioner:kubernetes.io/no-provisioner# indicates that this StorageClass does not support automatic provisioningvolumeBindingMode:WaitForFirstConsumer

Local volumes do not support dynamic provisioning in Kubernetes 1.35;however a StorageClass should still be created to delay volume binding until a Pod is actuallyscheduled to the appropriate node. This is specified by theWaitForFirstConsumer volumebinding mode.

Delaying volume binding allows the scheduler to consider all of a Pod'sscheduling constraints when choosing an appropriate PersistentVolume for aPersistentVolumeClaim.