How a Persistent Volume (PV) gets attached to a Kubernetes Pod

Kubernetes / By
Table of Contents

Introduction

Kubernetes provides robust mechanisms for managing storage, allowing applications to store data persistently, independent of their lifecycle. Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are the kubernetes concepts which relates to the persistent storage for the pods.

While the concept of abstracting storage is straightforward, the underlying process of how a PV actually gets attached and made available to a Pod is a complex dance involving multiple Kubernetes components.

In this blog, we will break down this intricate flow, step-by-step, using a detailed diagram as our guide. By the end, you’ll have a clear understanding of the journey a persistent volume takes from a user’s request to being mounted inside a Pod.

Kubernetes Components Involved

Before diving into the steps, let’s briefly introduce the key components in this orchestration:

  1. User/Client
    • The user or client Initiates the process by creating PVCs and Pods either manually or programatically.
    • Kubectl or other clients making API calls to kube-api server.
  2. API Server
    • The front-end of the Kubernetes control plane, exposing the Kubernetes API.
    • All communication happens through it, and it persists the cluster state in Etcd.
  3. Controller Manager
    • A daemon that embeds the core control loops. For storage, key controllers are:
      • PV Controller (Binder): Responsible for binding PVCs to PVs.
      • Attach/Detach Controller (AD Controller): Manages attaching and detaching volumes to/from nodes.
  4. Scheduler
    • It watches for new Pods with no assigned node and selects a node for them to run on.
    • If scheduler doesn’t find a suitable node, the pod would never gets scheduled and remains in pending state.
  5. CSI Driver (Controller Service)
    • The Container Storage Interface (CSI) provides a standard interface for storage vendors to integrate their storage systems with Kubernetes.
    • The Controller Service handles cluster-wide operations like volume provisioning, snapshotting, and attaching/detaching.
  6. Cloud Storage API
    • The actual storage system (e.g., AWS EBS, Google Persistent Disk, a SAN) that manages the physical storage.
  7. Kubelet
    • An agent that runs on each worker node. It’s responsible for managing Pods and their containers, including volume management (mounting, unmounting).
  8. CSI Driver (Node Service)
    • The Node Service part of the CSI driver runs on each worker node and handles node-specific operations like staging and publishing (mounting) volumes to Pods.
  9. Pod Container
    • The application container running inside the Pod, which consumes the mounted persistent storage.

The Detailed Flow: From Request to Mount

Let’s trace the journey of a Persistent Volume (PV), step by step, as depicted in our diagram below:

pv_to_pod_attachment_flow

1. User Initiates Storage Request: Create PVC & Pod

The journey begins with a user, typically via kubectl, defining their storage requirements.

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: my-persistent-storage
          mountPath: /usr/share/nginx/html
  volumes:
    - name: my-persistent-storage
      persistentVolumeClaim:
        claimName: my-app-pvc
  • User: Creates the PersistentVolumeClaim (PVC) and the Pod that references it. These YAML manifests are sent to the Kubernetes API Server.
  • API Server: Stores the PVC and Pod definitions in Etcd.

2. Provisioning & Binding (PV Controller / Binder)

This phase ensures that the requested storage (PVC) is matched with an actual PersistentVolume (PV).

  • API Server (Watch PVC)
    • The API Server informs the PV Controller (part of the Controller Manager) about the new PVC.
  • PV Controller (Binder)
    • It looks for an existing PersistentVolume that matches the PVC’s requirements (size, access modes, StorageClass). If found, it binds the PVC to that PV by updating their statuses.
    • If no suitable PV exists and the PVC specifies a StorageClass for dynamic provisioning, the PV Controller interacts with the CSI Driver’s Controller Service to provision a new physical volume on the underlying storage system.
    • Once provisioned, it creates a new PV object in Kubernetes and binds it to the PVC.
  • API Server (Update PV/PVC)
    • The PV Controller updates the API Server with the binding status, linking the PVC to a specific PV.

3. Pod Scheduling (Scheduler)

Once the PVC is bound to a PV, the Pod needs to be assigned to a worker node.

  • API Server (Watch Pod, Assign Node)
    • The API Server informs the Scheduler about the new Pod that needs to be scheduled.
  • Scheduler
    • It evaluates various factors (resource requests, node taints/tolerations, node affinity, and importantly, volume affinity/topology) to determine the best node for the Pod.
    • For volumes, the Scheduler ensures that the chosen node has access to the physical storage backend if it’s node-specific (e.g., in cloud environments, a disk might only be attachable to VMs in the same availability zone).
    • It then updates the Pod’s definition in the API Server with the selected nodeName.

4. Attaching the Volume to the Node (Attach/Detach Controller & CSI Controller Driver)

With the Pod scheduled to a specific node, the physical storage volume must now be made available to that node.

  • API Server (Watch Scheduled Pod)
    • The API Server notifies the Attach/Detach Controller about the Pod now having a nodeName.
  • AD Controller (Call Attach API)
    • It determines that the Pod’s required volume (the PV bound to the PVC) needs to be attached to the assigned node.
    • It creates a VolumeAttachment object in the API Server (if it doesn’t exist) and then makes an RPC (Remote Procedure Call) to the CSI Controller Driver’s ControllerPublishVolume method. This call instructs the CSI driver to attach the specified volume to the node identified.
  • CSI Controller Driver (API Call to Storage System)
    • The CSI Controller Driver translates the ControllerPublishVolume RPC into a specific API call to the underlying Cloud Storage API (e.g., AttachVolume to AWS EBS, AttachDisk to Google PD).
  • Cloud Storage API (Physically Attach Volume)
    • The storage system performs the actual operation of attaching the block device to the specified virtual machine (the worker node).
  • CSI Controller Driver (Success Status)
    • Upon successful attachment, the storage system returns a success status to the CSI Controller Driver.
  • CSI Controller Driver (Update VolumeAttachment Object)
    • The CSI Controller Driver updates the VolumeAttachment object in the API Server, indicating that the volume has been successfully attached to the node.

5. Mounting the Volume into the Pod (Kubelet & CSI Node Driver)

Finally, with the volume attached to the worker node, the Kubelet on that node takes over to mount it into the Pod.

  • API Server (Watch VolumeAttachment Status, Watch Pod)
    • The API Server informs the Kubelet on the worker node about the Pod assigned to it and the updated VolumeAttachment status (i.e., the volume is now attached to the node).
  • Kubelet (Volume Manager)
    • The Kubelet’s internal volume manager detects that the Pod needs a volume that is now attached to the node.
    • It then orchestrates the mounting process.
  • Kubelet (RPC: NodeStageVolume)
    • Kubelet makes an RPC to the CSI Node Driver’s NodeStageVolume method.
      • CSI Node Driver (Global Mount): This method is responsible for “staging” the volume. This typically involves:
        • Discovering the attached block device.
        • Formatting the device with a filesystem (if it’s a new volume).
        • Mounting the device to a global staging path on the node (e.g., /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv-name>/globalmount). This ensures the volume is ready for use by any Pod on the node.
  • CSI Node Driver (Success)
    • The Node Driver reports success to the Kubelet.
  • Kubelet (RPC: NodePublishVolume)
    • Kubelet then makes a subsequent RPC to the CSI Node Driver’s NodePublishVolume method.
      • CSI Node Driver (Bind Mount into Container Namespace)
        • This is the final step where the volume is made available inside the Pod.
        • The Node Driver performs a bind mount operation, taking the globally staged volume and mounting it to the specific mountPath requested by the Pod (e.g., /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<pv-name>/mount).
        • This mount is typically within the Pod’s private mount namespace, making it accessible to the containers.
  • Pod Container (PV Available for App)
    • Once NodePublishVolume completes successfully, the volume is mounted inside the Pod.
    • The application running in the container can now read from and write to the persistent storage at the specified mountPath.

Conclusion

  • The Kubernetes Persistent Volume attachment flow, while seemingly complex, is a meticulously designed process that ensures reliable and scalable storage for containerized applications.
  • By understanding the roles of the API Server, various controllers, the scheduler, Kubelet, and especially the CSI drivers, you can better troubleshoot storage issues and design robust, stateful applications on Kubernetes.
  • This distributed orchestration allows Kubernetes to abstract away the complexities of underlying storage systems, providing a consistent experience for developers and operators alike.

References

"Knowledge Sharing Is Powerful"

Related Posts

Scroll to Top