Troubleshooting NVMe disks

This document lists errors that you might encounter when using disks with thenonvolatile memory express (NVMe) interface.

You can use the NVMe interface for Local SSDs and persistent disks (Persistent Diskor Google Cloud Hyperdisk). Only themost recent machine series, such as Tau T2A, M3, C3, C3D, and H3 use theNVMe interface forPersistent Disk. Confidential VMs also use NVMe for Persistent Disk. All otherCompute Engine machine series use theSCSI disk interface forpersistent disks.

I/O operation timeout error

If you are encountering I/O timeout errors, latency could be exceeding thedefault timeout parameter for I/O operations submitted to NVMe devices.

Error message:

[1369407.045521] nvme nvme0: I/O 252 QID 2 timeout, aborting[1369407.050941] nvme nvme0: I/O 253 QID 2 timeout, aborting[1369407.056354] nvme nvme0: I/O 254 QID 2 timeout, aborting[1369407.061766] nvme nvme0: I/O 255 QID 2 timeout, aborting[1369407.067168] nvme nvme0: I/O 256 QID 2 timeout, aborting[1369407.072583] nvme nvme0: I/O 257 QID 2 timeout, aborting[1369407.077987] nvme nvme0: I/O 258 QID 2 timeout, aborting[1369407.083395] nvme nvme0: I/O 259 QID 2 timeout, aborting[1369407.088802] nvme nvme0: I/O 260 QID 2 timeout, aborting...

Resolution:

To resolve this issue, increase the value of the timeout parameter.

Note: Most of the operating system images provided by Google alreadyinclude this change.

  1. View the current value of the timeout parameter.

    1. Determine which NVMe controller is used by the persistent disk or LocalSSD volume.
      ls -l /dev/disk/by-id

    2. Display theio_timeout setting, specified in seconds, for the disk.

      cat /sys/class/nvme/CONTROLLER_ID/NAMESPACE/queue/io_timeout
      Replace the following:

      • CONTROLLER_ID: the ID of the NVMe diskcontroller, for example,nvme1
      • NAMESPACE: the namespace of the NVMe disk, forexample,nvme1n1

      If you only have a single disk that uses NVMe, then use the command:

      cat /sys/class/nvme/nvme0/nvme0n1/queue/io_timeout

  2. To increase the timeout parameter for I/O operations submitted to NVMedevices, add the following line to the/lib/udev/rules.d/65-gce-disk-naming.rules file, and then restart the VM:

    KERNEL=="nvme*n*", ENV{DEVTYPE}=="disk", ATTRS{model}=="nvme_card-pd", ATTR{queue/io_timeout}="4294967295"

Detached disks still appear in the operating system of a compute instance

On VMs that use Linux kernel version 6.0 to 6.2, operationsinvolving the Compute Engine API methodinstances.detachDisk or thegcloud compute instances detach-disk command might not work as expected.The Google Cloud console shows the device as removed, the compute instance metadata(compute disks describe command) shows the device as removed, but the devicemount point and any symlinks created by udev rules are still visible in theguest operating system.

Error message:

Attempting to read from the detached disk on the VM results in I/O errors:

sudo head /dev/nvme0n3head: error reading '/dev/nvme0n3': Input/output error

Issue:

Operating system images that use a Linux 6.0-6.2 kernel but don't includea backport of aNVMe fixfail to recognize when an NVMe disk is detached.

Resolution:

Reboot the VM to complete the process of removing the disk.

To avoid this issue, use an operating system with aLinux kernel version that doesn't have this problem:

  • 5.19 or older
  • 6.3 or newer

You can use theuname -r command in the guest OS to view the Linux kernelversion.

What's next?

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.