Cloud TPU Error Glossary

This document provides a glossary of common errors with solutions from the Cloud TPU service.

Invalid Accelerator Type

Error Message

generic::invalid_argument: Accelerator type v2-512 as preemptible (false) andreserved (false) is not available in zone us-central1-a, please contact support.

Solution

An invalid parameter has been provided to the create command. The availabilityof an accelerator in a zone depends on the following parameters: the type, thepreemptible flag, thespot flag, and thereserved flag. Thepreemptible,spot,andreserved flags can be changed by including or excluding them in the createcommand.

A TPU created with thereserved flag will use reservedcapacity. Including thespot orpreemptible flags will allow the TPU to be preempted byhigher priority TPUs. If none of these flags are provided, the TPU will be on-demand. It isnot a valid configuration to enable more than one of these flags. See thecreate command documentation for more information.

The accelerator types available in each zone can be found in theTPU regions and zones documentation or they can bequeried using theaccelerator-types list command. Change the create command to use oneof these accelerator types and try again or contact support if the problem persists.

Network Not Found

Error Message

  Cloud TPU received a bad request. The field "Network" cannot be "NETWORK_NAME":  requested resource not found

Solution

The NetworkNETWORK_NAME was not found. Ensure that the Network was createdand set up properly. SeeCreate and manage VPC networks for more information.

Service Account Permission Denied

Error Message

  generic::permission_denied: Cloud TPU got permissions denied when trying to  access the customer project. Make sure that the IAM account  'service-[project number]@cloud-tpu.iam.gserviceaccount.com' has the 'Cloud  TPU API Service Agent' role by following https://cloud.google.com/iam/docs/manage-access-service-accounts

Solution

This error occurs when a user attempts to create or list nodes in a projectwithout IAM authorization. A likely cause of this issue is that the Cloud TPU API serviceaccount does not have the required role for the project. TheManage access accounts documentationgives an overview of how to manage access. Follow theGrant or revoke a single role stepsand give the account 'service-PROJECT_NUMBER@cloud-tpu.iam.gserviceaccount.com' the role of 'Cloud TPU API Service Agent' (be sure to replacePROJECT_NUMBERwith your project number, which can be found in the project settings in the Google Cloud console). For moreinformation on service agents, see theService agents documentation.

Quota Exceeded

Error Message

You have reached XXXX limit. Please request an increase for the 'YYYY' quota forCompute Engine API by following https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota.

Solution

Your project has reached a quota limit. To learn more about working with quotas, see theCloud Quotas documentation. This shouldn't be confused with theTPU quota, which governs the usage of TPU pods.

You may request an increase to the appropriate limit by following the steps listed atRequest a quota adjustment. On the quotas page, you may search for the quota specified by the 'YYYY' part of this message. Some quotas are split across different regions or services. The error message will indicate which one needs to be increased.

The 'XXXX' and 'YYYY' parts of the message may be one of the following:

• HEALTH_CHECKS - 'Health checks' quota
• FIREWALLS - 'Firewall rules' quota
• NETWORK_ENDPOINT_GROUPS - 'Network endpoint groups' quota for this region
• READ_REQUESTS - 'Read requests per minute' quota for the Compute Engine API service
• OPERATION_READ_REQUESTS - 'Operation read requests per minute' quota

This request is typically processed within 2-3 business days. If the request is urgent, reach out to a customer engineer or technical account manager.