This page lists various issues that you might encounter when configuringVPC Service Controls.

Unexpected scoped policy behavior

You might notice some unexpected VPC Service Controls violations that yourscoped policy should allow. It is a known issue that if you don't have anorganization-level access policy, then you might experience some unexpectedissues with your scoped access policies.

To overcome this issue, create an access policy at the organization level usingthe following command:

gcloudaccess-context-managerpoliciescreate--organization<var>ORGANIZATION_ID</var>--title<var>POLICY_TITLE</var>

For more information, seeCreate an access policy.

Shared VPC

When using Shared VPC, a service perimeter that includes projects that belongto a Shared VPC network must also include the project thathosts the network.When projects that belong to a Shared VPC network are not in the same perimeteras the host project, services might not work as expected or might be blocked entirely.

Ensure that the Shared VPC network host is in the same service perimeter asthe projects connected to the network.

Unable to add a VPC network

The following error can occur when you try to add a VPC networkto a service perimeter:

ERROR: (gcloud.access-context-manager.perimeters.update) PERMISSION_DENIED: Permission 'compute.networks.get' denied on resource '//compute.googleapis.com/projects/PROJECT_NAME/global/networks/VPC_NETWORK_NAME' (or it may not exist)

This error occurs due to one of the following reasons:

• The VPC network doesn't exist.
• The VPC network exists, but doesn't have a subnet.
• The caller doesn't have the required permission.

To resolve this issue, complete the following steps:

1. Verify if the VPC network specified in the error messageexists byviewing thenetworks in yourproject.

   • Before verifying the VPC network, make sure that theCompute Engine API is enabled in the project associated with the API callby completing the following steps:

     1. In the Google Cloud console, go to theAPIs & Services page.
        Goto APIs & Services

     2. On theAPIs & Services page, verify if the Compute Engine API islisted.

     3. If the Compute Engine API is missing, enable the API.
        Enable theAPI

2. Verify if at least one subnet exists in the VPC network byviewing thesubnets. Ifthere are no subnets,add asubnet to theVPC network.

3. Check if the caller has the following permission on the host project of theVPC network:compute.networks.get. This permission lets youview VPC networks in a project.

   • Ask the administrator of the organization that owns the host project ofthe VPC network to grant the caller an IAMrole with thecompute.networks.get permission on the host project. Forexample, the Compute Network Viewer role.

     For more information about granting roles, seeManageaccess.

Ensure that you read thelimitationsassociated with using VPC networks in service perimeters.

Requests between perimeters

Normally, access levels are used to allow requests fromoutside a serviceperimeter for protected resourcesinside a perimeter.

However, a request from a project in a perimeter for a protected resourceinanother perimeter is denied, even if an access level normally allows the request.

For example, assume Project A in Perimeter 1 requests a resource from Project B.The resource in Project B is protected by Perimeter 2. Because Project A is ina perimeter, even if an access level for Perimeter 2 normally permits therequest for the protected resource, the request is denied.

Use one of the following approaches to facilitate requests between perimeters:

• Useegress policy and ingress policy.To allow requests fromanother perimeter to protected resources inyour perimeter, the other perimeter must use an egress policy and you must set an ingress policy in your perimeter.

• Use perimeter bridges.Bridges allow two or more projects in different perimeters to make requeststo any services in those projects. These requests are permitted even if theservices are protected by the respective perimeters.

• Ensure that both the requesting service and the target resource areunprotected by the perimeters. In this scenario, the operation succeedsbecause the services are not protected.

The email address is invalid or non-existent

When updating a perimeter that contains a deleted principal, you might encountertheThe email address is invalid or non-existent error.

To fix this issue, you must remove theinvalid email address from allperimeters:

1. Export all your perimeters.The following sample command exports a list of service perimeters in YAML format:

   gcloudaccess-context-managerperimeterslist\--policy=POLICY_NAME\--format="json(name,title,description,perimeterType,status,spec,useExplicitDryRunSpec)"\>my-perimeters.yaml

2. Remove theinvalid email address from themy-perimeters.yaml file andsave it asmy-perimeters-updated.yaml.

3. Bulk replace all your perimeters.

Ingress and egress rules violations

The audit log contains information about the ingress and egress rules violationsthat help you understand perimeter violations.

Ingress rule violation

An ingress rule violation indicates that an API client outside of the perimetertried to access a resource inside the perimeter. The service perimeter rejectsthe request as there are no matching ingress rules or access levels.

An ingress rule violation in the audit log contains the following details:

• The name of the perimeter in which the ingress rule violation occurred.
• The resource inside the perimeter that the API client outside the perimetertried to access.

In the following ingress rule violation example, an API client outside theperimeter tries to access the Cloud Storage bucketprod-protected-storage-bucket inside theperimeterprod-perimeter.

ingressViolations: [  0: {    targetResource: "projects/1234/buckets/prod-protected-storage-bucket"    servicePerimeter: "accessPolicies/123456789/servicePerimeters/prod-perimeter"  }]

To resolve this issue,create an ingress rulefor your perimeter. For more information about ingress rules, seeIngress rulesreference.

Egress rule violation

An egress rule violation in the audit log indicates one of the following events:

• An API client inside the perimeter tried to access a resource outside theperimeter.
• An API request that involves a resource inside the perimeter and a resourceoutside the perimeter. For example, a Cloud Storage client that calls a copycommand where one bucket is within the perimeter and the other bucket isoutside the perimeter.

The service perimeter rejects the request because there are no matching egressrules.An egress rule violation in the audit log includes the following details:

• The source type such as network or resource.
• The source, which is a resource or network, whose perimeter encountered an egressviolation.
• The perimeter that encountered an egress violation.
• The target resource outside the perimeter that the request tried to access.

In the following egress rule violation example, the API request includes aresource from projects/5678, which is inside the perimeterprod-perimeter,and an object from the Cloud Storage bucketexternal-storage-bucket, which isoutside the perimeter.

egressViolations: [  0: {    sourceType: "Resource"    source: "projects/5678"    targetResource: "projects/4321/buckets/external-storage-bucket/objects/corp-resources.json"    servicePerimeter: "accessPolicies/123456789/servicePerimeters/prod-perimeter"  }]

To resolve this issue,create an egress rulefor your perimeter. For more information about egress rules, seeEgress rulesreference.

Debugging requests blocked by VPC Service Controls

TheVPC Service Controls audit log is the primary tool fordebugging a request blocked by VPC Service Controls.

When access has been blocked unexpectedly, consult the audit logsin the project protected by the service perimeter. These logs containsignificant data about the requested resources and the reason why the requestwas denied. For information about diagnosing the audit logs, seeAccess the violation analyzer.

The following sections list theviolationReason values that you might encounterwhen using VPC Service Controls.

NETWORK_NOT_IN_SAME_SERVICE_PERIMETER

The reason for this issue might be one of the following:

• A client in a VPC network within a service perimeter tries to access a projectthat is not in the same perimeter. This request results in an egress violation.Create an egress rule to fix this issue.
• A client in a VPC network that is outside a service perimeter tries to accessa project that is protected by the service perimeter. This request results inan ingress violation. Create an ingress rule to fix this issue.

The client might send the request from a Compute Engine or Google Kubernetes Engine VM, orfrom an on-premises network through Cloud Interconnect or a VPN configured using a VPC network.

The following diagram shows that an egress violation occurs when a client in a VPCnetwork within a service perimeter tries to access a project outside the perimeter:

Here is an example of an egress violation:

egressViolations: [{  servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"  source: "projects/<NETWORK_PROJECT_NUMBER>"  sourceType: "Network"  targetResource: "projects/<RESOURCE_PROJECT_NUMBER>"}]

Where:

• <POLICY_NAME> is thenumeric name of your access policy.
• <PERIMETER_NAME> is the name of the service perimeter.
• <NETWORK_PROJECT_NUMBER> is the project number of the Google Cloud project that holds your VPC network.
• <RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.

The following diagram shows that an ingress violation occurs when a client outsidethe perimeter tries to access a project inside the perimeter:

Here is an example of an ingress violation:

ingressViolations: [{          targetResource: "projects/<RESOURCE_PROJECT_NUMBER>",      source: "projects/<NETWORK_PROJECT_NUMBER>"          servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"}]

Where:

• <RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.
• <NETWORK_PROJECT_NUMBER> is the project number of the Google Cloud project that holds your VPC network.
• <POLICY_NAME> is thenumeric name of your access policy.
• <PERIMETER_NAME> is the name of the service perimeter.

Resolution

To resolve this error,create an ingress or egressrule for yourperimeter.

RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER

This issue occurs when a single request is accessing multiple resources but theresources are not in the same service perimeter. This issue occurs irrespective ofwhere the client is located and whether the client has access to the resources.

The following diagram shows a client accessing resources from a project outside theperimeter and from a project within the service perimeter:

The following diagram shows a client accessing resources from projects which arein two different service perimeters but the perimeters do not communicate with each other:

Here is an example of an egress violation:

egressViolations: [{  servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER_NAME>"  source: "projects/<RESOURCE_PROJECT_INSIDE_THIS_PERIMETER>"  sourceType: "Resource"  targetResource: "projects/<RESOURCE_PROJECT_OUTSIDE_THIS-PERIMETER>"}]

Where:

• <POLICY_NAME> is thenumeric name of your access policy.
• <PERIMETER_NAME> is the name of the service perimeter.
• <RESOURCE_PROJECT_INSIDE_THIS_PERIMETER> is the project number of the Google Cloud project that is within the perimeter.
• <RESOURCE_PROJECT_OUTSIDE_THIS_PERIMETER> is the project number of the Google Cloud project that is outside the perimeter.

Resolution

To resolve this issue,create an egress rule for your perimeter.

NO_MATCHING_ACCESS_LEVEL

This issue occurs when the IP address, device requirement, or user identitydoesn't match any ingress rules or access levels assigned to the perimeter. Thismeans that a client that isn't a part of the Google Cloud network tries toaccess Google Cloud network resources from outside the perimeter. Forexample, the IP address corresponding to thecallerIp field of the auditrecord doesn't match any CIDR ranges defined in the access levels for theservice perimeter.

If the caller IP address is missing or appears as an internal IP address, thenthis violation might be due to a Google Cloud service that isn'tintegrated with VPC Service Controls. The reason could be that theGoogle Cloud service tries to access a protected service and fails, asexpected.

To fix this issue, we recommend creating an ingress rule instead of an accesslevel because an ingress rule provides granular access control.

The following diagram shows a client trying to access resources from outside theperimeter:

Here is an example of an ingress violation:

authenticationInfo: {  principalEmail: "EMAIL"}requestMetadata: {callerIp: "<PUBLIC_IP_ADDRESS>"deviceState: "Cross Organization"}ingressViolations: [        {          targetResource: "projects/<RESOURCE_PROJECT_NUMBER>",          servicePerimeter: "accessPolicies/<POLICY_NAME>/servicePerimeters/<PERIMETER-NAME>"        }  ]

Where:

• <EMAIL> is the email address of the service account or authenticated user.

  If you use a Google Cloud service thatVPC Service Controls doesn'tsupport, email addresses thatbelong to the domaingoogle.com are redacted and replaced withgoogle-internal.google-internal refers to internal Google-owned identities.

• <PUBLIC_IP_ADDRESS> is the IP address of thecaller. For a caller from the internet, this will be the public IPv4 or IPv6address.

• <RESOURCE_PROJECT_NUMBER> is the project number of the Google Cloud project that contains the resource.

• <POLICY_NAME> is thenumeric name of your access policy.

• <PERIMETER_NAME> is the name of the service perimeter.

Note that in this case,metadata.accessLevels could still be present sincethese access levels might not be specified in the violated perimeter.

SERVICE_NOT_ALLOWED_FROM_VPC

This issue occurs when a client tries to access Google Cloud resources from a VPC network.The client might send the request from a Compute Engine or Google Kubernetes Engine VM,or from an on-premises network through Cloud Interconnect or a VPN configured using a VPC network.

To fix this issue, ensure that the service being called is allowed by theVPC accessibleservices configuration of the service perimeter.

• Cloud Storage access from on-premises
• BigQuery access from VM outside of project
• Cross-project BigQuery query
• Move Cloud Storage file inside perimeter
• Move Cloud Storage file outside perimeter
• BigQuery dataset copy from VM inside perimeter
• Dataproc job reading from project
• Unsupported service with restricted VIP
• Log export to project outside perimeter
• BigQuery extract to Cloud Storage

Cloud Storage access from on-premises

In this example, VPC Service Controls blocks a request from an employeeworkstation (identified bycallerIp) to a Cloud Storage bucket in projectcorp-storage.

The request generates the following audit log record:

{insertId:"222lvajc6f7"logName:"projects/corp-storage/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"someone@google.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/_"]violationReason:"NO_MATCHING_ACCESS_LEVEL"}methodName:"google.storage.NoBillingOk"requestMetadata:{callerIp:"b1d5:d26d:5b17:43fe:d358:586b:db59:9617"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/690885588241"serviceName:"storage.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-27T21:40:43.823209571Z"resource:{labels:{method:"google.storage.NoBillingOk"project_id:"corp-storage"service:"storage.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-27T21:40:42.973784140Z"}

Thecorp-storage project is included in a service perimeter. The employeeworkstation is not a part of any networks within that perimeter. Because theemployee workstation exists outside the perimeter, the request is blocked.

BigQuery access from VM outside of project

In this example, a VM that belongs to project458854174376 (data-collector)attempts to run a BigQuery query against a dataset in project798816221974 (corp-resources-protected) and it is denied.

The VM uses the following query:

bq--project=corp-resources-protectedquery'select count(*) from babynames.yob2000'

The query returns the following output:

BigQuery error in query operation: VPC Service Controls: Request isprohibited by organization's policy. Operation ID:33643962-6a0f-4091-9283-bcdf7e9271f0

The following audit log record is generated:

{insertId:"1ei551d2pdq"logName:"projects/corp-resources-protected/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"714877721106-compute@developer.gserviceaccount.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/1004338142803"]violationReason:"NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"bigquery.googleapis.com/bigquery.jobs.create"requestMetadata:{callerIp:"10.105.0.2"callerNetwork:"//compute.googleapis.com/projects/ameet-dataflow/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/1004338142803"serviceName:"bigquery.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-28T23:06:13.579882505Z"resource:{labels:{method:"bigquery.googleapis.com/bigquery.jobs.create"project_id:"corp-resources-protected"service:"bigquery.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-28T23:06:12.799656975Z"}

In this example, theviolationReason isNETWORK_NOT_IN_SAME_SERVICE_PERIMETER.callerNetwork is included in additiontocallerIp. The IP address is private and the network is provided todisambiguate it. The relevant resources at issue here are listed in two places:VpcServiceControlAuditMetadata.resourceNames andrequestMetadata.callerNetwork (the project that owns the network).

The problem is that thecorp-resources-protected project is inside a serviceperimeter, whiledata-collector, the project that includes the network thatthe VM belongs to, is not. In this case, access is denied as expected.

Cross-project BigQuery query

In this example, a VM that belongs to theperimeter-network projectattempts to query the BigQuery instances of two differentprojects:corp-resources-protected, which is in the same service perimeter asperimeter-network, andcorp-resources-public, which is not.

The VM uses the following command:

bqquery--use_legacy_sql=false\'select count(priv.name),count(pub.name) from \    `corp-resources-protected.babynames.yob2000` as priv, \    `corp-resources-public.babynames.yob2000` as pub'

The query returns the following output:

BigQuery error in query operation: Error processing job'example:bqjob_r211e6f6eec928ffb_000001675c996aa8_1': VPC Service Controls:Request is prohibited by organization's policy. Operation ID:dc4fc177-4850-4fc5-b2e7-8c33f302149a

The following audit log record is generated:

{insertId:"17kg4exd24ag"logName:"projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/117961063178"1:"projects/690885588241"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"bigquery.googleapis.com/bigquery.tables.getData"requestMetadata:{callerIp:"130.211.225.66"callerNetwork:"//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/927005422713"serviceName:"bigquery.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-28T20:48:51.384237810Z"resource:{labels:{method:"bigquery.googleapis.com/bigquery.tables.getData"project_id:"perimeter-network"service:"bigquery.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-28T20:48:50.561884949Z"}

Looking atcallerNetwork andVpcServiceControlAuditMetadata.resourceNames, we can see three projects:perimeter-network,117961063178 (corp-resources-public), and690885588241(corp-resources-protected). Recall thatcorp-resources-public is not inthe same service perimeter asperimeter-network andcorp-resources-protected.

TheviolationReason,RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER indicates thatsome resource in the request is outside of a perimeter that applies to therequest. In this case, that resource iscorp-resources-public.

Move Cloud Storage file inside perimeter

In this example, a VM in projectperimeter-network uses a command to move afile from one Cloud Storage bucket, located in projectcorp-resources-protected, to another bucket, located in projectcorp-resources-public.

The VM uses the following command:

gcloudstoragemvgs://corp-resources-private-1/yob2000.txtgs://corp-resources-public-1/babynames/

The command returns the following output:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{insertId:"1xxnssmd2hqo"logName:"projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"storage-accessing@example.iam.gserviceaccount.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/_/buckets/corp-resources-public-1"]violationReason:"NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"google.storage.BillingRequiredRead"requestMetadata:{callerIp:"130.211.225.66"callerNetwork:"//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/927005422713"serviceName:"storage.googleapis.com"status:{…}}receiveTimestamp:"2018-11-28T00:45:31.531623485Z"resource:{labels:{method:"google.storage.BillingRequiredRead"project_id:"perimeter-network"service:"storage.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-28T00:45:31.351140381Z"}

In this case, the log is less clear because the method listed isBillingRequiredRead and the action taken ismove. This is a limitation ofVPC Service Controls's present audit log functionality.

While the reason is less clear, this audit log record indicates thatsome resource in the request is outside of a perimeter that applies to therequest. In this case, that resource iscorp-resources-public.

Move Cloud Storage file outside perimeter

In this example, a VM in projectpublic-network uses a command to move afile from one Cloud Storage bucket, located in projectcorp-resources-protected, to another bucket, located in projectcorp-resources-public.

Thecorp-resources-protected project is protected by a service perimeter. Thepublic-network andcorp-resources-public projects exist outside theperimeter.

The VM uses the following command:

gcloudstoragemvgs://corp-resources-private-1/yob2000.txtgs://corp-resources-public-1/babynames/

The command returns the following output:

Copying gs://corp-resources-private-1/yob2000.txt [Content-Type=text/plain]...AccessDeniedException: 403 Request violates VPC Service Controls.

The following audit log record is generated:

{insertId:"10moqhsch9v"logName:"projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"user@example.biz"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/_/buckets/corp-resources-private-1/objects/yob2000.txt"1:"projects/_/buckets/corp-resources-public-1/objects/out.txt"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"google.storage.Write"requestMetadata:{callerIp:"2620:15c:2c4:203:63d6:5eb8:418d:c034"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/1004338142803"serviceName:"storage.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-30T16:34:46.948010626Z"resource:{labels:{method:"google.storage.Write"project_id:"corp-resources-private"service:"storage.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-30T16:34:46.898098978Z"}

In this example, the audit log indicates that one cannot copy data across theboundary of a service perimeter (both resources are in the audit log record).Recall that the request originates from outside the perimeter (the VM inpublic-network), and that one of the buckets exists outside the perimeter(corp-resources-public-1).

From outside the perimeter, one is able to write to bucketcorp-resources-public-1, so the check that failed in theprevious example passes. However, the subsequent checkto actually copy the data fails.

This example demonstrates how sometimes a single user operation results inmultiple internal operations that must pass VPC Service Controls enforcement.

BigQuery dataset copy from VM inside perimeter

In this example, a VM in project927005422713 (perimeter-network) tries tocopy a BigQuery dataset from projectcorp-resources-privatetocorp-resources-public (117961063178).perimeter-network andcorp-resources-private share a perimeter, whilecorp-resources-public existsoutside the perimeter.

The VM uses the following command:

bqcpcorp-resources-private:babynames.yob2000\corp-resources-public:babynames.yob2000

The command returns the following output:

BigQuery error in cp operation: VPC Service Controls: Request is prohibited byorganization's policy. Operation ID: c00dbc44-460f-4bd0-9d09-cda98ac800f9

The following audit log record is generated:

{insertId:"146o5fd2hbp"logName:"projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/117961063178"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"bigquery.googleapis.com/bigquery.tables.get"requestMetadata:{callerIp:"131.201.221.16"callerNetwork:"//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/927005422713"serviceName:"bigquery.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-28T00:27:05.688803777Z"resource:{labels:{method:"bigquery.googleapis.com/bigquery.tables.get"project_id:"perimeter-network"service:"bigquery.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-28T00:27:05.378584819Z"}

In this example, there is no single underlying API action that shows all theresources at play in this request due to limitations of the logging mechanismand the distributed architecture of BigQuery.

The audit log record indicates that the operation failed because in order tocopy the data, BigQuery must access the target project(corp-resources-public) using the network in projectperimeter-network(the source of the request). Recall thatcorp-resources-public is outsidethe perimeter that protectsperimeter-network. The request is denied as anattempt to exfiltrate data tocorp-resources-public.

This example illustrates that one conceptual operation, such as copying data,can trigger multiple attempts to access data from different storage systems,such as Cloud Storage, BigQuery, andBigtable. Based on how the operation is executed, the generatedaudit log record differs from the original user command. Also, when multiplechecks within a given service are made and potentially fail, the generated auditlog record looks different from the original user command.

Dataproc job reading from project

This example shows how to debug indirect VPC Service Controls errors thatoccur when using data processing services like Dataproc.

In this example, a Dataproc cluster is running in a projectprotected by VPC Service Controls.Hello-world.py is a pyspark job thatattempts to access data from Cloud Storage bucket inside the perimeterand then write it to another bucket that exists outside the perimeter.VPC Service Controls blocks the operation that writes data to a bucketoutside the perimeter.

The following command is used to executeHello-world.py:

gclouddataprocjobssubmitpysparkhello-world.py--clustertest-cluster-new2

The command returns the following output:

Job[50f16ca8-5102-442b-a545-eed5e4f5f5da]submitted.Waitingforjoboutput...18/11/2900:31:34INFOorg.spark_project.jetty.util.log:Logginginitialized@2552ms18/11/2900:31:34INFOorg.spark_project.jetty.server.Server:jetty-9.3.z-SNAPSHOT18/11/2900:31:34INFOorg.spark_project.jetty.server.Server:Started@2640ms18/11/2900:31:34INFOorg.spark_project.jetty.server.AbstractConnector:StartedServerConnector@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}18/11/2900:31:34INFOcom.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase:GHFSversion:1.6.4-hadoop218/11/2900:31:35INFOorg.apache.hadoop.yarn.client.RMProxy:ConnectingtoResourceManagerattest-cluster-new2-m/10.246.0.3:803218/11/2900:31:37INFOorg.apache.hadoop.yarn.client.api.impl.YarnClientImpl:Submittedapplicationapplication_1522454176466_0005Traceback(mostrecentcalllast):File"/tmp/50f16ca8-5102-442b-a545-eed5e4f5f5da/hello-world.py",line8,in<module>lear.saveAsTextFile("gs://corp-resources-public-1/out.txt")File"/usr/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py",line1553,insaveAsTextFileFile"/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py",line1133,in__call__File"/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py",line319,inget_return_valuepy4j.protocol.Py4JJavaError:Anerroroccurredwhilecallingo49.saveAsTextFile.:java.io.IOException:Erroraccessing:bucket:corp-resources-public-1,object:out.txtatcom.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.wrapException(GoogleCloudStorageImpl.java:1767)$sp(PairRDDFunctions.scala:961)(truncated)Causedby:com.google.api.client.googleapis.json.GoogleJsonResponseException:403Forbidden{"code":403,"errors":[{"domain":"global","message":"Request violates VPC Service Controls.","reason":"vpcServiceControls"}],"message":"Request violates VPC Service Controls."}atcom.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)(truncated)18/11/2900:31:43INFOorg.spark_project.jetty.server.AbstractConnector:StoppedSpark@1f1c18ec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}ERROR:(gcloud.dataproc.jobs.submit.pyspark)Job[50f16ca8-5102-442b-a545-eed5e4f5f5da]enteredstate[ERROR]whilewaitingfor[DONE].

Note the IO exception that occurs when thesaveAsTextFile method is called.Cloud Storage returns a403 error with the messageRequest violates VPC Service Controls. The error indicates that theCloud Storage audit log operation must be reviewed.

In the audit logs for theperimeter-network project, where the command wasexecuted, there is an audit log record for thesaveAsTextFile operation:

{insertId:"qdj1o9d1run"logName:"projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"1004338142803-compute@developer.gserviceaccount.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/_/buckets/corp-resources-public-1/objects/out.txt"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"google.storage.BillingRequiredRead"requestMetadata:{callerIp:"10.246.0.3"callerNetwork:"//compute.googleapis.com/projects/corp-resources-private/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/1004338142803"serviceName:"storage.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-29T00:31:43.666227930Z"resource:{labels:{method:"google.storage.BillingRequiredRead"project_id:"corp-resources-private"service:"storage.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-29T00:31:43.608250320Z"}

Due to audit log limitations, themethodName for Cloud Storageis listed asRead even though it is actually awrite operation. The auditlog record indicates that the operation failed because a network in projectcorp-resources-private was attempting to access the data (writing, in thiscase) of a resource in bucketcorp-resources-public-1. Due to the limitationsof the Cloud Storage audit log, it is not clear which project bucketcorp-resources-public-1 belongs to.

To identify the project that containscorp-resources-public-1, use thefollowing command:

gcloudstoragelsgs://corp-resources-public-1--buckets--log-http2>&1|grepprojectNumber

The command returns the following output:"projectNumber": "117961063178",

117961063178 is projectcorp-resources-public, which is outside perimeter.Thus, the failure is expected.

Error due to unsupported services

Some Google Cloud services depend on other Google Cloud services aspart of their implementation. If an unsupported service such asApp Engine is used inside a project that is protected by a perimeter, theservice's resources might not be accessible.

For information about known problematic cases, seeKnown service limitations.

Unsupported service with restricted VIP

Attempting to access an API that is not supported by the VPC Service Controlsrestricted VIP results in a403 error. For example, VPC Service Controlsdoes not support App Engine, so the App Engine Admin API isnot available when using the restricted VIP.

For example, suppose the following command is used to list all App Engineservices within a service perimeter:

gcloudappserviceslist

The command returns the following output:

ERROR: (gcloud.app.services.list) User [***] does not have permission to access apps instance [***] (or it may not exist): <!DOCTYPE html><html lang=en>  <meta charset=utf-8>  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">  <title>Error 403 (Forbidden)!!1</title>  <style>    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){ #logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}  </style>  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>  <p><b>403.</b> <ins>That's an error.</ins>  <p>Your client does not have permission to get URL <code>/v1/apps/***/services</code> from this server.  <ins>That's all we know.</ins>

This type of error is expected for services that are not supported byVPC Service Controls and not available on the restricted VIP. If this erroroccurs for a service thatis supported by VPC Service Controls, then werecommend that you check theknown servicelimitationsfor that service to see if it is a known limitation. Otherwise,the issue mustbe reported.

Log export to project outside perimeter

In this example, a log export is blocked by VPC Service Controls.The export destination, projectcorp-resources-public, is outside of theVPC Service Controls perimeter, while the sink is created on projectperimeter-network, which is inside the perimeter.

gcloudloggingsinksdescribeexample-sink

destination:bigquery.googleapis.com/projects/corp-resources-public/datasets/logsfilter:|-resource.type="audited_resource"resource.labels.service="bigquery.googleapis.com"name:example-sinkoutputVersionFormat:V2writerIdentity:serviceAccount:p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com

{insertId:"e5i2i8cbqw"logName:"projects/perimeter-network/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"p927005422713-439672@gcp-sa-logging.iam.gserviceaccount.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"corp-resources-public"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"google.cloud.bigquery.v2.TableDataService.InsertAll"requestMetadata:{callerIp:"2002:a49:8c51::"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/927005422713"serviceName:"bigquery.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-11-29T17:32:19.287138882Z"resource:{labels:{method:"google.cloud.bigquery.v2.TableDataService.InsertAll"project_id:"perimeter-network"service:"bigquery.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-11-29T17:32:19.054662413Z"}

bqextractbabynames.yob2000

gs://corp-resources-public-1/export.txtWaitingonbqjob_r47ee34109d02b41_000001676b27157c_1...(1s)Currentstatus:DONEBigQueryerrorinextractoperation:Errorprocessingjob'corp-resources-private:bqjob_r47ee34109d02b41_000001676b27157c_1':AccessDenied:BigQueryBigQuery:Permissiondeniedwhilewritingdata.

{insertId:"4gbh6pe8jld7"logName:"projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fdata_access"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"storage-accessing@example.iam.gserviceaccount.com"}methodName:"jobservice.jobcompleted"requestMetadata:{callerIp:"10.5.0.4"callerNetwork:"//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"callerSuppliedUserAgent:"google-api-python-client/1.6.5 (gzip),gzip(gfe)"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/corp-resources-private/jobs/bqjob_r47ee34109d02b41_000001676b27157c_1"serviceData:{@type:"type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData"jobCompletedEvent:{eventName:"extract_job_completed"job:{jobConfiguration:{extract:{destinationUris:[0:"gs://corp-resources-public-1/export.txt"]sourceTable:{datasetId:"babynames"projectId:"corp-resources-private"tableId:"yob2000"}}}jobName:{jobId:"bqjob_r47ee34109d02b41_000001676b27157c_1"location:"US"projectId:"corp-resources-private"}jobStatistics:{createTime:"2018-12-01T19:03:03.908Z"endTime:"2018-12-01T19:03:05.494Z"startTime:"2018-12-01T19:03:04.013Z"}jobStatus:{additionalErrors:[0:{code:7message:"Access Denied: BigQuery BigQuery: Permission denied while writing data."}]error:{code:7message:"Access Denied: BigQuery BigQuery: Permission denied while writing data."}state:"DONE"}}}}serviceName:"bigquery.googleapis.com"status:{code:7message:"Access Denied: BigQuery BigQuery: Permission denied while writing data."}}receiveTimestamp:"2018-12-01T19:03:05.532169998Z"resource:{labels:{project_id:"corp-resources-private"}type:"bigquery_resource"}severity:"ERROR"timestamp:"2018-12-01T19:03:05.503Z"}

{insertId:"1bq397kcfj1"logName:"projects/corp-resources-private/logs/cloudaudit.googleapis.com%2Fpolicy"protoPayload:{@type:"type.googleapis.com/google.cloud.audit.AuditLog"authenticationInfo:{principalEmail:"storage-accessing@example.iam.gserviceaccount.com"}metadata:{@type:"type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"resourceNames:[0:"projects/1004338142803"1:"projects/_/buckets/corp-resources-public-1"]violationReason:"RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER"}methodName:"google.storage.BillingRequiredRead"requestMetadata:{callerIp:"10.5.0.4"callerNetwork:"//compute.googleapis.com/projects/perimeter-network/global/networks/__unknown__"destinationAttributes:{}requestAttributes:{}}resourceName:"projects/1004338142803"serviceName:"storage.googleapis.com"status:{code:7details:[0:{@type:"type.googleapis.com/google.rpc.PreconditionFailure"violations:[0:{type:"VPC_SERVICE_CONTROLS"}]}]message:"Request is prohibited by organization's policy"}}receiveTimestamp:"2018-12-01T19:03:05.617451586Z"resource:{labels:{method:"google.storage.BillingRequiredRead"project_id:"corp-resources-private"service:"storage.googleapis.com"}type:"audited_resource"}severity:"ERROR"timestamp:"2018-12-01T19:03:05.420005215Z"}

{  "protoPayload": {    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",    "status": {      "code": 7,      "message": "Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: kmpY9Fgfuhgi2NE90lURjFWuiS1nGRqxCw4L12HdW8h46Un__-_LZw",      "details": [        {          "@type": "type.googleapis.com/google.rpc.PreconditionFailure",          "violations": [            {              "type": "VPC_SERVICE_CONTROLS",              "description": "kmpY9Fgfuhgi2NE90lURjFWuiS1nGRqxCw4L12HdW8h46Un__-_LZw"            }          ]        }      ]    },    "authenticationInfo": {      "principalEmail": "my-user@example.iam.gserviceaccount.com",      "serviceAccountKeyName": "//iam.googleapis.com/projects/my-project/serviceAccounts/my-user@example.iam.gserviceaccount.com/keys/<code><var>ACCOUNT_KEY</var></code>"    },    "requestMetadata": {      "callerIp": "gce-internal-ip",      "requestAttributes": {},      "destinationAttributes": {}    },    "serviceName": "cloudfunctions.googleapis.com",    "methodName": "google.cloud.functions.v1.CloudFunctionsService.ListFunctions",    "resourceName": "<code><var>PROJECT_ID_1</var></code>",    "metadata": {      "violationReason": "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER",      "resourceNames": [        "projects/<code><var>PROJECT_ID_2</var></code>/locations/-"      ],      "securityPolicyInfo": {        "servicePerimeterName": "accessPolicies/<code><var>ACCESS_POLICY</var></code>/servicePerimeters/us_sandbox",        "organizationId": "<code><var>ORGANIZATION_ID</var></code>"      },      "deviceState": "Unknown",      "vpcServiceControlsUniqueId": "kmpY9Fgfuhgi2NE90lURjFWuiS1nGRqxCw4L12HdW8h46Un__-_LZw",      "ingressViolations": [        {          "targetResource": "projects/<code><var>PROJECT_ID_1</var></code>",          "servicePerimeter": "accessPolicies/<code><var>ACCESS_POLICY</var></code>/servicePerimeters/<code><var>PERIMETER_NAME</var></code>",          "source": "<code><var>PROJECT_ID_2</var></code>"        }      ],      "@type": "type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"    }  },  "insertId": "tzf7fd103i",  "resource": {    "type": "audited_resource",    "labels": {      "service": "cloudfunctions.googleapis.com",      "method": "google.cloud.functions.v1.CloudFunctionsService.ListFunctions",      "project_id": "<code><var>PROJECT_ID_2</var></code>"    }  },  "timestamp": "2024-04-02T19:56:10.770681816Z",  "severity": "ERROR",  "logName": "projects/<code><var>PROJECT_ID_2</var></code>/logs/cloudaudit.googleapis.com%2Fpolicy",  "receiveTimestamp": "2024-04-02T19:56:11.463811603Z"}

Cloud NAT hasan integration withPrivate Google Accessthat automatically enables Private Google Access on the resource's subnet,and keeps the traffic to Google APIs and services internal as opposed torouting it to the internet using the Cloud NAT gateway externalIP address.

In this case, as the traffic is routed within the internal Google network, theRequestMetadata.caller_ip field of theAuditLog object is redacted togce-internal-ip. To fix this, instead of using the Cloud NATgateway external IP address in the access level forIP-based allowlist,configure an ingress rule to allow from the project or service account.

What's next

• Learn how toretrieve VPC Service Controls errors from audit logs.
• Learn how totroubleshoot common issues related to other Google Cloud services.
• Learn how todiagnose an access denial in violation analyzer and view its comprehensive evaluation report.