This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Agent policies enable automated installation and maintenance of theGoogle Cloud Observability agentsacross a fleet of Compute Engine VMs that matchuser-specified criteria. You can create a policy for aGoogle Cloud project that governs existing and new VMs associated with thatGoogle Cloud project, ensuring properinstallation, uninstallation, and optionalauto-upgrade of all Google Cloud Observability agentson those VMs.

You create and manage agent policies by using thegcloud beta compute instances ops-agents policies commandgroup in the Google Cloud CLI or theagent-policy Terraform module.Agent policies use theVM Manager suite of tools inCompute Engine to manageOS policies,which can automate the deployment and maintenance of software configurationslike theGoogle Cloud Observability agents: the Ops Agent, the legacy Monitoring agent, and the legacy Logging agent.

Supported operating systems

You can apply an agent policy toCompute EngineVM instances running the operating systems shown in the following table:

Create an agent policy

This section describes using the Google Cloud SDK for managing agent policies.For information about using Terraform, seeTerraform integration.

To create an agent policy by using the Google Cloud CLI, complete the followingsteps:

1. If you haven't done so already, install theGoogle Cloud CLI.

   The agent policies described in this document use thebeta command group.

2. If you haven't done so already, install thebeta component of thegcloud CLI:

   gcloud components install beta

   To check if you have thebeta component for the installed, run:

   gcloud components list

   If you previously installed thebeta component, ensure you have thelatest version:

   gcloud components update

3. Download and use the following script to enable the APIs and to set theproper permissions for using the Google Cloud CLI:set-permissions.sh.

   For information about the script, refer toTheset-permissions.shscript.

   Note: Theset-permissions.sh script is designed to run on Linux. If you'reusing a machine on a different operating system, then run the script fromCloud Shell.

4. Use thegcloud beta compute instances ops-agents policiescreate commandto create a policy. For the syntax of the command, see thegcloud beta compute instances ops-agents policiescreatedocumentation.

   For examples showing how to format the command, see theExamplessection in the Google Cloud CLI documentation.

   For more information about the other commands in the command group andthe available options, see thegcloud beta compute instances ops-agents policies documentation.

Best practices for using agent policies

To control the impact to production systems during rollout, we recommend thatyou use instance labels and zones to filter the instances that the policyapplies to.

Here is an example of a phased rollout plan for Debian 11 VMs in a projectcalledmy_project:

Phase 1: Create a policy namedops-agents-policy-safe-rollout to install thelegacy Logging agent and Monitoring agent on all VMs with the labelsenv=test andapp=myproduct.

gcloudbetacomputeinstances\ops-agentspoliciescreateops-agents-policy-safe-rollout\--agent-rules="type=logging,version=current-major,package-state=installed,enable-autoupgrade=true;type=metrics,version=current-major,package-state=installed,enable-autoupgrade=true"\--os-types=short-name=debian,version=11\--group-labels=env=test,app=myproduct\--project=my_project

For more information about specifying the operating system, seegcloud beta compute instances ops-agents policiescreate.

Phase 2: Update that policy to target VMs in a single zone that have thelabelsenv=prod andapp=myproduct.

gcloud beta compute instances \    ops-agents policies update ops-agents-policy-safe-rollout \    --group-labels=env=prod,app=myproduct \    --zones=us-central1-c \

Phase 3: Update that policy to clear the zones filter so it rolls out globally

gcloud beta compute instances \    ops-agents policies update ops-agents-policy-safe-rollout \    --clear-zones

Policies on VMs that predate OS Config

You might need to manually install and configure the OS Config agent onVMs that predate OS Config. For information about manually installingand verifying the OS Config agent, see theVM Manager verification checklist.

Troubleshoot beta agent policies

This section provides information to help you resolve problems with beta agentpolicies for the Ops Agent, the legacy Monitoring agent, and the legacy Logging agent.

Theops-agents policy commands fail

When agcloud beta compute instances ops-agents policies command fails, the response shows avalidation error. Correct the errors by fixing the command arguments andflags as suggested by the error message.

In addition to the validation errors, you might see errors that indicate thefollowing conditions:

• Insufficient IAM permission
• The OS Config API isn't enabled
• The specified policy doesn't exist

The following sections describe these conditions in more detail.

Insufficient IAM permission

If agcloud beta compute instances ops-agents policies command fails with a permission error, thenmake sure you have run theset-permissions.sh script as described inCreate an agent policyto set up the OS Config policy roles:

• GuestPolicy Admin (roles/osconfig.guestPolicyAdmin): Provides full access to guest policies.
• GuestPolicy Editor (roles/osconfig.guestPolicyEditor): Allows users to get, update, and list guest policies.
• GuestPolicy Viewer (roles/osconfig.guestPolicyViewer): Provides read-only access to get and list guest policies.

For more information about theset-permissions.sh script, seeTheset-permissions.sh script.

OS Config API isn't enabled

A sample error looks like the following:

API [osconfig.googleapis.com] not enabled on projectPROJECT_ID.Would you like to enable and retry (this will take a few minutes)?(y/N)?

You can entery to enable the API, or you can run theset-permissions.shscript, described inCreate an agent policy,to grant all the necessary permissions. If you entery at theprompt in the error message, then you still need to run theset-permissions.sh script to set the needed permissions.

To verify that the OS Config API is enabled for the project, run thefollowing commands:

gcloudserviceslist--projectPROJECT_ID|greposconfig.googleapis.com

The expected output follows:

osconfig.googleapis.com    Cloud OS Config API

The policy already exists

A sample error looks like the following:

ALREADY_EXISTS: Requested entity already exists

This error means this policy already exists with the same name, project ID andregion. You can use thegcloud beta compute instances ops-agents policiesdescribe command to confirm this.

The policy does not exist

A sample error looks like the following:

NOT_FOUND: Requested entity was not found

This error might meant that the policy was never created, the policy has beendeleted, or that the specified policy ID is incorrect. Make sure thePOLICY_ID used in agcloud beta compute instances ops-agents policiesdescribe,update ordelete command corresponds to an existing policy. To get a list of agentpolicies, use thegcloud beta compute instances ops-agents policieslist command.

The policy is created, but seems to have no effect

OS Config agents are deployed to each Compute Engine instance to manage thepackages for the Logging and Monitoring agents.The policy may seem to have no effect if the underlying OS Config agent isn'tinstalled.

gcloud compute sshinstance-id \    --projectproject-id \    -- sudo systemctl status google-osconfig-agent

    google-osconfig-agent.service - Google OSConfig Agent    Loaded: loaded (/lib/systemd/system/google-osconfig-agent.service; enabled; vendor preset:    Active: active (running) since Wed 2020-01-15 00:14:22 UTC; 6min ago    Main PID: 369 (google_osconfig)     Tasks: 8 (limit: 4374)    Memory: 102.7M    CGroup: /system.slice/google-osconfig-agent.service            └─369 /usr/bin/google_osconfig_agent

    Status   Name               DisplayName    ------   ----               -----------    Running  google_osconfig_a… Google OSConfig Agent

If the OS Config agent is not installed, then you might be using anoperating system that doesn't support VM Manager. TheCompute EngineOperating system details documentindicates which VM Manager features are supported for eachCompute Engine operating system.

If the operating system supports VM Manager, then youcaninstall the OS Config agent manually.

The OS Config agent is installed, but it does not install the Logging agent

To verify if there are any errors when the OS Config agent applies policies, youcan check the OS Config agent's log. This can be done either by usingLogs Explorer or using SSH or RDP to check individual Compute Engineinstances.

To view OS Config agent logs inLogs Explorer,use the following filter:

resource.type="gce_instance"logId(OSConfigAgent)

To view OS Config agent logs, do the following:

gcloudcomputesshINSTANCE_ID\--projectPROJECT_ID\--sudocat/var/log/messages\|grep"OSConfigAgent\|google-fluentd\|stackdriver-agent"

gcloudcomputesshINSTANCE_ID\--projectPROJECT_ID\--sudocat/var/log/syslog\|grep"OSConfigAgent\|google-fluentd\|stackdriver-agent"

If there is an error connecting to the OS Config service, make sure you run theset-permissions.sh script as described inCreate an agent policyto set up the OS Config metadata.

To verify that the OS Config metadata is enabled, you can run the followingcommand:

gcloud compute project-info describe \    --projectPROJECT_ID \    | grep "enable-osconfig\|enable-guest-attributes" -A 1

The expected output follows:

- key: enable-guest-attributes  value: 'TRUE'- key: enable-osconfig  value: 'TRUE'

Observability agents are installed, but not functioning properly

For information about debugging specific agents, see the following documents:

• Troubleshoot the Ops Agent
• Troubleshoot the legacy Logging agent
• Troubleshoot the legacy Monitoring agent

Enable debug-level logs for the OS Config agent

It can be useful to enable debug-level logging in the OS Config agent whenreporting an issue.

You can set theosconfig-log-level: debug metadata to enable debug-levellogging for the OS Config agent. The collected logs have more information tohelp with the investigation.

To enable debug-level logging for the entire project, run the following command:

gcloud compute project-info add-metadata \    --projectPROJECT_ID \    --metadata osconfig-log-level=debug

To enable debug-level logging for one VM, run the following command:

gcloud compute instances add-metadataINSTANCE_ID \    --projectPROJECT_ID \    --metadata osconfig-log-level=debug

Helper scripts

This section provides additional information about the helper scriptsdescribed in this document:

• Theset-permissions.sh script
• Thediagnose.sh script

Theset-permissions.sh script

After you download theset-permissions.sh script, youcan use the script to perform the following actions, based on the argumentsyou provide:

• Enable the Cloud Logging API, the Cloud Monitoring API, and theOS Config API for the project.

• Grant the Identity and Access Management rolesLogs Writer(roles/logging.logWriter) andMonitoring Metric Writer(roles/monitoring.metricWriter)to theCompute Engine default serviceaccountso that the agents can write logs and metrics to theLogging and Cloud Monitoring APIs.

• Enable the OS Config metadata for the project so that the OS Config agenton each VM is active.

• Grant one of the following IAM roles to the non-owner usersuser or service account needed to create and manage policies. Projectowners have full access to create and manage policies; all other usersor service accounts must be granted one of the following roles:

  • GuestPolicy Admin (roles/osconfig.guestPolicyAdmin): Provides full access to guest policies.
  • GuestPolicy Editor (roles/osconfig.guestPolicyEditor): Allows users to get, update, and list guest policies.
  • GuestPolicy Viewer (roles/osconfig.guestPolicyViewer): Provides read-only access to get and list guest policies.

  When running the script, you only need to specify theguestPolicy*part of the role name. The script supplies theroles/osconfig.part of the name.

The following examples show some common invocations for the script.For more information, see the comments in the script itself.

To enable the APIs, grant the necessary roles to the default service account,and enable the OS Config metadata for a project, run the script as follows:

bash set-permissions.sh --project=PROJECT_ID

To additionally grant one of the OS Config roles to a user who doesnot have the Owner (roles/owner) role on the project, run the script asfollows:

bash set-permissions.sh --project=PROJECT_ID \  --iam-user=USER_EMAIL \  --iam-permission-role=guestPolicy[Admin|Editor|Viewer]

To additionally grant one of the OS Config roles to a non-defaultservice account, run the script as follows:

bash set-permissions.sh --project=PROJECT_ID \  --iam-service-account=SERVICE_ACCT_EMAIL \  --iam-permission-role=guestPolicy[Admin|Editor|Viewer]

Thediagnose.sh script

Given a project ID, a Compute Engine instance ID, and the agent policy ID, thediagnose.sh script automatically collectsthe necessary information to help diagnose issues with the policy:

• The OS Config agent version
• The underlying OS Config guest policy
• The policies that are applicable to this Compute Engine instance
• The agent package repositories that are pulled on to thisCompute Engine instance

To invoke the script, run the following command:

bash diagnose.sh --project-id=PROJECT_ID \   --gce-instance-id=INSTANCE_ID \  --policy-id=POLICY_ID

Terraform integration

To learn how to apply or remove a Terraformconfiguration, seeBasic Terraformcommands. For information about how Terraform works, seeUsing Terraform.

Terraform support for agent policies is built on top of the Google Cloud CLIcommands. To create an agent policy by using Terraform, follow the Terraformmoduleagent-policy instructions.You can also find example policies in theexamples directory.