You create and manage agent policies by using thegcloud compute instances ops-agents policies commandgroup in the Google Cloud CLI or theops-agent-policy Terraform module.Agent policies use theVM Manager suite of tools inCompute Engine to manageOS policies,which can automate the deployment and maintenance of software configurationslike theOps Agent. These policies can't be applied to the legacyMonitoring agent or the legacy Logging agent.

Before you begin

Theops-agent-policy Terraform module is built on top of thegcloud compute instances ops-agents policies commands from the Google Cloud SDK. For information about how Terraform works, seeUsing Terraform.

Before using the Google Cloud CLI or the Terraform module to create agentpolicies, complete the following steps:

1. If you are going to use thegcloud compute instances ops-agents policies commands and ifyou haven't done so already, then install theGoogle Cloud CLI.

2. If you are going to use the Terraform module, then do the following:

   1. For information about installing Terraform, seeInstall and configureTerraform.Cloud Shell has Terraform already installed.

   2. Clone theterraform-google-cloud-operations repository, whichcontains theops-agent-policy module:

      git clone https://github.com/terraform-google-modules/terraform-google-cloud-operations

3. Download and run theprepare-for-ops-agents-policies.sh script toenable the required APIs and to set the proper permissions for using theGoogle Cloud CLI or Terraform.

   For information about the script, seeTheprepare-for-ops-agents-policies.shscript.

   Note: Theprepare-for-ops-agents-policies.sh script is designed to run on Linux. If you'reusing a machine on a different operating system, then run the script fromCloud Shell.

Uninstall the legacy Monitoring agent and Logging agent

Verify that the OS Config agent is installed

You might need to manually install and configure the OS Config agent onVMs that predate OS Config. For information about manually installingand verifying the OS Config agent, see theVM Manager verification checklist.

Find values for operating-system information

If you want to apply agent policies to specific operating systemsor versions, you need to know the values that OS Config uses to referto them.

To find values for theosShortName andosVersion fields for a VM, usethe following commands:

gcloud compute instances os-inventory describeINSTANCE_NAME \--zoneZONE | grep "^ShortName: "

gcloud compute instances os-inventory describeINSTANCE_NAME \--zoneZONE | grep "^Version: "

These commands require the OS Config agent to be installed on the VM.

Create an agent policy to manage the Ops Agent

gcloud compute instances ops-agents policiescreatePOLICY_ID \  --zoneZONE \  --filepath/to/policy-description-file.yaml \  --projectPROJECT_ID

agentsRule:packageState:installed|removedversion:latest|2.*.*|2.x.y

Verify the status of agent policies

This section describes how to check the status of created policiesand the installation of the Ops Agent. This information can also help withtroubleshooting your agent policies.

Compute EngineOS policies page

The Compute EngineOS policies page provides information aboutthe agent policies that manage the Ops Agent and about VMs on theVM instances tab. For example:

• TheState column indicates whether a policy was successfully installed ("Compliant"), in progress ("Pending"), possibly failed ("Unknown"), or missing ("No policies").
• TheVM monitored column indicates whether the Ops Agent is being managed by the OS Config ("Monitored") or not ("Not monitored").

  If a policy is "Compliant" but the VM shows "Not monitored", then there might be a problem installing the Ops Agent. You might, for example, have alegacy agent already installed.

In the Google Cloud console, go to theOS policies page:

Go toOS policies

If you use the search bar to find this page, then select the result whose subheading isCompute Engine.

TheVM instances on the Compute EngineOS policiestabshows information about agents managed by all OS policies in yourGoogle Cloud project. These policies are labeledgoog-ops-agent-policy.

• Thegoog-ops-agent-policy indicator includes several types of policies:
  • Policies created by using thegcloud compute instances ops-agents policies commands.
  • Policies created for you if you requestedOps Agent installation when you created the VM.
  • Policies created by using Terraform to manage the Ops Agent.

  To distinguish between policies, use theOS policy assignments tab on the page to see the policy IDs for all the policy assignments in your Google Cloud project.

• ThisVM monitored column doesn't reflect the installation of the Ops Agent by other means, like manual installation or bybeta agent policies.

Cloud MonitoringVM Instances page

TheVM Instances page in Cloud Monitoring includes anAgent column that lists the agent installed on each VM, and for theOps Agent, includes an indicator for installed agents that are older thanthe latest version.

In the Google Cloud console, go to theVM Instances page:

Go toVM Instances

If you use the search bar to find this page, then select the result whose subheading isMonitoring.

Example policy configurations

This section describes examples for setting up agent policies when usingeither the Google Cloud SDK or Terraform.

gcloud compute instances ops-agents policiescreatePOLICY_ID \  --zone us-central1-a \  --file agent-policy-description.yaml \  --projectPROJECT_ID

agentsRule:packageState:installedversion:latestinstanceFilter:all:True

agentsRule:packageState:removedversion:latestinstanceFilter:all:True

agentsRule:packageState:installedversion:latestinstanceFilter:inclusionLabels:-labels:env:prod-labels:app:web

agentsRule:packageState:installedversion:latestinstanceFilter:exclusionLabels:-labels:env:prodapp:web6inventories:-osShortName:debianosVersion:'11'

agentsRule:packageState:installedversion:2.*.*instanceFilter:inventories:-osShortName:rhelosVersion:'7.*'-osShortName:debianosVersion:'11'

Troubleshoot GA agent policies

This section provides information to help you resolve problems withGA agent policies for the Ops Agent. The information described inVerify the status of agent policies might also behelpful.

Theops-agents policy commands fail

When agcloud compute instances ops-agents policies command fails, the response shows avalidation error. Correct the errors by fixing the command arguments andflags as suggested by the error message.

In addition to the validation errors, you might see errors that indicate thefollowing conditions:

• Insufficient IAM permission
• The OS Config API isn't enabled
• The specified policy doesn't exist

The following sections describe these conditions in more detail.

Insufficient IAM permission

If agcloud compute instances ops-agents policies command fails with a permission error, thenmake sure you have run theprepare-for-ops-agents-policies.sh script as described inBefore you beginto set up the OS Config policy roles:

• OSPolicyAssignment Admin (roles/osconfig.osPolicyAssignmentAdmin): Provides full access to OS policy assignments.

• OSPolicyAssignment Editor (roles/osconfig.osPolicyAssignmentEditor): Allows users to get, update, and list OS policy assignments.

• OSPolicyAssignment Viewer (roles/osconfig.osPolicyAssignmentViewer): Provides read-only access to get and list OS policy assignments.

For more information about theprepare-for-ops-agents-policies.sh script, seeTheprepare-for-ops-agents-policies.sh script.

OS Config API isn't enabled

A sample error looks like the following:

API [osconfig.googleapis.com] not enabled on projectPROJECT_ID.Would you like to enable and retry (this will take a few minutes)?(y/N)?

You can entery to enable the API, or you can run theprepare-for-ops-agents-policies.shscript, described inBefore you begin,to grant all the necessary permissions. If you entery at theprompt in the error message, then you still need to run theprepare-for-ops-agents-policies.sh script to set the needed permissions.

To verify that the OS Config API is enabled for the project, run thefollowing commands:

gcloudserviceslist--projectPROJECT_ID|greposconfig.googleapis.com

The expected output follows:

osconfig.googleapis.com    Cloud OS Config API

The policy already exists

A sample error looks like the following:

ALREADY_EXISTS: Requested entity already exists

This error means this policy already exists with the same name, project ID andregion. You can use thegcloud compute instances ops-agents policiesdescribe command to confirm this.

The policy does not exist

A sample error looks like the following:

NOT_FOUND: Requested entity was not found

This error might meant that the policy was never created, the policy has beendeleted, or that the specified policy ID is incorrect. Make sure thePOLICY_ID used in agcloud compute instances ops-agents policiesdescribe,update ordelete command corresponds to an existing policy. To get a list of agentpolicies, use thegcloud compute instances ops-agents policieslist command.

The policy is created but seems to have no effect

OS Config agents are deployed to each Compute Engine instance to manage thepackages for the Logging and Monitoring agents.The policy may seem to have no effect if the underlying OS Config agent isn'tinstalled.

gcloud compute sshinstance-id \    --projectproject-id \    -- sudo systemctl status google-osconfig-agent

    google-osconfig-agent.service - Google OSConfig Agent    Loaded: loaded (/lib/systemd/system/google-osconfig-agent.service; enabled; vendor preset:    Active: active (running) since Wed 2020-01-15 00:14:22 UTC; 6min ago    Main PID: 369 (google_osconfig)     Tasks: 8 (limit: 4374)    Memory: 102.7M    CGroup: /system.slice/google-osconfig-agent.service            └─369 /usr/bin/google_osconfig_agent

    Status   Name               DisplayName    ------   ----               -----------    Running  google_osconfig_a… Google OSConfig Agent

If the OS Config agent is not installed, then you might be using anoperating system that doesn't support VM Manager. TheCompute EngineOperating system details documentindicates which VM Manager features are supported for eachCompute Engine operating system.

If the operating system supports VM Manager, then youcaninstall the OS Config agent manually.

The OS Config agent is installed but does not install the Ops Agent

To verify if there are any errors when the OS Config agent applies policies, youcan check the OS Config agent's log. This can be done either by usingLogs Explorer or using SSH or RDP to check individual Compute Engineinstances.

To view OS Config agent logs inLogs Explorer,use the following filter:

resource.type="gce_instance"logId(OSConfigAgent)

To view OS Config agent logs, do the following:

gcloudcomputesshINSTANCE_ID\--projectPROJECT_ID\--sudocat/var/log/messages\|grep"OSConfigAgent\|google-fluentd\|stackdriver-agent"

gcloudcomputesshINSTANCE_ID\--projectPROJECT_ID\--sudocat/var/log/syslog\|grep"OSConfigAgent\|google-fluentd\|stackdriver-agent"

If there is an error connecting to the OS Config service, make sure you run theprepare-for-ops-agents-policies.sh script as described inBefore you beginto set up the OS Config metadata.

To verify that the OS Config metadata is enabled, you can run the followingcommand:

gcloud compute project-info describe \    --projectPROJECT_ID \    | grep "enable-osconfig\|enable-guest-attributes" -A 1

The expected output follows:

- key: enable-guest-attributes  value: 'TRUE'- key: enable-osconfig  value: 'TRUE'

The Ops Agent is installed but not functioning properly

For more information about debugging Ops Agent problems, seeTroubleshoot the Ops Agent.

Enable debug-level logs for the OS Config agent

It can be useful to enable debug-level logging in the OS Config agent whenreporting an issue.

You can set theosconfig-log-level: debug metadata to enable debug-levellogging for the OS Config agent. The collected logs have more information tohelp with the investigation.

To enable debug-level logging for the entire project, run the following command:

gcloud compute project-info add-metadata \    --projectPROJECT_ID \    --metadata osconfig-log-level=debug

To enable debug-level logging for one VM, run the following command:

gcloud compute instances add-metadataINSTANCE_ID \    --projectPROJECT_ID \    --metadata osconfig-log-level=debug

Helper scripts

This section provides additional information about the helper scriptsdescribed in this document:

• Theprepare-for-ops-agents-policies.sh script
• Thediagnose_policies.sh script

Theprepare-for-ops-agents-policies.sh script

After you download theprepare-for-ops-agents-policies.sh script, youcan use the script to perform the following actions, based on the argumentsyou provide:

• Enable the Cloud Logging API, the Cloud Monitoring API, and theOS Config API for the project.

  If the OS Config API was not already enabled, it is enabled inlimited-feature mode, which is sufficient for managing agent policies.For more information, seePricing.

• Grant the Identity and Access Management rolesLogs Writer(roles/logging.logWriter) andMonitoring Metric Writer(roles/monitoring.metricWriter)to theCompute Engine default serviceaccountso that the agents can write logs and metrics to theLogging and Cloud Monitoring APIs.

• Enable the OS Config metadata for the project so that the OS Config agenton each VM is active.

• Grant one of the following IAM roles to the non-owner usersuser or service account needed to create and manage policies. Projectowners have full access to create and manage policies; all other usersor service accounts must be granted one of the following roles:

  • OSPolicyAssignment Admin (roles/osconfig.osPolicyAssignmentAdmin): Provides full access to OS policy assignments.
  • OSPolicyAssignment Editor (roles/osconfig.osPolicyAssignmentEditor): Allows users to get, update, and list OS policy assignments.
  • OSPolicyAssignment Viewer (roles/osconfig.osPolicyAssignmentViewer): Provides read-only access to get and list OS policy assignments.

  When running the script, you can specify the OSPolicyAssignment roles asadmin,editor, orviewer. The script maps those values to theroles/osconfig.osPolicyAssignment* role names.

The following examples show some common invocations for the script.For more information, see the comments in the script itself.

To enable the APIs, grant the necessary roles to the default service account,and enable the OS Config metadata for a project, run the script as follows:

bash prepare-for-ops-agents-policies.sh --project=PROJECT_ID

To additionally grant one of the OS Config roles to a user who doesnot have the Owner (roles/owner) role on the project, run the script asfollows:

bash prepare-for-ops-agents-policies.sh --project=PROJECT_ID \  --iam-user=USER_EMAIL \  --iam-policy-access=[admin|editor|viewer]

To additionally grant one of the OS Config roles to a non-defaultservice account, run the script as follows:

bash prepare-for-ops-agents-policies.sh --project=PROJECT_ID \  --iam-service-account=SERVICE_ACCT_EMAIL \  --iam-policy-access=[admin|editor|viewer]

Thediagnose_policies.sh script

Given a project ID, a Compute Engine instance ID, a Compute Engine zone, and the agent policy ID, thediagnose_policies.sh script automatically collectsthe necessary information to help diagnose issues with the policy:

• The OS Config agent version
• The underlying OS policy assignment
• The OS policy assignments that are applicable to thisCompute Engine instance
• A description of this Compute Engine instance

To invoke the script, run the following command:

bash diagnose_policies.sh --project-id=PROJECT_ID \   --gce-instance-id=INSTANCE_ID \  --policy-id=POLICY_ID \  --zone=ZONE

Pricing

Thegcloud compute instances ops-agents policies commands are implemented by usingOS policy assignmentresources fromVM Manager.Theprepare-for-ops-agents-policies.sh script, described inBefore you begin, setsup VM Manager inlimited-feature mode(OSCONFIG_B),which is sufficient for creating and managing agent policies. There is no costto use VM Manager in limited mode.

If you have configured VM Manager in full-feature mode(OSCONFIG_C), then you might incur costs.