This page provides instructions for troubleshooting common issues found withinstalling or interacting with the Logging agent.

Checklist

If you are having trouble installing or using the Logging agent,here are some things to check:

• If Linux installation commands result in errors, then make sure that youprefix the installation commands withsudo.

• Verify that the agent service is running on your VM instance:

  • For a Windows VM, use the following PowerShell command:

    Get-Service -Name StackdriverLogging

    Search for a service calledStackdriver Logging. If the agent is notrunning, you might need to restart it.

  • For a Linux VM, use the following command:

    sudo service google-fluentd status

    If the agent is not running, you might need to restart it using thefollowing command:

    sudo service google-fluentd restart

    If the restart fails, and the log output shows "Disabled via metadata",you are likely running an image fromGoogle Cloud Marketplace,where the Logging agent is disabled by default. Thegoogle-logging-enable instance metadata key controls the Logging agentenablement status, where a value of0 disables the agent. To re-enablethe agent, either remove thegoogle-logging-enable key or set itsvalue to1. For more information, seeCreate an instance with the logging agent disabled).

    If the agent is not disabled via metadata, reinstall the agent.See the following section,Reinstalling the Logging agent.

• See if the agent has written error messages to the logs.

  • On Windows, as of version v1-9, the Logging agent saves itslogs inC:\Program Files (x86)\Stackdriver\LoggingAgent\fluentd.log.

    There is no way to get the logs for prior versions of the agent.

  • On Linux, the Logging agent is afluentd package and logsmessages to/var/log/google-fluentd/google-fluentd.log:

    • If you see HTTP 429 errors, you might have exceeded yourLogging API quotas. You can see youravailable quota by selectingAPIs & services >Dashboard inthe Google Cloud console. Choose the Logging API.

    • If you see API access or authorization problems, go toVerifyingCompute Engine credentials.

• If the agent seems to be running normally, but you are not getting data,then you should check that the agent is sending data to the correct project.See the following section,Verifying Compute Engine credentials.

• If the agent fails to authorize, check whether the credentials for yourprivate key aremissing or invalid.

Verify the agent installation

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. At the top of the page, choose the project containing your VM instance:

   • ForCompute Engine VM instances, choose the Google Cloud projectthat contains the VM instance.

3. In thewindows tabs, choose the resource for your VM instance:

   • ForCompute Engine, chooseGCE VM Instance.
   • Selectsyslog (Linux),fluent.info (Windows), orAll logs.

If you see a log entry, "Successfully sent gRPC to Logging API,"then the agent installation is complete. This message generated once when theagent is installed and also each the agent is restarted.

For more information about the Logs Explorer,seeUsing the Logs Explorer.

Test the agent

If you suspect that the agent is not working, check that it is running and tryto send a test message to Logging:

Find your test message

After sending a test message, look for it in theLogs Explorer:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. At the top of the page, choose the project containing your VM instance:

   • ForCompute Engine VM instances, choose the Google Cloud projectthat contains the VM instance.

3. In thewindows tabs, choose the resource for your VM instance:

   • ForCompute Engine, chooseGCE VM Instance.
   • Selectsyslog (Linux),fluent.info (Windows), orAll logs.

4. You should see a log entry with your test message. If so, then the Loggingagent is operating correctly.

Verify Compute Engine credentials

For a Compute Engine VM instance to run the agent without private-keycredentials, the instance must have suitableaccessscopes and theservice account identitybeing used by the instance must have suitable IAM permissions.

When you create a VM instance, the default scope and service account settingsare sufficient to run the agents. Very old instances, or instancesfor which you have changed the default settings, might not have suitablecredentials.

Failure to load default credentials

In case there areCould not load the default credentials failures in theLogging log file, this implies the agent may be failing toconnect to the Compute Engine Metadata Server.

The error log looks like the following:

Startinggoogle-fluentd1.8.4:/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:74:in`get_application_default': Could not load the default credentials. Browse to (RuntimeError) https://developers.google.com/accounts/docs/application-default-credentials for more information.

One potential cause to this is if the VM has custom proxy setup. To fix this,refer to theProxy setup instructionto exclude the Compute Engine Metadata Server (metadata.google.internal, or169.254.169.254) from going through the proxy. If the error persists, thenremove the default Compute Engine service account from the VM and re-add it.

Verify access scopes

To verify the access scopes, do the following:

1. In the Google Cloud console, go to theVM instances page:

   Go toVM instances

   If you use the search bar to find this page, then select the result whose subheading isCompute Engine.

2. Click the name of your VM instance. The detail page for your instanceappears.

3. In theCloud API access scopes section, clickDetails to see thelist of APIs. Look for the following entries:

   1. If you see "This instance has full API access to all Google CloudServices," then your access scopes are adequate.
   2. If you see next toStackdriver Logging API, an older name forthe Cloud Logging API, that you haveWrite Only orFull permission, then your instance'saccess scopes are adequate for the Cloud Logging agent.
   3. If you see next toStackdriver Monitoring API, an older name forthe Cloud Monitoring API, that you haveWrite Only orFull permission, then your instance'saccess scopes are adequate for the Cloud Monitoring agent.

Correct the problem

If you do not have suitable access scopes in your Compute Engine instance,add the needed access scopes to yourinstance.

The following table shows the scopes relevant to the Logging andMonitoring agents:

Verify default service account permission

Even if your Compute Engine VM instance's access scopes are adequate, yourinstance's default service account might not provide the rightIAM permissions for the agent.

To verify the default service account permission, start by locating thedefault service account:

1. In the Google Cloud console, go to theVM instances page:

   Go toVM instances

   If you use the search bar to find this page, then select the result whose subheading isCompute Engine.

2. Click the name of your VM instance. The detail page for your instanceappears.

3. Look for theService account heading on the page. Thedefault service account for the instance is listed. It mightlook like the following:

   [ID]-compute@developer.gserviceaccount.com

4. In the Google Cloud console, go to theIAM page:

   Go toIAM

   If you use the search bar to find this page, then select the result whose subheading isIAM & Admin.

5. SelectView By: Principals. You should see a list of people, groups, andservice accounts. In theRole column are the roles each principalhas in your project.

6. In the row for your instance's default service account, you should see oneor more roles:

   • If you seeEditor, that role is adequate for all the agents.TheEditor role might automatically be granted to the defaultservice account depending on yourorganization policy configuration.
   • If you seeLogs Writer, that role is sufficient for the Loggingagent. For other Logging roles that include the write permission, seeAccess Control for Cloud Logging.
   • If you seeMonitoring Metric Writer, that role is sufficient for theMonitoring Agent. For other Monitoring roles that include the writepermission, seeAccess Control for Cloud Monitoring.

Correct the problem

If your default service account does not have adequate roles, try editing theroles for your service account in theIAM & admin > IAM page. Add the properLogging or Monitoringroles to authorize the agent(s):Logging > Logs WriterorMonitoring > Monitoring Metric Writer.

Verify private-key credentials

On Compute Engine VM instances, you can configure the agent to use a non-defaultservice account that has the proper authorization.

To configure the agent this way, you must create private-key credentials for thedesignated service account and give those credentials to the agent.

1. The agent looks for an environment variable,GOOGLE_APPLICATION_CREDENTIALS, which holds the name of a file whichcontains the private-key credentials.

2. If the environment variable is not present, thenthe agent will look for credentials in a default location:

   Linux

   /etc/google/auth/application_default_credentials.json

   Windows

   C:\ProgramData\Google\Auth\application_default_credentials.json

3. If the default location doesn't contain the credentials, the agent uses theapplication default credentials from metadata server.

sudo cat $GOOGLE_APPLICATION_CREDENTIALSsudo cat /etc/google/auth/application_default_credentials.json

{"type":"service_account","project_id":"[YOUR-PROJECT-ID]","private_key_id":"[YOUR-PRIVATE-KEY-ID]","private_key":"[YOUR-PRIVATE-KEY]","client_email":"[YOUR-PROJECT-NUMBER]-[YOUR-KEY]@developer.gserviceaccount.com","client_id":"[YOUR-CLIENT-ID]","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://accounts.google.com/o/oauth2/token","auth_provider_x509_cert_url":"{x509-cert-url}","client_x509_cert_url":"{client-x509-cert-url}"}

If there are no credential files present, then seeAdding credentials.

Credentials can be revoked using theIAM & Admin > Service accounts sectionof the Google Cloud console. If valid credentials aren't present, seeAdding credentialsto replace the existingcredentials or to add new ones.

If the service account is the correct one but the private key has been revoked,then you can create a new private key and copy it to your instance. SeeCreating service account keys.

Otherwise, you must create a new service account as described in the sectionAdding credentials.

Verify Log Exclusion queries

View yourcurrent exclusion queriesto ensure that the logs you are looking for are not accidentally excluded.

Verify Firewall

To see if your instance has access tologging.googleapis.com, run thefollowing Linux command on your instance:

curl -sSL 'https://logging.googleapis.com/$discovery/rest?version=v2' | head

The command can take some time to finish when the firewall blocks outboundtraffic. Sample output that indicates a firewall issue:

curl: (7) Failed to connect to 2607:f8b0:4001:c03::5f: Network is unreachable

VisitFirewall Rules forinformation on how to set up rules for outbound traffic.

Reinstall the agent

Installing the most recent version of the agent can solvemany problems:

• If you are sure that the problem is not related to credentials, you can skipaheadtoInstalling on Linux and Windows.

• For a full install of the agent and any needed credentials,seeInstall the Logging agent.

Other common issues

The following table lists some common problems that you may encounter with theCloud Logging agent and tells you how to fix them.

On Linux, the Logging agent records errors in/var/log/google-fluentd/google-fluentd.log. On Windows, the Logging agentrecords errors inC:\Program Files (x86)\Stackdriver\LoggingAgent\fluentd.log(starting at version v1-9).The error classGoogle::APIClient::ClientError indicates there is a problemwith permissions or API access.

You may start seeing errors after the agent has been running successfully.For example, someone might have revoked the required permissions from yourproject or your VM instance.

Limitation on log throughput

The maximum log throughput that the Logging agent can process isCPU bounded. CPU usage tends to grow when the log throughput grows. But theagent, with the default configuration, can use up to only one CPU core. So whenthe log throughput spikes, the agent might hit a CPU usage limit. If thesespikes are just temporary, the Logging agent buffers the logs andlater catches up to eventually process them. If the log throughput consistentlystays high, the logs might overflow the buffer and eventually are lost.

Typically, when each log entry is 1000 byte raw text and contains no additionalformat processing, the Logging agent hits the one core CPU limitat about 5,500 log entries per second. If the log entries require advancedprocessing, for example JSON or Regex parsing, the maximum log entries persecond might be lower.

If you need higher log throughput, you could consider using theOpsAgent. On Linux, for log entries thatare 1000 byte raw text and involve no additional processing, the Ops Agent canprocesses about 160,000 log entries per second.

Maximum log size exceeded

If one or more log entries exceeded the maximum size limit, then you might findentries in thefluentd logs similar to the following:

Dropping 1 log message(s) error_class="Google::Apis::ClientError" error="Invalid request"

or

Dropping 1 log message(s) error="3:Log entry with size 1000515 bytes exceeds maximum size of 112640 bytes" error_code="3"

To resolve this error, trim your log entries so that they don't exceed themaximum size limit. For example, the following sample code trims logs with tagmytag, with the data in the fieldmessage:

#CloudLoggingonlysupportslogentriesthatareupto256KiBinsize.#Trimtheentriestojustunderthatsizetoavoiddroppingthem.<filter[MY_TAG]>@typerecord_transformerenable_rubytrue<record>message${record['message'].length >256000?"[Trimmed]#{record['message'][0..256000]}...":record['message']}</record></filter>

Logs are duplicated

LogEntry.insertIDis added in the processing pipeline within the agent. IfinsertID is differentamong the duplicate logs, this indicates the logs are tailed from the log filesmultiple times. This could happen in the presence of log rotation, or when thepos file is missing or corrupted. To reduce the chance of this issue, ensure theposition files for anyin_tail input are not configured to be in the/var/logfolder or any other folders that may have log rotation enabled.

The logging pipeline also relies on theLogEntry.timestampfield to de-duplicate logs. Make sure the actual timestamp of the log entry isparsed properly. If Fluentd is not set up toparse the original timestamp from the log entry, then Fluentd uses the time whenit processes the log entry. So if the input is read multiple times, even thoughthe timestamp in the log line is the same, Fluentd may treat them as differentlog entries with different timestamps.

Repeated Audit log errors:Data points cannot be written more than 24h in the past

There is a known issue affecting versions 1.8.5 through 1.9.3 (inclusive) whichcauses logs like the following to appear repeatedly inData Access audit logs, when the agent hasbeen running for over 24 hours:

Field timeSeries[0].points[0].interval.end_time had an invalid value of "2021-10-20T20:16:34.010866-07:00": Data points cannot be written more than 24h in the past.

The solution is toupgrade your agentto 1.9.4 or later.

Unicode characters in logs get replaced by spaces or '�'

By default, thein_tail input expects the input files to be ASCII encoded, soit replaces any non-ASCII character by a space. To actually ingest UTF-8 encodedfiles, you need to supply two options in thein_tail configuration:

<source>@typetail…encodingUTF-8from_encodingUTF-8</source>

Both options are necessary. If only theencoding option is supplied, non-ASCIIcharacters in the ingested logs will be replaced by '�'.

Removed agent reported by Google Cloud console as installed

After you uninstall the agent, the Google Cloud console might take up to onehour to report this change.

Logging agent doesn't appear in Windows Uninstall a program list

To uninstall the Logging agent when it isn't listed in the Windows ControlPanel'sUninstall a program list, rununinstall.exe from the directorywhere you installed it.