Logs are a good way to troubleshoot problems with your Apigee hybridinstallation. SeeLogging for details.

kubectl logs -nAPIGEE_NAMESPACE apigee-synchronizer-apigee-example-test

Synchronizer log entries

The following table describes the fields that you will see inSynchronizer log files.

For example:

2019-02-0500:00:59,098org:hybridenv:testtarget:action:SYNCcontext-id:mode:Apigee-Timer-5INFOCONFIG-CHANGE-SyncContext.syncUpStream():Upstreamsyncstartedwithhttps://apigee.googleapis.com

Understanding Synchronizer logs

If you need to troubleshoot a problem with the Synchronizer,look at the log files. The following table may help youtroubleshoot errors may see in the Synchronizer logs. See alsoLogging.

Error troubleshooting

Logs are a good way to troubleshoot problems with yourinstallation. SeeLogging for details.

Synchronizer fails to start

Below are a few errors and their remediation explained. Each ofthese errors message start with an ErrorCode of type sync.xxxxx.Below are possible error codes

sync.NullOrEmptyConfig

Message: Null or empty config supplied viacom.apigee.sync.config

Reason: You will see this error message whenthe Synchronizer's configuration properties were not suppliedproperly.

Remedy: Make sure all configuration propertiesare populated with appropriate values.

sync.ConfigFileNotFound

Reason: The Synchronizer was set to read itsconfiguration properties from a file but the file could not beloaded

Remedy: make sure the file is available andaccessible during Synchronizer bootup

sync.IncorrectSettings

Reason: One or more required configurationproperties are missing from the Synchronizer configuration or theyvalues that are outside the expected range. The error should havemore details about the exact property that is causing the problem.These errors are in-built into the parser to make sure that andsetting errors are caught even before the service starts.

The following table lists some possible IncorrectSettingserrors:

Remedy: Fix the setting to include the requiredblock

sync.ReplicationRootDoesNotExist

Reason: The Synchronizer was supplied with areplication root which does not exist

Remedy: make sure the supplied root folderexists and is writeable

sync.ReadOnlyReplicationRoot

Reason: The Synchronizer was supplied with areplication root which do not have write permissions

Remedy: make sure the supplied root folderexists and is writeable

sync.ReplicationRootNotADir

Reason: The Synchronizer was supplied with areplication root which is possibly a file

Remedy: make sure the supplied root folderexists and is writeable

sync.InvalidControlPlaneURL

Reason: The Synchronizer was supplied with aconfig URL which is empty

Remedy: Make sure the supplied root folderexists and is writeable

sync.InvalidControlPlaneSubscriptionURL

Reason: The Synchronizer was supplied with an incorrect, incomplete, or empty config URL for subscription creation.

Remedy: Correct the URL to a valid config URL for pub/sub subscription creation to handle signals from control plane.

sync.InvalidControlPlaneUnsubscriptionURL

Reason: The Synchronizer was supplied with an incorrect, incomplete, or empty config URL for subscription deletion.

Remedy: Correct the URL to a valid config URL for pub/sub subscription deletion to handle signals from control plane.

sync.CannotCreateEnvironmentFolder

Reason: The Synchronizer tried to create afolder for replicating environment qa but could not create therequired folder

Remedy: make sure the replication root does nothave a conflicting path with the same name which is eitherread-only of a file

sync.DuplicateEnvironmentEntries

Reason: The same environment was added multipletimes in the configuration.

Remedy: Every environment can be specified onlyonce.

sync.DiskExceedsAllowedCapacity

Reason: The supplied storage disk is already atthreshold capacity supplied via the configuration properties.

Remedy: Make sure there is enough disk space orcheck if the storage settings can be tweaked to allow more diskspace.

sync.OwnershipClaimFailed

Reason: Synchronizer tried to claim ownershipon the environment folder but failed

Remedy: every Synchronizer will first try tomake sure it is a sole writer to the file system. The errorindicates the path and the reason for the failure. Address the rootcause

Synchronizer errors during runtime

The following Synchronizer errors can occur during runtime.

sync.replicators.PollReturnedError

Associated message will include the upstream status. Below arethe error statuses

status 0

Reason: Indicates a connectivity issue. Themanagement plane was not reachable

Remedy: Make sure the management plane detailsare correct and thecontracts API isreachable

status 404

Reason: Most likely the org/env name is notfound on the server

Remedy: Make sure the management plane detailsare correct and thecontracts API isreachable

status 401

Reason: Most likely credentials supplied do nothave the org admin privileges.

Remedy: Make sure the management planecredentials are correct and thecontracts API isreachable

status 403

Reason: Indicates the environment being polledfor is not set up as hybrid environment

Remedy: Make sure the org has the required flag"features.hybrid.enabled=true"

sync.OwnershipClaimedByOther

Reason: During the operation Synchronizer foundthat the file root is being accessed by another instance ofSynchronizer. (This can happen when a file system is shared amongSynchronizers.)

Remedy: Address the root problem to make sureno two Synchronizer instances share the same file system. TheSynchronizer would have marked itself down and would be in anunrecoverable position here. It would also make sure the otherSynchronizer which is accessing the same file root gets notifiedabout it via their ownership files

sync.DiskExceedsAllowedCapacity

Reason: During the operation Synchronizerrealized that the disk occupancy has exceeded the allowedlimit.

Remedy: At this point Synchronizer would abortany downloads and try a stale file cleanup operation. However thismay become a recurring condition and the only way to recovercompletely is to make sure the Synchronizer has enough disk memoryto operate.

Diagnose contract download failures

If you see "upstream contract poll" in an error message, trycalling the contract API and check the details, where username isyour Apigee username (your email address). The contract representsthe entire collection of data that the Synchronizer downloads fromthe management plane.

curl https://apigee.googleapis.com/v1/organizations/org/environments/env/contracts/1.0 -uusername

curl https://$CONTROL_PLANE_LOCATION-apigee.googleapis.com/v1/organizations/org/environments/env/contracts/1.0 -uusername