This topic discusses steps you can take to troubleshoot and fix problems with themessage processor. The message processor is part of theapigee-runtime component. See alsoRuntime service configuration overview.

Readiness probe fails with HTTP status code 500

Symptom

One or moreapigee-runtime pods are not in the Ready state.

Error message

When you usekubectl to describe a failedapigee-runtime pod, you see the error:

For example:

kubectl describe pod -n hybrid \apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7...apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7Readiness probe failed: HTTP probefailed with statuscode: 500...

Possible causes

The error means that no activecontract is available for the message processor to serve the traffic. In this state, the message processor cannot call itself "ready".

Diagnosis: Synchronizer to management plane connection problem

To diagnose asynchronizer to management plane connection problem, do the following:

1. List the pods in the cluster:

   kubectl get pods -nnamespace

2. Open a shell in anapigee-synchronizer pod:

   kubectl exec -it -nnamespacesynchronizer-pod-name bash

   For example:

   kubectl exec -it -n apigee apigee-synchronizer-apigee-gcp-prod1-test-blue-cnj5x bash

3. Go to the following directory:

   cd/opt/apigee/var/log/apigee-synchronizerlsdr-xr-sr-x4apigeeapigee4096Sep1216:52750drwxr-sr-x2apigeeapigee4096Sep1216:52cachedFiles-rw-r--r--1apigeeapigee22295Sep1216:52config.log-rw-r--r--1apigeeapigee76Sep1216:52versions.properties

4. Check the active version inversion.properties. For example:

   cat versions.properties  #active repository version  #Sat Dec 14 19:45:00 GMT 2019  active.version=749

5. Be sure the value ofactive.version matches the name of the contract folder number. In the above example (also shown below), the folder name is 750; therefore, they do not match:

   dr-xr-sr-x 4 apigee apigee  4096 Sep 12 16:52750

6. Exit the pod shell.

Resolution

Ifversion.properties is missing, or if there is a version mis-match as explained above, check thesynchronizer logs to try to determine why the most current contracts are not being downloaded. For example:

kubectl logs -f -nnamespacesynchronizer-pod-name

For information on interpreting the synchronizer logs, seeSynchronizer logs. If the synchronizer is down, try restarting it usingapigeectl. For example:

$APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml --org --envenv-name

Diagnosis: Message processor to synchronizer connection problem

To diagnose amessage processor to synchronizer connection problem, do the following:

1. List the pods in the cluster:

   kubectl get pods -nnamespace

2. Check theruntime logs to try to figure out why the contracts are not being downloaded:

   kubectl logs -f -nnamespacepod-name

   For example:

   kubectl logs -f -n apigee apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7

   It is possible if the new MP comes up as part of an autoscale or pod restart, that the MP might not load the contracts. Generally, the problem occurs when the synchronizer is down, preventing the MP from loading the contracts. For example:

   2019-09-1316:59:24,331podName:N/AipAddress:N/AdpColor:N/AHttpClient@331886186-301DEBUGo.e.j.c.AbstractConnectionPool-AbstractConnectionPool$1.failed():Connection1/64creationfailedjava.net.UnknownHostException:apigee-synchronizer-apigee-gcp-prod1-test.hybrid.svc.cluster.localatjava.net.InetAddress.getAllByName0(InetAddress.java:1281)atjava.net.InetAddress.getAllByName(InetAddress.java:1193)atjava.net.InetAddress.getAllByName(InetAddress.java:1127)atorg.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:167)atorg.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)atorg.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)atjava.lang.Thread.run(Thread.java:748)atorg.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)atjava.lang.Thread.run(Thread.java:748)

Resolution

Try restarting the synchronizer. For example:

$APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml --org --envenv-name

Readiness probe fails for an invalid encryption key

Symptom

Theapigee-runtime pods are not in the Ready state.

Diagnosis

When you usekubectl to describe a failedapigee-runtime pod, you see this error:Readiness probe failed: Probe hybrid-encryption-key-validation-probe failed.For example:

$ kubectl describe pod -nnamespaceapigee-runtime-pod-name...Readiness probe failed:Probe hybrid-encryption-key-validation-probe failed due tocom.apigee.probe.model.ProbeFailedException{ code = hybrid.encryption.key.InvalidEncryptionKey,message = Invalid encryption key. Please check the length of the encryption key, associatedcontexts = []}...

Resolution

Supported encryption key lengths are 16 or 24 or 32 bytes and the key's value has to be base64 encoded. For more information on creating a properly formatted key, seeData encryption.

Viewing message processor logs

For details about viewing and interpreting message processor logs, seeRuntime logs.

Call an API proxy from the runtime pod

In some situations to help isolate a problem, you might want to check if you can make an API proxy call directly from inside theapigee-runtime pod, and thereby bypassing the ingress gateway.

1. Execute the following command to forward to port 8443. This allows you to call the API in the pod:

   kubectl port-forward -nnamespaceapigee-runtime-pod-name 8443:8443

2. Call a deployed API proxy. For example, where the proxy basepath isilove-apis:

   curl -k -v https://0:8443//ilove-apis    < HTTP/1.1 200 OK    < X-Powered-By: Apigee    < Access-Control-Allow-Origin: *    < X-Frame-Options: ALLOW-FROM RESOURCE-URL    < X-XSS-Protection: 1    < X-Content-Type-Options: nosniff    < Content-Type: text/html; charset=utf-8    < Content-Length: 18    < ETag: W/"12-Jb9QP1bUxNSmZkxQGt5KLQ"    < Date: Fri, 13 Sep 2019 18:33:46 GMT    < Via: 1.1 google    < X-Apigee.Message-ID: 016f5f7f-c59e-404c-86e8-7b0737833f982    < X-Apigee.dp.color: blue    < X-Apigee.proxy: /organizations/my-org/environments/test/apiproxies/i-loveapis/revisions/1    < X-Apigee.proxy.basepath: /ilove-apis    < X-Apigee.target-latency: 9    <    * Connection #0 to host 0 left intact    <H2>I <3 APIs

Check the management API

You can use the API described below to check if the management API is working properly.

1. Get the names of the pods in your cluster:

   kubectl get pods -nnamespace

2. Useport-forwarding to gain access theapigee-runtime pod. The syntax for port forwarding is as follows:

   kubectl port-forward -nnamespacepodname 8843:8843

   For example:

   kubectl port-forward -n apigee \    apigee-runtime-my-organization-test-blue-57965b7789-6j4bp 8843:8843

3. Then, in another terminal window, use a utility such ascurl to send a request to theclassification/tree API, as the following example shows:

   curl -k https://0:8843/v1/classification/tree

   Here is an example response, which lists information about the deployed proxies in the "test" environment:

   [ {  "condition" : "(always matches)",  "virtualHost" : {    "env" : "test",    "name" : "default",    "org" : "my-organization",    "tree" : {      "elements" : [ {        "application" : "myproxy",        "basePath" : "/myproxy",        "name" : "default",        "revision" : "1"      } ],      "name" : "IdentificationTree"    }  }} ]

Using DEBUG mode

To help with troubleshooting, you can enable DEBUG mode to include more detailed information in theapigee-runtime pod logs.

1. List the pods in your namespace:

   kubectl get pods -nnamespace

2. Pick any one of the listedapigee-runtime pods to debug.
3. Execute the port forwarding command for that pod. For example:

   kubectl port-forward -n hybrid apigee-runtime-hybrid-prod-blue-fcpdd 8843

4. Open another terminal and call the following API to enable debugging:

   curl "https://0:8843/v1/logsessions?sessions=debug" -X POST -v -k

5. Run thekubectl logs command to check the log that is in DEBUG Mode. For example:

   kubectl logs -f -n hybrid apigee-runtime-hybrid-prod-blue-fcpdd

6. When you are finished examining the DEBUG log, reset the log level to INFO (the default). For example:

   curl "https://0:8843/v1/logsessions?sessions=info" -X POST -v -k

7. Run thekubectl logs command to make sure the log is back in INFO mode.