Ingress mTLS CA cert invalid

You're viewingApigee andApigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Note: This document is applicable for Apigee hybrid users only.

Symptom

AnApigeeIssue withReasonAIS_INGRESS_MTLS_CA_CERT_INVALID is shown when viewing resources in theapigee namespace.

Error messages

After runningkubectl -n apigee get apigeeissues, an error like the following is displayed:

NAME                                    SEVERITY   REASON                             DOCUMENTATION                                                                                  AGEca-cert-invalid-my-org-my-virtualhost   ERROR      AIS_INGRESS_MTLS_CA_CERT_INVALID   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_INGRESS_MTLS_CA_CERT_INVALID   5h18m

Possible causes

CauseDescription
Ingress mTLS CA certificate is not PEM-encodedThe configured ingress mTLS CA certificate is not PEM-encoded.
Ingress mTLS CA certificate is malformedThe configured ingress mTLS CA certificate is malformed.

Cause: Ingress mTLS CA certificate is not PEM-encoded

The ingress mTLS CA certificate stored in the Kubernetes secret referenced by anApigeeRouteConfig is missing the PEM-encoding text blocks (or they are malformed).

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissueISSUE_NAME

WhereISSUE_NAME is the name of the issue. For example,ca-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:ca-cert-invalid-my-org-my-virtualhostNamespace:apigeeLabels:ais-reason=AIS_INGRESS_MTLS_CA_CERT_INVALIDAnnotations:<none>APIVersion:apigee.cloud.google.com/v1alpha1Kind:ApigeeIssueMetadata:CreationTimestamp:2023-06-12T17:03:43ZGeneration:1OwnerReferences:APIVersion:apigee.cloud.google.com/v1alpha2Kind:ApigeeOrganizationName:my-orgUID:7e83a52c-ce00-4bed-98be-55835ada1817ResourceVersion:3281563UID:adc775c2-376d-4bf9-9860-500b2b2b8273Spec:Details:CAcertificateinsecret"my-org-my-virtualhost"forApigeeRouteConfig"my-org-my-virtualhost"isnotPEMencodedDocumentation:https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_INGRESS_MTLS_CA_CERT_INVALIDReason:AIS_INGRESS_MTLS_CA_CERT_INVALIDSeverity:ERROREvents:<none>
Spec:Details:CAcertificateinsecret"my-org-my-virtualhost"forApigeeRouteConfig"my-org-my-virtualhost"isnotPEMencoded

The contents ofSpec.Details explains the name of theApigeeRouteConfig that refers to the Kubernetes secret containing the ingress mTLS CA certificate which is not PEM-encoded.

Resolution

  1. Usekubectl describe to display the name of theApigeeRouteConfig which has the ingress mTLS CA cert which is not PEM-encoded:

    kubectl -n apigee describe apigeeissueISSUE_NAME

    WhereISSUE_NAME is the name of the issue. For example,ca-cert-invalid-my-org-my-virtualhost.

    In this example, theApigeeRouteConfigmy-org-my-virtualhost is referencing the secretmy-org-my-virtualhost.

  2. Determine thevirtualhost name from theApigeeRouteConfig.

    TheApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.

    In this example,my-org is the Apigee organization and the virtualhost name ismy-virtualhost.

  3. Find the corresponding virtualhost in youroverrides.yaml.

    virtualhosts:...- name: my-virtualhost  selector:    app: apigee-ingressgateway  caCertPath: ./certs/ca.pem  sslCertPath: ./certs/vhost.pem  sslKeyPath: ./certs/vhost.key...

  4. Validate the contents of the file provided via the path declared incaCertPath. Ensure the content contains the required starting and ending certificate blocks. For example:

    -----BEGIN CERTIFICATE-----MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL... <contents omitted> ...eWJyaWQuZTJlLmFwaWdlZWtzLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCMMM=-----END CERTIFICATE-----
    Tip: Useopenssl to validate the CA certificate.

  5. Once the contents and format of the file have been fixed, apply the change to the virtualhost:

    helm upgradeENV_GROUP_NAME apigee-virtualhost/ \  --namespace apigee \  --set envgroup=ENV_GROUP_NAME \  -f overrides.yaml

Cause: Ingress mTLS CA certificate is malformed

The ingress mTLS CA certificate stored in the Kubernetes secret referenced by anApigeeRouteConfig is malformed.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissueISSUE_NAME

WhereISSUE_NAME is the name of the issue. For example,ca-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:ca-cert-invalid-my-org-my-virtualhostNamespace:apigeeLabels:ais-reason=AIS_INGRESS_MTLS_CA_CERT_INVALIDAnnotations:<none>APIVersion:apigee.cloud.google.com/v1alpha1Kind:ApigeeIssueMetadata:CreationTimestamp:2023-06-12T17:03:43ZGeneration:1OwnerReferences:APIVersion:apigee.cloud.google.com/v1alpha2Kind:ApigeeOrganizationName:my-orgUID:7e83a52c-ce00-4bed-98be-55835ada1817ResourceVersion:3281563UID:adc775c2-376d-4bf9-9860-500b2b2b8273Spec:Details:CAcertificateinsecret"my-org-my-virtualhost"forApigeeRouteConfig"my-org-my-virtualhost"isinvalidDocumentation:https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_INGRESS_MTLS_CA_CERT_INVALIDReason:AIS_INGRESS_MTLS_CA_CERT_INVALIDSeverity:ERROREvents:<none>
Spec:Details:CAcertificateinsecret"my-org-my-virtualhost"forApigeeRouteConfig"my-org-my-virtualhost"isinvalid

The contents ofSpec.Details explains the name of theApigeeRouteConfig that refers to the Kubernetes secret containing the ingress mTLS CA certificate which is malformed and failed to parse.

Resolution

  1. Usekubectl describe to display the name of theApigeeRouteConfig which has the ingress mTLS CA cert which is malformed:

    kubectl -n apigee describe apigeeissueISSUE_NAME

    WhereISSUE_NAME is the name of the issue. For example,ca-cert-invalid-my-org-my-virtualhost.

    In this example, theApigeeRouteConfigmy-org-my-virtualhost is referencing the secretmy-org-my-virtualhost.

  2. Determine thevirtualhost name from theApigeeRouteConfig.

    TheApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.

    In this example,my-org is the Apigee organization and the virtualhost name ismy-virtualhost.

  3. Find the corresponding virtualhost in youroverrides.yaml.

    virtualhosts:...- name: my-virtualhost  selector:    app: apigee-ingressgateway  caCertPath: ./certs/ca.pem  sslCertPath: ./certs/vhost.pem  sslKeyPath: ./certs/vhost.key...

  4. Validate the contents of the file provided via the path declared incaCertPath. Ensure the contents contain the entire certificate. For example:

    -----BEGIN CERTIFICATE-----MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQELBQAwQDE+MDwGA1UEAww1bWF4bWlsbGlvbi1oeWJyaWQtc3RhZ2luZy1kZXYuaHli... <contents omitted> ...0TPPaik8U9dtFXMGT1DJkjPRPO0Jw5rlU8DKlNA/Kkv52aKJZNwB/cwmvoa/BFjiPIPa9wY=-----END CERTIFICATE-----
    Tip: Useopenssl to validate the CA certificate.

  5. Once the contents and format of the file have been fixed, apply the change to the virtualhost:

    helm upgradeENV_GROUP_NAME apigee-virtualhost/ \  --namespace apigee \  --set envgroup=ENV_GROUP_NAME \  -f overrides.yaml

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contactGoogle Cloud Customer Care:

  1. The Google Cloud Project ID.
  2. The name of the Apigee hybrid organization.
  3. TheSpec.Details field of theApigeeIssue.
  4. (optional) File referenced bycaCertPath for the affected virtualhost.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-17 UTC.