When multiple authorization headers are present in a request, the Apigee ingress gateway doesn't concatenate them into a single header. This results in the Message Processor returning a500 error with a"Duplicate Header "authorization"" message.

Workaround: If multiple authorization headers are present in the request, concatenate them into a single header. For example:

curl -s https://my-hostname.net/mocktarget/echo \          -H 'authorization: a, b'

Apigee Model Armor policies, including SanitizeUserPrompt and SanitizeModelResponse, currently do not function as expected when used in API proxies handling WebSocket connections. See WebSockets with Apigee for more information on WebSocket configuration.

Following the upgrade to1-16-0-apigee-3, some Apigee organizations may encounterHTTP 500 errors stemming from ajava.lang.NoClassDefFoundError within a Java Callout policy. The error appears similar to the following:

{"fault":{"faultstring":"Failed to execute JavaCallout. org/apache/commons/lang/StringUtils","detail":{"errorcode":"steps.javacallout.ExecutionError"}}}

This error occurs when a Java Callout policy incorrectly depends on libraries intended for Apigee's internal use instead of providing its own dependencies.

Action required: To prevent this issue, developers must ensure that all Java Callouts are self-contained and utilize their own dedicated libraries, avoiding any reliance on Apigee's internal class path.

In the API proxy that utilizes the Apache Commons JAR library, import the Apache Commons JAR file to include as a resource for the proxy. For more information, seeJava resource guidelines.

Upgrading theapigee-org chart can fail if themintTaskScheduler.serviceAccountPath property is not set, even if Monetization is not enabled:

Error: UPGRADE FAILED: execution error at (apigee-org/templates/mint-task-scheduler-gsa-secret.yaml:12:63): mintTaskScheduler.serviceAccountPath is required!

Workaround: Remove theapigee-org/templates/mint-task-scheduler-gsa-secret.yaml andapitee-org/templates/mint-task-scheduler-sa.yaml files.

You can optionally move the files to a separate location outside of the Helm chart directory. Alternatively you can download them again if you want to enable Monetization in the future, following the instructions inStep 2: Download the Apigee Helm charts.

For example, from thehelm-charts/ directory:

1. ls apigee-org/templates/

   Output:

   apigee-org-guardrails.yaml              mart-sa.yamlapigee-proxy-chaining-certificate.yaml  mint-task-scheduler-gsa-secret.yamlapigee-proxy-chaining-route.yaml        mint-task-scheduler-sa.yamlax-hash-salt-secret.yaml                NOTES.txtconnect-agent-gsa-secret.yaml           organization.yamlconnect-agent-sa.yaml                   udca-gsa-secret.yamldata-encryption-secret.yaml             udca-sa.yamlencryption-keys-secret.yaml             watcher-gsa-secret.yaml_helpers.tpl                            watcher-sa.yamlmart-gsa-secret.yaml

2. (Optional:)

   cp apigee-org/templates/mint-task-scheduler-gsa-secret.yaml /tmp/cp apigee-org/templates/mint-task-scheduler-sa.yaml /tmp/

3. rm apigee-org/templates/mint-task-scheduler-gsa-secret.yaml

4. rm apigee-org/templates/mint-task-scheduler-sa.yaml

5. ls apigee-org/templates/

   Output:

   apigee-org-guardrails.yaml              mart-gsa-secret.yamlapigee-proxy-chaining-certificate.yaml  mart-sa.yamlapigee-proxy-chaining-route.yaml        NOTES.txtax-hash-salt-secret.yaml                organization.yamlconnect-agent-gsa-secret.yaml           udca-gsa-secret.yamlconnect-agent-sa.yaml                   udca-sa.yamldata-encryption-secret.yaml             watcher-gsa-secret.yamlencryption-keys-secret.yaml             watcher-sa.yaml_helpers.tpl

Theapigee-pull-push.sh script can return aNo such image error message, for example:

Error response from daemon: No such image: gcr.io/apigee-release/hybrid/apigee-stackdriver-logging-agent:latest

Workaround: Edit theHELM_CHARTS_DIR/apigee-operator/etc/tools/apigee-pull-push.sh script to change line 114 in thedocker_tag()function from:

  docker tag "${source}/$i" "${dest}/$i:${TAG}"

To:

  docker tag "${source}/$i:${TAG}" "${dest}/$i:${TAG}"

Whenmetrics.serviceAccountRef ormetrics.serviceAccountSecretProviderClass is specified in the overrides.yaml file, the telemetry role will target the wrong service account.

Workaround: Patch_helper.tpl in theapigee-operator/ chart and reapply it.

1. Editapigee-operator/templates/_helpers.tpl and remove the followingbold lines:

   {{- define "metricsSA" -}}  {{- $metricsName := "apigee-metrics" }}  {{- $telemetryName := "apigee-telemetry" -}}  {{- $generatedName := include "orgScopeEncodedName" (dict "name" .Values.org) -}}  {{- if .Values.gcp.workloadIdentity.enabled -}}  {{- printf "%s-sa" $metricsName -}}{{- else if .Values.serviceAccountSecretProviderClass -}}<-- DELETE  {{- .Values.serviceAccountSecretProviderClass -}}<-- DELETE  {{- else if .Values.metrics.serviceAccountRef -}}<-- DELETE  {{- .Values.metrics.serviceAccountRef -}}<-- DELETE  {{- else if .Values.multiOrgCluster -}}  {{- printf "%s-%s" $metricsName $generatedName -}}  {{- else -}}  {{- printf "%s-%s" $metricsName $telemetryName -}}  {{- end -}}{{- end -}}

   The resulting section should look like the following:

   {{- define "metricsSA" -}}  {{- $metricsName := "apigee-metrics" }}  {{- $telemetryName := "apigee-telemetry" -}}  {{- $generatedName := include "orgScopeEncodedName" (dict "name" .Values.org) -}}  {{- if .Values.gcp.workloadIdentity.enabled -}}  {{- printf "%s-sa" $metricsName -}}  {{- else if .Values.multiOrgCluster -}}  {{- printf "%s-%s" $metricsName $generatedName -}}  {{- else -}}  {{- printf "%s-%s" $metricsName $telemetryName -}}  {{- end -}}{{- end -}}

2. Reapply theapigee-operator chart.

   helm upgrade operator apigee-operator/ \    --namespaceAPIGEE_NAMESPACE \    --atomic \    -foverrides.yaml

For a workaround, follow the steps inKnown issue 416634326.

The use of wildcards (*) in Apigee proxy basepaths may conflict with other explicit basepaths, resulting in a404 error. For example, use of the following basepaths for two proxies deployed in the same environment may result in a404 error when callingProxy-2:

Proxy-1: /a/v1/bProxy-2: /a/*/c

In this case, calls to the explicit base path will resolve successfully, but calls to theProxy-2 may return a404, ifthe wildcard path is evaluated as/a/v1/c.

Fixed: This issue is fixed in Apigee and in hybrid 1.14.3 and later. However, the fix is not enabled by default. If you wish to enable the use of wildcards in basepaths:

• Customers with Apigee orgs should reach out to Apigee support to enable the fix if needed.
• Customers with Apigee hybrid orgs should use the following procedure to enable the fix if needed.

Procedure: To enable use of wildcards (*) in Apigee proxy basepaths in Apigee hybrid:

1. Add the following stanza to youroverrides.yaml file before upgrading to hybrid 1.14.3 or later:

   runtime:  cwcAppend:    conf_message-processor-communication_classificationV2.enabled: "true"

2. Apply the changes to theapigee-env chart. Repeat the change for each environment in your installation.

   helm upgradeENV_RELEASE_NAME apigee-env/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --set env=ENV_NAME \  -fOVERRIDES_FILE

   • ENV_RELEASE_NAME is a name used to keep track of installation and upgrades of theapigee-env chart. This name must be unique from the other Helm release names in your installation. Usually this is the same asENV_NAME. However, if your environment has the same name as your environment group, you must use different release names for the environment and environment group, for exampledev-env-release anddev-envgroup-release. For more information on releases in Helm, seeThree big concepts in the Helm documentation.
   • ENV_NAME is the name of the environment you are upgrading.
   • OVERRIDES_FILE is your overrides file.
3. Upgrade to hybrid 1.14.3 or later.

LogTimer usage in SecurityPolicy can cause a memory leak.

In some circumstances, Apigee hybrid's logger threads could consume all available memory. For example, frequent log entries documenting permissions errors associated with the Javacallout could cause OOM.

The Apigee Extension Processor does not support more than 100kb of data processing as part of request and response body events.

When updating a keystore or truststore without creating a new keystore or truststore, runtime updates may fail and cause the following intermittent error:

  {"fault":{"faultstring":"SSL Handshake failed sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","detail":{"errorcode":"messaging.adaptors.http.flow.SslHandshakeFailed"}}}

The error is intermittent because the keystore or truststore update may fail on one runtime pod but succeed on other pods. To avoid this issue, update the keystore or truststore by creating a new keystore or truststore. Repoint your reference to the new keystore or truststore, as described in When a cert expires.

Starting with Apigee hybridversion 1.14.0, automatic addition of Zipkin trace headers (x-b3-*) has been removed.

N/A

If you only specifyhttpProxy, you must also ensure that*.googleapis.com is allowlisted for your Apigee Runtime pods to enable connectivity to these services.

SeeConfigure forward proxying for API proxies.

Presence ofistio.io Custom Resource Definitions (CRDs) in an Apigee hybrid cluster may cause failure inapigee-ingressgateway-manager pods.

During Apigee Hybrid upgrade from older versions to 1.14.2 or later, presence of existingistio.io CRDs may cause failed readiness probes in thediscovery containers of theapigee-ingressgateway-manager pods.

Workaround: Theistio.io CRDs are not required by Apigee hybrid v1.14.2 or newer. There are two options to fix this issue:

• Delete theistio.io CRDs if you are not using Istio for any purpose other than Apigee in your cluster.
• Update theapigee-ingressgateway-manager clusterrole to add permissions foristio.io.

Afer each of the above options, you will need to restart yourapigee-ingressgateway-manager pods.

After you have completed the above options, you will need to restart yourapigee-ingressgateway-manager pods.

1. List theingress-manager pods to reinstall or recreate:

   kubectl get deployments -nAPIGEE_NAMESPACE

   Example output:

   NAME                            READY   UP-TO-DATE   AVAILABLE   AGEapigee-controller-manager       1/1     1            1           32dapigee-ingressgateway-manager   2/2     2            2           32d

2. Restart theingress-manager pods:

   kubectl rollout restart deployment -nAPIGEE_NAMESPACE apigee-ingressgateway-manager

3. After a few minutes, monitor theapigee-ingressgateway-manager pods:

   watch -n 10 kubectl -nAPIGEE_NAMESPACE get pods -l app=apigee-ingressgateway-manager

   Example output:

   NAME                                             READY   STATUS    RESTARTS   AGEapigee-ingressgateway-manager-12345abcde-678wx   3/3     Running   0          10mapigee-ingressgateway-manager-12345abcde-901yz   3/3     Running   0          10m

ApigeeTelemetry can become stuck increating state.

This issue has been seen on OpenShift installations.

Workaround: Modify theapigee-operator chart template to create the correct clusterrole access.

1. Edit thehelm-charts/apigee-operator/templates/apigee-operators.yaml template file, and locate the definition of the-apigee-manager-role- clusterrole. It starts with:

   kind: ClusterRolemetadata:  name:apigee-manager-role-{{ include 'namespace' }}rules:  ...

2. Find the- apiGroups: apiregistration.k8s.io block, and add theapiservices/finalizers resource to the list of resources:

   - apiGroups:  - apiregistration.k8s.io  resources:  - apiservices  -apiservices/finalizers  verbs:  - create  - delete  - get  - patch  - update

3. Find the- apiGroups: authorization.k8s.io block, and add the- apiGroups: apigee.cloud.google.com block after the end of the block with the following text:

   - apiGroups:  - apigee.cloud.google.com  resources:  - apigeetelemetries/finalizers  verbs:  - get  - patch  - update

   For example:

   - apiGroups:  - authorization.k8s.io  resources:  - subjectaccessreviews  verbs:  - create  - get  - list- apiGroups:- apigee.cloud.google.comresources:- apigeetelemetries/finalizersverbs:- get- patch- update

4. Apply the changes to theapigee-operator chart:

   Dry run:

   helm upgrade operator apigee-operator/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --atomic \  -fOVERRIDES_FILE \  --dry-run=server

   upgrade the chart:

   helm upgrade operator apigee-operator/ \  --install \  --namespaceAPIGEE_NAMESPACE \  --atomic \  -fOVERRIDES_FILE \

APIproducts are limited to 50 paths. If you add additional paths, an error message is displayed:Operation group limit of 51 exceeded in Operation Config

Workaround: Use wildcard patterns to combine resource paths and operations as described in Configuring resource paths.

Workaround: Create multiple API Products. This workaround has been verified and is viable for those who add a lot of API Proxies to an API Product. The App that is associated with your API Products would need to be updated to include the new one, but credentials and client-side request details have been proven to not need changing.

Error using Custom Reports and Stats APIs when selecting by fee type for installations using Monetization.

Workaround: Fetch all fee types on the client side before filtering.

Scaling of Istio ingress pods can cause a503 error.

Scaling of the Istio ingress pods can occasionally cause a503 error. If a503 error occurs, logs for the load balancer display the following message:statusDetails: backend_connection_closed_before_data_sent_to_client.

Workaround: Manually scale the Istio ingress pods.

ESS andnon-ESS Cassandra credential rotation does not work in organizations with Enhanced Proxy Limits.

Runtime traffic is not affected.

Backups taken from the multi-region setup usingGCP andHYBRID Cloud Providers contain information about all the Hybrid regions that existed when the snapshot was taken. And, since those regions do not exist anymore, the restore job will conflict with the current state of Cassandra and fail with the following error:Unrecognized strategy option passed to NetworkTopologyStrategy.Note: If at least one Hybrid region is still alive and healthy, you should follow theregion recovery guide instead of restoring Cassandra from a snapshot.

TheNimbus JOSE + JWT library may cause ajava.lang.ClassCircularityError when using a JavaCallout policy .

If you have a Apigee hybrid-enabled organization with a JavaCallout policy that uses the Nimbus JOSE + JWT library, do not upgrade to hybrid 1.12.4, hybrid 1.13.3, hybrid 1.14.1.

ESS andnon-ESS Multi-region Cassandra credential rotation will fail in every region except the first.

Runtime traffic is not affected.

Follow the providedworkaround to resolve this issue.

Cassandra pods will be restarted after applying Cassandra overrides changes, such when re-enabling backup, which will trigger this issue. The logs of a Cassandra pod in the CrashLoopBackoff state will show the following error:Cannot change the number of tokens from 512 to 256.

Follow the providedworkaround to resolve this issue.

In some rare cases, the Apigee backup job doesn't clean up the Cassandra intermediate snapshots it creates while taking backups usingHYBRID orGCP Cloud Providers. This only happens if an underlying issue prevents the backup process from successfully connecting to the remote server or Cloud Storage. If the connection issue persists, those leftover Cassandra snapshots can accumulate over time, utilizing storage on Cassandra's disks. If you're affected, please fix the underlying connection issue and then follow the steps provided in theCassandra troubleshooting guide to clear Cassandra snapshots manually.

When a KeyValueMapOperations policy is used withapiproxy scope, and the policy's<Put> operation is called in a shared flow via a flow hook, the KVM entry is created under the shared flow name. It is expected to be created under the API proxy name.

384937220

When there are multiple virtual hosts, the Helm release creation may fail due to conflictingApigeeRoute names. The workaround is to run the following commands for every virtual host when creating:

kubectl annotate ar apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -nAPIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwritekubectl annotate cert apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -nAPIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwrite

where:

• PROJECT_ID_SUFFIX is a unique suffix for internal chaining for your project in Kubernetes. You can find this suffix with the following command:

  kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chaining

  Your output will look something like:

  kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chainingapigee-ingressgateway-internal-chaining-my-project--1234567    ClusterIP  34.118.226.140  <none>    15021/TCP,443/TCP    5d6h

  In the example output,my-project--1234567 is thePROJECT_ID_SUFFIX.

• APIGEE_NAMESPACE is your Apigee namespace.
• NEW_ENV_GROUP_NAME is the name the additional environment group. Update this value for each virtual host.

N/A

If a user provides an invalid service account to the UpdateControlPlaneAccess api, the operation goes on a retry loop effectively locking the organization from invoking the API until the operation times out.

When using theGCP Cloud Provider, the Apigee backup job is unable to upload to Cloud Storage buckets with retention policies. Backup files may be left in the Cloud Storage bucket with 0 byte file size.

Workaround: Disable retention policies on the Cloud Storage bucket.

341099433

apigee-logger utilizes Google IAM service accounts for shipping logs to Cloud Logging. This is due to FluentBit's lack of support for Workload Identity Federation, which prevents theapigee-logger from utilizing this feature.

N/A

N/A

270574696

268104619

364872027

For Apigee and Apigee hybrid versions 1.13 and higher, any deviations in the required PEM format of keys used inApigee JWS or JWT policies may result in a parsing error. For example, placing any character other an a newline (/n) immediately before the"-----END" line (post-encapsulation boundary) is not allowed and will result in an error.

To prevent this error, make sure that no characters other than a newline, such as trailing spaces or slashes, immediately precede the post-encapsulation boundary.

For more information about the encoding used for public or private keys, seeIETF RFC 7468.

310191899

The following endpoints may experience timeouts when used with a high volume of queries per second (QPS):

• organizations.environments.apis.revisions.
  deployments.deploy
• organizations.environments.apis.revisions.
  deployments.undeploy
• organizations.environments.sharedflows.revisions.
  deployments.deploy
• organizations.environments.sharedflows.revisions.
  deployments.undeploy

To reduce the likelihood of timeouts, we recommend setting a target of 1 QPS when using these endpoints or checking the status of a deployment before attempting another deployment.

329304975

Apigee is enforcing a temporary limit of 1000 basepaths per environment to avoid potential failures when deploying API proxy revisions.

While this limit is in place, you can deploy up to 1000 API proxy revisions (each containing a single basepath) per environment. If your API proxies or revisions contain more than one basepath, the total number of basepaths per environment must not exceed 1000.

333791378

For the steps required to install a patch for the workaround, see Troubleshooting.

310384001

289583112

If the OASValidation policy specifies an <OASResource> with security requirements set at aglobal level, the security requirements are not enforced.

Workaround: To ensure enforcement, all security requirements must be set at theoperation level in the OpenAPI specification passed in the<OASResource> element of the OASValidation policy.

205666368

SeeAbout setting TLS options in a target endpoint or target server.

295929616

Installing or upgrading to Apigee hybrid 1.10.0 through 1.10.2 could fail on OSE due to out-of-memory issues. Fixed in Apigee hybrid version 1.10.3.

292118812

If the firewall forces all traffic through a forward-proxy,apigee-udca may go into a crash-loop backoff state.

292558790

• The OASValidation Policy fails when the JSON content does not match the anticipated pattern. For example, if a header is expecting a value in the format<text>@<text> and is populated with text missing the@ symbol, the policy will fail with anUnable to parse JSON error.
• If the OASValidation policy specifies an <OASResource> containing apath parameter that utilizes a$ref schema, the policy will fail with anUnable to parse JSON - Unrecognized token error.

  Workaround: Do not use$ref in thepath parameters of the OpenAPI spec specified in the <OASResource> element.

297012500

• Apigee deployment will fail for OAS Validation policy when using circular references for OpenAPI 3.0.0 specification as it gets into an infinite loop.

Workaround: Use an OpenAPI specification yaml without circular references.

289254725

Proxy deployments that include the OASValidation policy may fail if:

• The OpenAPI specification used for validation in the OASValidation policy is in YAML format, and
• The YAML-formatted OpenAPI specification contains a floating number. For example:

  schema:type:numberexample:2.345

284500460

To avoid increasing latency in responses to the client, the Message Logging policy should be attached to thePostClientFlow. For more information on using policies in PostClientFlows, see Controlling API proxies with flows.

282997216

Use only alphanumeric characters for the Cassandra Jolokia password. Using special characters (including but not limited to "!", "@", "#", "$", "%", "^", "&", & "*") can cause Cassandra startup to fail.

270371160

Apigee Ingress gateway only supports TLS1.2+, and not earlier versions of TLS.

269139342

Apigee organization validation does not follow HTTP Forward proxy rules set inoverrides.yaml. SetvalidateOrg: false to skip this validation.

266452840

In certain circumstances, web sockets are not working for Apigee X and Apigee Hybrid when using Anthos Service Mesh1.15.3-asm.6.

287922301

Due to current design, inaccurate monetization rates may be assessed for transactions that occur near a volume-band threshold. This manifests as too many transactions being assessed at the pre-threshold rate, and is more likely to occur when many transactions for a given developer and rate plan combination occur within a 30 second window. This means that monetization reporting will over-report charges with decreasing banded rates, and under-report charges when increasing banded rates. This issue occurs in Apigee X and Apigee Hybrid and has no known workaround. Resolution has begun and will be available in a future release.

242213234

This error might be returned when attempting to load API products: "Products were not loaded successfully. Error: no connections available from the Apigee connect agent(s)."

The problem occurs after enabling VPC service control in the Google Cloud project and addingiamcredentials.googleapis.com as one of the restricted services in the service perimeter.

Workaround: Manually create an egress rule, such as the following:

-egressTo:    operations:    -serviceName: "iamcredentials.googleapis.com"        methodSelectors:        -method:    resources:    -projects/608305225983  egressFrom:    identityType: ANY_IDENTITY

247540503

In certain circumstances at very high throughput a race condition with encryption key lookup can cause KVM lookup failures.

258699204

If you see problems with the apigee-telemetry-app or apigee-telemetry-proxy pods not running, change themetrics resources requests and resources limits properties to match the following defaults in Configuration property reference: metrics.

Apply the changes withapigeectl apply with the‑‑telemetry flag:

apigeectl apply --telemetry -foverrides.yaml

260324159

API proxies and shared flows could take around 20 to 30 minutes to deploy in the runtime plane in certain circumstances due to a 'socket closed' error in synchronizer.

214447386

This is expected to occur every minute and does not affect your billing cost.

260772383

If installing hybrid on AKS, you may see this error:

envoy config listener '0.0.0.0_443' failed to bind or apply socket options: cannot bind '0.0.0.0:443': Permission denied

Workaround: Add the followingsvcAnnotations stanza to the overrides file:

ingressGateways:
- name: INGRESS_NAME
  ...
  svcAnnotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

See Configure the hybrid runtime. See also Use an internal load balancer with AKS.

241786534

When using Org-scoped UDCA, MART is sometimes unable to connect to FluentD. Org-scoped UDCA is the default in Apigee hybrid version 1.8. SeeorgScopedUDCA in the Configuration property reference.

After migration ofapigee-logger fromfluend tofluent-bit in Apigee hybrid version 1.6.6, the logger stopped working on Anthos BareMetal with CentOS or RHEL.

Currentapigee-logger configurations, including liveness probes, are incompatible with the Kubernetes runtime, resulting in logs not being shipped to Cloud Logging as expected. This problem also causes theapigee-logger pods to crash regularly. This issue affects Apigee hybrid installations on AKS, Anthos Bare Metal, and other platforms. Note that in some cases this issue results in excessive log volume.

In the Apigee UI, youcannot view, confirm deployment status, or manage your archive deployments, as described Deploying an API proxy, or use the Debug UI as described in Using Debug. As a workaround, you can use gcloud or the API to List all archive deployments in an environment and use the Debug API.

Rolling back an archive deployment is not currently supported. To remove a version of an archive deployment you need to either redeploy a previous version of an archive or delete the environment.

421402073

Google authentication in ServiceCallout and ExternalCallout policies, as described in Using Google Authentication, is not supported in Apigee in VS Code.

x-b3 headers not available when DistributedTrace is disabled

In the Apigee 1-15-0-apigee-5 and hybrid 1.15.0 releases, Apigee stopped sendingx-b3 headers to target endpoints when Distributed Trace is disabled. However, some Apigee installations were relying onx-b3 headers being sent regardless of whether Distributed Trace was enabled or disabled. Therefore this change has been reverted starting in Apigee 1-15-0-apigee-7.

Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called:

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1,

name: [:authority], value: [domain_name]

If you see this error, you can take either of the following actions to correct the problem:

• Modify the target service to omit the Host header in the response.
• Remove the Host header using the AssignMessage policy in your API proxy if necessary.

420985360

• Apigee supports OpenAPI Specification 3.0 when you publish your APIs using SmartDocs on your portal, though a subset of features are not yet supported. For example,allOf properties for combining and extending schemas.

  If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release.

• Try this API has the following limitations:
  • TheAccept header is set toapplication/json regardless of the value set forconsumes in the OpenAPI Specification.
  • Request headers of type other than string are not supported.

• Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time.
• If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation.
• When customizing your portal theme, it may take up to 5 minutes for changes to fully apply.

Search will be integrated into the integrated portal in a future release.

Single logout (SLO) with the SAML identity provider is not supported for custom domains. To enable a custom domain with a SAML identity provider, leave the Sign-out URL field blank when you configure SAML settings.

• API Proxy request and response counts (for proxy and targets) show abnormal spikes

  Here is a sample showing such a spike:

  (view larger image)

  
• Due to a bug, the system registers the count incorrectly for a brief period and the count is corrected. This happens where there is a reduction in API traffic (which results in a scale down of API gateways).
• To distinguish actual spikes in requests vs. this issue, please consult the API Analytics page (specifically theProxy Performance andTarget Performance pages)

Affected Metrics:

• apigee.googleapis.com/proxyv2/request_count
• apigee.googleapis.com/proxyv2/response_count
• apigee.googleapis.com/targetv2/request_count
• apigee.googleapis.com/targetv2/response_count

New metrics

You can use the new metrics to avoid this issue.

For Apigee hybrid, see:Metrics collection overview andView metrics.

Workaround: Disable the logging agent on hybrid.

Workaround: To maintain the fire and forget behavior:

1. Add<Response>calloutResponse</Response> to the ServiceCallout.
2. SetcontinueOnError totrue.

Workaround: Avoid using more than one SpikeArrest policy in the proxy to prevent the issue.

For a particular key, if there is continuous traffic, the key may not be rate limited at the updated rate. If there is five minutes of no traffic for a particular key, the rate will be reflected.

Workaround: Redeploy the proxy with a new reference variable if the rate has to take effect immediately. Or use two conditional spike arrests with different flow variables to adjust the rate.

API Monitoring of Configurable API Proxies may display a fault code of '(not set)' for responses with a non-2xx status from the target.

If a proxy sets two or moreio.timeout.millis values in two or more flows using the same target host, only oneio.timeout.millis value is honored.

During upgrade to Apigee hybrid 1.8.x, after runningapigeectl init and confirming thatcheck-ready succeeded, you may notice, if you view the pods, that the Cassandra schema validation job is in an error state.This is a harmless condition, and you can safely move to the next step in the upgrade procedure.

Deploying proxies with the same path to multiple environments that are attached to the same instance and environment group is not allowed and should return a warning message about a base path conflict. Instead, no error is shown and the deployments appear to succeed.

Workaround: When deploying and after deployment, verify that there are not base path conflicts with deployed proxies and correct as needed.

When attempting to save a previously deployed proxy, the deployment might fail with an error stating that the revision is immutable.

Workaround: Click the dropdown arrow next to theSave button and selectSave as new revision. Then reattempt the deployment.

When a call is made to a gRPC Target Server, the only trailer that's returned is the "grpc-status" trailer. All other trailers are removed from the response.

An infinite recursion occurs within the OASValidation policy when a backend response contains a discriminator type that is the same as the parent schema. This problem can lead to aStackOverflowError.

Use separate target endpoint definitions for SSE (server sent-event) targets. Mixing SSE and non-SSE target endpoints together might result in inconsistent behavior such as emptyresponse.content flow variables.

Previously undetected DNS configuration problems might now cause DNS errors. Apigee removed the automatic DNS fallback functionality that was in 1-16-0-apigee-2. This removal eliminates the platform's resiliency to DNS misconfigurations and might now result in DNS errors.

Find related errors by checking the runtime logs for DNS resolution errors.

All proxy deployments fail locally with error messages includingError parsing deployment report as JSON:

06/13/20256:45PMEMULATORINFODeployingenvironmenttestenv,toContainerapigee-1.14.2(1.14.2)06/13/20256:45PMEMULATORERRORErrorparsingdeploymentreportasJSON06/13/20256:45PMEMULATORERRORnull06/13/20256:45PMEMULATORERROREnvironmenttestenvdeployment,toContainerapigee-1.14.2(1.14.2)failedwithError:50006/13/20256:45PMEMULATORERRORDeploytoContainerapigee-1.14.2(1.14.2),failedwithError:Environmenttestenvdeployment,toContainerapigee-1.14.2(1.14.2)failedwithError:500