- Notifications
You must be signed in to change notification settings - Fork95
Add auth response caching for NGINX Ingress Controller deployments#661
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:master
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Add support for caching Houston /v1/authorization responses in NGINXIngress Controller via the nginx.ingress.kubernetes.io/auth-cache-keyand auth-cache-duration annotations.This reduces Houston load for customers with high numbers of concurrentAirflow users. Lab testing showed 99.5% reduction in auth requests(from 385 to 2 requests in 60 seconds for a single user).Configuration:- ingress.authCache.enabled (default: true)- ingress.authCache.key (default: "$http_cookie")- ingress.authCache.duration (default: "200 15m, 401 403 1m")Only applies when authSidecar.enabled is false (NGINX Ingress Controllerdeployments). Auth caching for authSidecar deployments will be addressedin a separate PR.
vishwas-astro commentedNov 4, 2025
Thanks for the detailed PR description@leestro .. Seems like a really good performance improvement for APC. @rishkarajgi@pgvishnuram@karankhanchandani do you want to include this into 1.0.1 and also backport it to 0.37 release branch to be included in the next patch release (whenever we decided to release one)? cc:@nic-astro |
| {{- if and .Values.ingress.auth.enabled .Values.ingress.authCache.enabled }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | ||
| {{- end }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
| {{- ifand .Values.ingress.auth.enabled.Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} | |
| {{- if .Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@pgvishnuram can you explain your logic?ingress.auth.enabled defaults totrue
if a customer setsingress.auth.enabled: false that would make the auth cache pointless, no?
| {{- if and .Values.ingress.auth.enabled .Values.ingress.authCache.enabled }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | ||
| {{- end }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
| {{- ifand .Values.ingress.auth.enabled.Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} | |
| {{- if .Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} |
| {{- if and .Values.ingress.auth.enabled .Values.ingress.authCache.enabled }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | ||
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | ||
| {{- end }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
| {{- ifand .Values.ingress.auth.enabled.Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} | |
| {{- if .Values.ingress.authCache.enabled }} | |
| "nginx.ingress.kubernetes.io/auth-cache-key":{{ .Values.ingress.authCache.key | quote }} | |
| "nginx.ingress.kubernetes.io/auth-cache-duration":{{ .Values.ingress.authCache.duration | quote }} | |
| {{- end }} |
Extends the auth response caching implementation to support authSidecardeployments (OpenShift customers who cannot use NGINX Ingress Controllerdue to ClusterRole restrictions).This change implements caching directly in the authSidecar nginxConfigMaps by adding:1. proxy_cache_path directive in the http block2. proxy_cache directives in the /auth location3. Cookie header pass-through (required for cache key)4. Conditional proxy_buffering on (required for caching to work)Configuration:- authSidecar.authCache.enabled (default: true)- authSidecar.authCache.key (default: "$http_cookie")- authSidecar.authCache.duration (default: "200 15m, 401 403 1m")Lab testing with APC 0.35.0 showed:- Before: 113 auth requests to Houston in 5 minutes- After: 1 initial auth request, then 100% cache hits- Reduction: ~99.5% fewer Houston requestsAffected templates:- templates/webserver/webserver-auth-sidecar-configmap.yaml- templates/api-server/api-server-auth-sidecar-configmap.yaml- templates/dag-deploy/dag-server-auth-sidecar-configmap.yaml- templates/flower/flower-auth-sidecar-configmap.yamlBug fix:- Fixed ingress annotation merging when authSidecar is enabled- Previously lost critical annotations like kubernetes.io/ingress.class- Now properly merges webserverAnnotations with authSidecar.annotations
Uh oh!
There was an error while loading.Please reload this page.
Add auth response caching for both NGINX Ingress Controller and authSidecar deployments
Summary
This PR adds support for caching Houston
/v1/authorizationresponses in both deployment modes:This reduces Houston load for customers with high numbers of concurrent Airflow users.
Related Issue
https://github.com/astronomer/issues/issues/8056
Problem
Every request to the Airflow webserver, Flower UI, or dag-server triggers an auth subrequest to Houston's
/v1/authorizationendpoint. This includes page loads, static assets, API calls, and AJAX requests. For customers with many active Airflow users, this creates significant load on Houston.Lab testing showed that a single user browsing the Airflow UI generated 385 auth requests to Houston in 60 seconds (6.4 requests/second). For customers with 50-100 concurrent users, this can result in hundreds of requests per second to Houston.
Solution
NGINX Ingress Controller Deployments
For deployments using NGINX Ingress Controller, this PR adds the
nginx.ingress.kubernetes.io/auth-cache-keyandnginx.ingress.kubernetes.io/auth-cache-durationannotations to the Airflow, Flower, and dag-server Ingress resources when:authSidecar.enabled: false(uses NGINX Ingress Controller)ingress.auth.enabled: true(authentication is enabled)ingress.authCache.enabled: true(new configuration, enabled by default)authSidecar Deployments
For deployments using authSidecar (OpenShift customers who cannot use NGINX Ingress Controller due to ClusterRole restrictions), this PR implements caching directly in the authSidecar nginx ConfigMaps by adding:
proxy_cache_pathdirective in the http blockproxy_cachedirectives in the /auth locationproxy_buffering on(required for caching to work)This applies when:
authSidecar.enabled: trueauthSidecar.authCache.enabled: true(new configuration, enabled by default)Configuration
New values in
values.yaml:Cache Key: Uses
$http_cookieto uniquely identify users by their session cookies.Cache Duration:
The 15-minute TTL is conservative compared to the default JWT validity of 24 hours. This provides a significant performance improvement while maintaining reasonable security posture.
Testing
NGINX Ingress Controller
Lab testing with APC 0.35.0 on EKS demonstrated:
authSidecar
Lab testing with APC 0.35.0 showed:
Test results show the first request after login is a cache miss (validates with Houston), and all subsequent requests within 15 minutes are cache hits (served from NGINX cache).
Bug Fix
Fixed ingress annotation merging when authSidecar is enabled. Previously, enabling authSidecar would cause the loss of critical ingress annotations like
kubernetes.io/ingress.class. This PR properly mergeswebserverAnnotationswithauthSidecar.annotations.Security Considerations
Trade-off: Cached auth responses mean that revoked credentials or changed permissions may remain valid for up to the cache TTL (default 15 minutes).
Rationale:
Customers can adjust the TTL by configuring the
durationvalues or disable caching entirely by settingenabled: false.Breaking Changes
None. This is opt-in functionality that defaults to enabled. Customers with custom configurations will have these settings added automatically, which should be transparent to both NGINX Ingress Controller and authSidecar deployments.
Files Changed
values.yaml: Addingress.authCacheandauthSidecar.authCacheconfigurationtemplates/ingress.yaml: Add conditional auth cache annotations and fix annotation mergingtemplates/webserver/webserver-auth-sidecar-configmap.yaml: Add auth caching directivestemplates/api-server/api-server-auth-sidecar-configmap.yaml: Add auth caching directivestemplates/dag-deploy/dag-server-auth-sidecar-configmap.yaml: Add auth caching directivestemplates/flower/flower-auth-sidecar-configmap.yaml: Add auth caching directives