- Notifications
You must be signed in to change notification settings - Fork832
Open
Description
Detailed Description of the Problem
When HAProxy is using a Lua code for OAuth2 (https://github.com/TimWolla/haproxy-auth-request), it started crashing recently.
The last known working HAProxy was 3.1.0-f2b9791 2024/11/26, while the latest tested 3.1.5-076df02 2025/02/20 is already failing. So the issue was introduced in some of the 3.1.1 - 3.1.5 releases.
Our HAProxy is running alongside of the HAProxy Ingress Controller deployed from a standard Helm chart, it's not a standalone run.
Expected Behavior
HAProxy doesn't freeze and crash.
Steps to Reproduce the Behavior
Very vague:
- Configure some Lua OAuth2 validation code
- HAProxy starts crashing
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
The configuration is generated by HAProxy Ingress controller, but the only relevant part to Lua looks like:global###_config-snippet_### BEGIN lua-prepend-path /lua/?.lua lua-load /lua/auth-request.lua lua-load /lua/services.lua lua-load /lua/responses.lua...###_config-snippet_### ENDbackendxxx... http-request lua.auth-intercept xxx /oauth2/auth GET '*' '*' '-' http-request redirect location https://xxx?rd=https://%[hdr(host)]%[capture.req.uri]if !use-basic-auth !{ var(txn.auth_response_successful) -m bool } http-request set-header X-WEBAUTH-USER%[var(req.auth_response_header.gap_auth)]unless use-basic-auth http-request set-header X-WEBAUTH-TYPE "oauth2"unless use-basic-auth...
Output ofhaproxy -vv
HAProxy version 3.1.5-076df02 2025/02/20 - https://haproxy.org/Status: stable branch - will stop receiving fixes around Q1 2026.Known bugs: http://www.haproxy.org/bugs/bugs-3.1.5.htmlRunning on: Linux 5.10.207 #1 SMP Tue Jan 14 08:15:54 UTC 2025 x86_64Build options : TARGET = linux-musl CC = cc CFLAGS = -O2 -g -fwrapv OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1 DEBUG = Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIBDefault settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).Built with OpenSSL version : OpenSSL 3.3.3 11 Feb 2025Running on OpenSSL version : OpenSSL 3.3.3 11 Feb 2025OpenSSL library supports TLS extensions : yes/ $ haproxy -vvHAProxy version 3.1.5-076df02 2025/02/20 - https://haproxy.org/Status: stable branch - will stop receiving fixes around Q1 2026.Known bugs: http://www.haproxy.org/bugs/bugs-3.1.5.htmlRunning on: Linux 5.10.207 #1 SMP Tue Jan 14 08:15:54 UTC 2025 x86_64Build options : TARGET = linux-musl CC = cc CFLAGS = -O2 -g -fwrapv OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1 DEBUG = Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIBDefault settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).Built with OpenSSL version : OpenSSL 3.3.3 11 Feb 2025/ $ haproxy -vvHAProxy version 3.1.5-076df02 2025/02/20 - https://haproxy.org/Status: stable branch - will stop receiving fixes around Q1 2026.Known bugs: http://www.haproxy.org/bugs/bugs-3.1.5.htmlRunning on: Linux 5.10.207 #1 SMP Tue Jan 14 08:15:54 UTC 2025 x86_64Build options : TARGET = linux-musl CC = cc CFLAGS = -O2 -g -fwrapv OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1 DEBUG = Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIBDefault settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).Built with OpenSSL version : OpenSSL 3.3.3 11 Feb 2025Running on OpenSSL version : OpenSSL 3.3.3 11 Feb 2025OpenSSL library supports TLS extensions : yesOpenSSL library supports SNI : yesOpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3OpenSSL providers loaded : defaultBuilt with Lua version : Lua 5.4.6Built with the Prometheus exporter as a serviceBuilt with network namespace support.Built with libslz for stateless compression.Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBINDBuilt with PCRE2 version : 10.43 2024-02-16PCRE2 library supports JIT : yesEncrypted password support via crypt(3): yesBuilt with gcc compiler version 13.2.1 20240309Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OKTotal: 3 (3 usable), will use epoll.Available multiplexer protocols :(protocols marked as <default> cannot be specified using 'proto' keyword) quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG <default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP side=FE|BE mux=PASS flags=NO_UPGAvailable services : prometheus-exporterAvailable filters :[BWLIM] bwlim-in[BWLIM] bwlim-out[CACHE] cache[COMP] compression[FCGI] fcgi-app[SPOE] spoe[TRACE] trace
Last Outputs and Backtraces
Redacted log attached.