- Notifications
You must be signed in to change notification settings - Fork8k
Description
Description
I apologize in advance for the low quality issue report. I'm hoping that someone can point me in the right direction on next steps for completing a more detailed analysis.
I am running Apache 2.4.54 mpm_event with PHP 8.1.16 on a CentOS 8 Stream machine. When we gradually introduce production traffic load to this server, it begins to segfault every 25-45 minutes. I've been unable to find any correlation between the PHP code being executed and the segmentation faults. There is no particular request or script that reliably triggers a segmentation fault.
I captured core dumps for the segfaults and ran them throughgdb
.gdb
produced the following traces. I also ran valgrind in an attempt to capture debug info on memory problems (see below).
(gdb) bt#0 0x00007fb84f317161 in php_handler (r=0x7fb7582003b8) at /usr/src/debug/php-8.1.16-1.el8.x86_64/sapi/apache2handler/sapi_apache2.c:597#1 0x0000557a01527f38 in ap_run_handler (r=r@entry=0x7fb7582003b8) at config.c:169#2 0x0000557a015284f6 in ap_invoke_handler (r=r@entry=0x7fb7582003b8) at config.c:443#3 0x0000557a0153e1ec in ap_internal_redirect (new_uri=<optimized out>, r=<optimized out>) at http_request.c:790#4 0x00007fb84fb12a69 in handler_redirect (r=0x7fb7581f8c80) at mod_rewrite.c:5293#5 0x0000557a01527f38 in ap_run_handler (r=r@entry=0x7fb7581f8c80) at config.c:169#6 0x0000557a015284f6 in ap_invoke_handler (r=r@entry=0x7fb7581f8c80) at config.c:443#7 0x0000557a0153eed3 in ap_process_async_request (r=0x7fb7581f8c80) at http_request.c:452#8 0x0000557a0153f042 in ap_process_request (r=r@entry=0x7fb7581f8c80) at http_request.c:487#9 0x00007fb85139bb76 in h2_task_process_request (c=0x7fb758194300, task=<optimized out>) at h2_task.c:671#10 h2_task_process_conn (c=0x7fb758194300) at h2_task.c:713#11 h2_task_process_conn (c=0x7fb758194300) at h2_task.c:700#12 0x0000557a01531a08 in ap_run_process_connection (c=c@entry=0x7fb758194300) at connection.c:42#13 0x00007fb85139ced7 in h2_task_do (task=0x7fb75824d210, thread=thread@entry=0x557a02546cc8, worker_id=<optimized out>) at h2_task.c:631#14 0x00007fb8513a0c80 in slot_run (thread=0x557a02546cc8, wctx=0x557a02557700) at h2_workers.c:263#15 0x00007fb85622b1ca in start_thread () from /lib64/libpthread.so.0#16 0x00007fb855c93e73 in clone () from /lib64/libc.so.6
Thread 703 "httpd" received signal SIGSEGV, Segmentation fault.[Switching to Thread 0x7fb7bc9ff700 (LWP 31768)]0x00007fb84f317161 in php_handler (r=0x7fb7582003b8) at /usr/src/debug/php-8.1.16-1.el8.x86_64/sapi/apache2handler/sapi_apache2.c:597597 ctx = SG(server_context);(gdb) bt full#0 0x00007fb84f317161 in php_handler (r=0x7fb7582003b8) at /usr/src/debug/php-8.1.16-1.el8.x86_64/sapi/apache2handler/sapi_apache2.c:597 ctx = 0x557a02269ef8 conf = 0x557a0222e230 brigade = 0x0 bucket = 0x7fb758201918 rv = 0 parent_req = 0x0#1 0x0000557a01527f38 in ap_run_handler (r=r@entry=0x7fb7582003b8) at config.c:169 pHook = <optimized out> n = 7 rv = -1#2 0x0000557a015284f6 in ap_invoke_handler (r=r@entry=0x7fb7582003b8) at config.c:443 handler = <optimized out> p = <optimized out> result = 0 old_handler = 0x0 ignore = <optimized out>#3 0x0000557a0153e1ec in ap_internal_redirect (new_uri=<optimized out>, r=<optimized out>) at http_request.c:790 access_status = <optimized out> new = 0x7fb7582003b8#4 0x00007fb84fb12a69 in handler_redirect (r=0x7fb7581f8c80) at mod_rewrite.c:5293No locals.#5 0x0000557a01527f38 in ap_run_handler (r=r@entry=0x7fb7581f8c80) at config.c:169 pHook = <optimized out> n = 6 rv = -1#6 0x0000557a015284f6 in ap_invoke_handler (r=r@entry=0x7fb7581f8c80) at config.c:443 handler = <optimized out> p = <optimized out> result = 0 old_handler = 0x7fb84fb1a44a "redirect-handler" ignore = <optimized out>#7 0x0000557a0153eed3 in ap_process_async_request (r=0x7fb7581f8c80) at http_request.c:452 c = <optimized out> access_status = 0#8 0x0000557a0153f042 in ap_process_request (r=r@entry=0x7fb7581f8c80) at http_request.c:487 bb = 0x0 b = <optimized out> c = 0x7fb758194300 rv = <optimized out>#9 0x00007fb85139bb76 in h2_task_process_request (c=0x7fb758194300, task=<optimized out>) at h2_task.c:671 req = <optimized out> cs = 0x7fb758194998 r = 0x7fb7581f8c80 req = <optimized out> cs = <optimized out> r = <optimized out>--Type <RET> for more, q to quit, c to continue without paging--#10 h2_task_process_conn (c=0x7fb758194300) at h2_task.c:713 ctx = <optimized out> ctx = <optimized out>#11 h2_task_process_conn (c=0x7fb758194300) at h2_task.c:700 ctx = <optimized out>#12 0x0000557a01531a08 in ap_run_process_connection (c=c@entry=0x7fb758194300) at connection.c:42 pHook = <optimized out> n = 1 rv = -1#13 0x00007fb85139ced7 in h2_task_do (task=0x7fb75824d210, thread=thread@entry=0x557a02546cc8, worker_id=<optimized out>) at h2_task.c:631 c = 0x7fb758194300#14 0x00007fb8513a0c80 in slot_run (thread=0x557a02546cc8, wctx=0x557a02557700) at h2_workers.c:263 slot = 0x557a02557700#15 0x00007fb85622b1ca in start_thread () from /lib64/libpthread.so.0No symbol table info available.#16 0x00007fb855c93e73 in clone () from /lib64/libc.so.6No symbol table info available.
The trace ends at line 597 ofsapi/apache2handler/sapi_apache2.c
which contains this line:ctx = SG(server_context);
in this block of code:
#define PHPAP_INI_OFF php_apache_ini_dtor(r, parent_req); conf = ap_get_module_config(r->per_dir_config, &php_module); /* apply_config() needs r in some cases, so allocate server_context early */ ctx = SG(server_context); if (ctx == NULL || (ctx && ctx->request_processed && !strcmp(r->protocol, "INCLUDED"))) {normal: ctx = SG(server_context) = apr_pcalloc(r->pool, sizeof(*ctx)); /* register a cleanup so we clear out the SG(server_context) * after each request. Note: We pass in the pointer to the * server_context in case this is handled by a different thread. */ apr_pool_cleanup_register(r->pool, (void *)&SG(server_context), php_server_context_cleanup, apr_pool_cleanup_null); ctx->r = r; ctx = NULL; /* May look weird to null it here, but it is to catch the right case in the first_try later on */ } else { parent_req = ctx->r; ctx->r = r; } apply_config(conf);
I'm guessing that this isn't enough information to figure out what's going on. Could anyone recommend next steps for troubleshooting this?
PHP Version
8.1.16
configure
line:
./configure --prefix=/usr/local --enable-fpm --disable-fileinfo --enable-bcmath --enable-calendar --with-libxml --enable-soap --enable-mbstring --enable-pdo --enable-sockets --with-zip --with-apxs2=/usr/local/apache/bin/apxs --with-bz2 --with-curl=/usr/local --with-gettext --with-libdir=lib64 --with-openssl=/usr --with-openssl-dir=/usr --with-pdo-pgsql=/usr/pgsql-11 --with-pgsql=/usr/pgsql-11 --with-pic --with-zlib --with-zlib-dir=/usr --enable-opcache --enable-debug --enable-gd --with-jpeg PKG_CONFIG_PATH=:/usr/local/lib/pkgconfig:/usr/local/bin/pkgconfig
Dynamically loaded extensions:
zend_extension=/usr/local/lib/php/extensions/debug-zts-20210902/opcache.soopcache.memory_consumption=256opcache.interned_strings_buffer=16opcache.max_accelerated_files=5000opcache.revalidate_freq=0;opcache.validate_timestamps=1opcache.fast_shutdown=1extension=/usr/local/lib/php/extensions/debug-zts-20210902/memcached.so;
We experimented with disabling Zend opcache, but the segfaults persisted.
Operating System
CentOS 8 Stream