Topic: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

CentOS updated the openssl libraries yesterday, which may or may not be relevant. Since the change, I've been having lots of segfaults. I told Apache to give me core dumps, and a stack trace shows..

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fa8219d39d4 in zend_hash_clean () from /etc/httpd/modules/libphp5.so
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-56.el6.centos.3.x86_64
(gdb) where
#0  0x00007fa8219d39d4 in zend_hash_clean () from /etc/httpd/modules/libphp5.so
#1  0x00007fa8219bb5eb in zend_cleanup_function_data () from /etc/httpd/modules/libphp5.so
#2  0x00007fa8219d3623 in zend_hash_reverse_apply () from /etc/httpd/modules/libphp5.so
#3  0x00007fa8219b743b in ?? () from /etc/httpd/modules/libphp5.so
#4  0x00007fa8219c6332 in zend_deactivate () from /etc/httpd/modules/libphp5.so
#5  0x00007fa821963952 in php_request_shutdown () from /etc/httpd/modules/libphp5.so
#6  0x00007fa821a6c8e7 in ?? () from /etc/httpd/modules/libphp5.so
#7  0x00007fa82d314fa0 in ap_run_handler ()
#8  0x00007fa82d31885e in ap_invoke_handler ()
#9  0x00007fa82d323fb0 in ap_process_request ()
#10 0x00007fa82d320df8 in ?? ()
#11 0x00007fa82d31cac8 in ap_run_process_connection ()
#12 0x00007fa82d328d3d in ?? ()
#13 0x00007fa82d329049 in ?? ()
#14 0x00007fa82d329ccc in ap_mpm_run ()
#15 0x00007fa82d300a80 in main ()

My gdb skills are non-existent.
It's the same spot in each core dump. I am now in guessing mode.. so I've rebooted the machine, run ldconfig, restarted httpd. The machine is not short of memory.

It's not happening on my other machine which runs the same systems, but is less busy.

So basically help!

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

Hmm here's another closedown problem

Program terminated with signal 11, Segmentation fault.
#0  0x00007fa8219bb026 in destroy_op_array () from /etc/httpd/modules/libphp5.so
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-56.el6.centos.3.x86_64
(gdb) where
#0  0x00007fa8219bb026 in destroy_op_array () from /etc/httpd/modules/libphp5.so
#1  0x00007fa8219d38bb in zend_hash_destroy () from /etc/httpd/modules/libphp5.so
#2  0x00007fa8219c6618 in ?? () from /etc/httpd/modules/libphp5.so
#3  0x00007fa821962f9a in php_module_shutdown () from /etc/httpd/modules/libphp5.so
#4  0x00007fa821963049 in php_module_shutdown_wrapper () from /etc/httpd/modules/libphp5.so
#5  0x00007fa821a6b7e1 in ?? () from /etc/httpd/modules/libphp5.so
#6  0x00007fa82bdeb99e in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
#7  0x00007fa82d3289ec in ?? ()
#8  0x00007fa82d328e19 in ?? ()
#9  0x00007fa82d329049 in ?? ()
#10 0x00007fa82d329ccc in ap_mpm_run ()
#11 0x00007fa82d300a80 in main ()

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

Extensions used ("rpm -qa php\* | sort") ?
Recently updated package (/Var/log/yum.log)  ?


I don't think this can be related to openssl update (which is only a security bugfix and should be fully compatible)

As the segfault occurs in php_request_shutdown, this mean the request have been served (btw should not happen)
Could be very hard to track, as this is related to a bad memory management during runtime, from any extension, and raising the segault later...

Desktop: Fedora 25 x86_64 + rpmfusion + remi-test + remi-dev
Laptop:  Fedora 26 x86_64 + remi (SCL only)
Hosting Server: CentOS 6.9 x86_64 + EPEL + remi + remi-php70

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

OK - here's the first core with debuginfo installed

Program terminated with signal 11, Segmentation fault.
#0  zend_hash_clean (ht=0x7fa83312ed78) at /usr/src/debug/php-5.6.30/Zend/zend_hash.c:582
582            p = p->pListNext;
Missing separate debuginfos, use: debuginfo-install fftw-3.2.1-3.1.el6.x86_64 libICE-1.0.6-1.el6.x86_64 libc-client-2007e-11.el6.x86_64 libidn-1.18-2.el6.x86_64 libtidy-0.99.0-19.20070615.1.el6.x86_64 libtool-ltdl-2.2.6-15.5.el6.x86_64 tokyocabinet-1.4.33-6.el6.x86_64
(gdb) where
#0  zend_hash_clean (ht=0x7fa83312ed78) at /usr/src/debug/php-5.6.30/Zend/zend_hash.c:582
#1  0x00007fa8219bb5eb in zend_cleanup_op_array_data (function=<value optimized out>) at /usr/src/debug/php-5.6.30/Zend/zend_opcode.c:134
#2  zend_cleanup_function_data (function=<value optimized out>) at /usr/src/debug/php-5.6.30/Zend/zend_opcode.c:141
#3  0x00007fa8219d3623 in zend_hash_reverse_apply (ht=0x7fa82f119ee0, apply_func=0x7fa8219bb5c0 <zend_cleanup_function_data>)
    at /usr/src/debug/php-5.6.30/Zend/zend_hash.c:729
#4  0x00007fa8219b743b in shutdown_executor () at /usr/src/debug/php-5.6.30/Zend/zend_execute_API.c:283
#5  0x00007fa8219c6332 in zend_deactivate () at /usr/src/debug/php-5.6.30/Zend/zend.c:960
#6  0x00007fa821963952 in php_request_shutdown (dummy=<value optimized out>) at /usr/src/debug/php-5.6.30/main/main.c:1899
#7  0x00007fa821a6c8e7 in php_apache_request_dtor (r=0x7fa832c03538) at /usr/src/debug/php-5.6.30/sapi/apache2handler/sapi_apache2.c:507
#8  php_handler (r=0x7fa832c03538) at /usr/src/debug/php-5.6.30/sapi/apache2handler/sapi_apache2.c:679
#9  0x00007fa82d314fa0 in ap_run_handler (r=0x7fa832c03538) at /usr/src/debug/httpd-2.2.15/server/config.c:158
#10 0x00007fa82d31885e in ap_invoke_handler (r=0x7fa832c03538) at /usr/src/debug/httpd-2.2.15/server/config.c:376
#11 0x00007fa82d323fb0 in ap_process_request (r=0x7fa832c03538) at /usr/src/debug/httpd-2.2.15/modules/http/http_request.c:282
#12 0x00007fa82d320df8 in ap_process_http_connection (c=0x7fa832bdd608) at /usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
#13 0x00007fa82d31cac8 in ap_run_process_connection (c=0x7fa832bdd608) at /usr/src/debug/httpd-2.2.15/server/connection.c:43
#14 0x00007fa82d328d3d in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:670
#15 0x00007fa82d329049 in make_child (s=0x7fa82f0a8850, slot=13) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:773
#16 0x00007fa82d329ccc in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
    at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:908
#17 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:1112
#18 0x00007fa82d300a80 in main (argc=1, argv=0x7ffded6fff78) at /usr/src/debug/httpd-2.2.15/server/main.c:763

and here's the second

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fa8219bb026 in destroy_op_array (op_array=0x7fa832d58110) at /usr/src/debug/php-5.6.30/Zend/zend_opcode.c:369
369        if (--(*op_array->refcount)>0) {
(gdb) where
#0  0x00007fa8219bb026 in destroy_op_array (op_array=0x7fa832d58110) at /usr/src/debug/php-5.6.30/Zend/zend_opcode.c:369
#1  0x00007fa8219d38bb in zend_hash_destroy (ht=0x7fa82f119ee0) at /usr/src/debug/php-5.6.30/Zend/zend_hash.c:548
#2  0x00007fa8219c6618 in zend_shutdown () at /usr/src/debug/php-5.6.30/Zend/zend.c:841
#3  0x00007fa821962f9a in php_module_shutdown () at /usr/src/debug/php-5.6.30/main/main.c:2477
#4  0x00007fa821963049 in php_module_shutdown_wrapper (sapi_globals=<value optimized out>) at /usr/src/debug/php-5.6.30/main/main.c:2445
#5  0x00007fa821a6b7e1 in php_apache_child_shutdown (tmp=<value optimized out>) at /usr/src/debug/php-5.6.30/sapi/apache2handler/sapi_apache2.c:398
#6  0x00007fa82bdeb99e in run_cleanups (pool=0x7fa832bdb388) at memory/unix/apr_pools.c:2314
#7  apr_pool_destroy (pool=0x7fa832bdb388) at memory/unix/apr_pools.c:782
#8  0x00007fa82d3289ec in clean_child_exit (code=0) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:199
#9  0x00007fa82d328e19 in accept_mutex_on (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:215
#10 child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:576
#11 0x00007fa82d329049 in make_child (s=0x7fa82f0a8850, slot=32) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:773
#12 0x00007fa82d329ccc in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
    at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:908
#13 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:1112
#14 0x00007fa82d300a80 in main (argc=1, argv=0x7ffded6fff78) at /usr/src/debug/httpd-2.2.15/server/main.c:763

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

Listing yum is endless.. Here's /etc/php.d

20-bcmath.ini    20-intl.ini       20-sockets.ini    30-pdo_pgsql.ini
20-bz2.ini       20-ldap.ini       20-sqlite3.ini    30-pdo_sqlite.ini
20-calendar.ini  20-mbstring.ini   20-sysvmsg.ini    30-wddx.ini
20-ctype.ini     20-mcrypt.ini     20-sysvsem.ini    30-xmlreader.ini
20-curl.ini      20-mysqlnd.ini    20-sysvshm.ini    30-xmlrpc.ini
20-dba.ini       20-odbc.ini       20-tidy.ini       40-imagick.ini
20-dom.ini       20-pdo.ini        20-tokenizer.ini  40-json.ini
20-exif.ini      20-pgsql.ini      20-xml.ini        40-ncurses.ini
20-fileinfo.ini  20-phar.ini       20-xmlwriter.ini  40-radius.ini
20-ftp.ini       20-posix.ini      20-xsl.ini        40-ssh2.ini
20-gd.ini        20-shmop.ini      30-mysql.ini      40-xcache.ini
20-gettext.ini   20-simplexml.ini  30-mysqli.ini     40-xcache.orig
20-iconv.ini     20-snmp.ini       30-pdo_mysql.ini  40-zip.ini
20-imap.ini      20-soap.ini       30-pdo_odbc.ini

I tried turning off cache just in case..

Yum update since Jan

Jan 16 07:09:19 Updated: php-pear-Net-SMTP-1.7.3-1.el6.remi.noarch
Jan 19 07:09:55 Updated: php-pecl-imagick-3.4.3-0.5.RC2.el6.remi.5.6.x86_64
Jan 20 03:12:01 Updated: php-common-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:01 Updated: php-pdo-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:02 Updated: php-cli-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:02 Updated: php-xml-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:02 Updated: php-xmlrpc-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:02 Updated: php-devel-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-odbc-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-mysqlnd-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-pgsql-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-soap-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-tidy-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-dba-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-gd-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-snmp-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-process-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-mbstring-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-mcrypt-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-imap-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-ldap-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-bcmath-5.6.30-1.el6.remi.x86_64
Jan 20 03:12:03 Updated: php-intl-5.6.30-1.el6.remi.x86_64
Jan 21 08:18:36 Updated: php-pecl-jsonc-1.3.10-2.el6.remi.5.6.x86_64
Jan 21 08:18:36 Updated: php-pecl-jsonc-devel-1.3.10-2.el6.remi.5.6.x86_64
Jan 25 07:09:12 Updated: php-pecl-imagick-3.4.3-0.6.RC3.el6.remi.5.6.x86_64
Jan 27 03:35:02 Updated: php-pecl-imagick-3.4.3-0.7.RC4.el6.remi.5.6.x86_64
Jan 30 07:15:16 Updated: php-pecl-imagick-3.4.3-0.8.RC4.el6.remi.5.6.x86_64
Feb 03 07:09:24 Updated: php-pecl-imagick-3.4.3-1.el6.remi.5.6.x86_64
Feb 03 07:09:24 Updated: php-pear-PHP-CodeSniffer-2.8.0-1.el6.remi.noarch
Feb 05 08:43:48 Updated: 1:php-pear-1.10.1-9.el6.remi.noarch
Feb 09 07:14:51 Updated: 1:php-pear-1.10.1-10.el6.remi.noarch
Feb 23 07:49:58 Updated: 1:php-pear-1.10.1-11.el6.remi.noarch

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

Notice: Xcache is a dead project (should prefer opcache for opcode caching, and APCu or other cache for user data)

When does the segfault start ?

Desktop: Fedora 25 x86_64 + rpmfusion + remi-test + remi-dev
Laptop:  Fedora 26 x86_64 + remi (SCL only)
Hosting Server: CentOS 6.9 x86_64 + EPEL + remi + remi-php70

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

How I hate Wordpress... it looks like the problem was related to a Wordpress cron job running out of memory at which point I'll guess all bets are off.  Of course, I cannot prove that. Thanks for your help as usual.

Re: CentOS 6 - Apache2.2 PHP5.6 segfault in zend_hash_clean

I was planning to bring caching up-to-date :-)