1. Problem Description:

Our production apache22/mod_perl2 web server segfaults regularly. I
have tracked down this problem as much as I can, and could use some
help. This is running on FreeBSD; the same problem occurs on FreeBSD
7.0, 7.2, and 8.0. apache22 is installed from ports, with WITH_DEBUG
and WITH_EXECPTION_HOOK; mod_perl2 is installed from ports, my
attempts to compile it with debug flags appear so far to be
unsuccessful. (detailed version info below)

For no reason I can fathom, I can not seem to get apache to dump core
when it seg faults. I've set CoreDumpDirectory, I've set ulimit
coredumpsize, nothing. No matter; I was able to install
mod_whatkilledus and mod_backtrace.

mod_whatkilledus shows that the segfaults happen on a wide variety of
requests; most requests to our main web server are handled by
mod_perl, either with a ResponseHandler or a FilterHandler.

mod_backtrace shows that the seg fault is always happening in
modperl_perl_global_request_save:

[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace backtrace for sig 11 (thread "pid" 6955)
[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace main() is at 8063880
0x8085a26 <ap_run_fatal_exception+70> at /usr/local/sbin/httpd
0x808752e <ap_mpm_set_exception_hook+238> at /usr/local/sbin/httpd
0x8087569 <ap_mpm_set_exception_hook+297> at /usr/local/sbin/httpd
0xbfbfffb4
0x287a83c5 <modperl_perl_global_request_save+213> at /usr/local/libexec/apache22/mod_perl.so
0x2879982d <modperl_response_handler_cgi+141> at /usr/local/libexec/apache22/mod_perl.so
0x807bc66 <ap_run_handler+70> at /usr/local/sbin/httpd
....

So, any general pointers for what direction to go next would be
appreciated. I would like to have mod_backtrace tell me what in my
perl code is triggering the error, in which case I could work around
it. Should I continue to pursue recompiling mod_perl2 and possibly
perl itself with more debugging turned on?
(http://perl.apache.org/docs/1.0/guide/help.html)

2. Used Components and their Configuration:

*** mod_perl version 2.000004

*** using /usr/local/lib/perl5/site_perl/5.8.9/mach/Apache2/BuildConfi g.pm

*** Makefile.PL options:
MP_APR_LIB => aprext
MP_APXS => /usr/local/sbin/apxs
MP_COMPAT_1X => 1
MP_GENERATE_XS => 1
MP_LIBNAME => mod_perl
MP_USE_DSO => 1


*** The httpd binary was not found

# I ran it by hand instead:

bsdtest# /usr/local/sbin/httpd -V
Server version: Apache/2.2.14 (FreeBSD)
Server built: Dec 30 2009 09:46:16
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.8, APR-Util 1.3.9
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"

[Wed Dec 30 15:49:00 2009] [notice] Apache/2.2.14 (FreeBSD) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.8.9 configured -- resuming normal operations

*** (apr|apu)-config linking info

-L/usr/local/lib -laprutil-1 -lexpat -liconv -L/usr/local/lib
-L/usr/local/lib -lapr-1 -lcrypt -pthread



*** /usr/local/bin/perl -V
Summary of my perl5 (revision 5 version 8 subversion 9) configuration:
Platform:
osname=freebsd, osvers=8.0-release, archname=i386-freebsd-64int
uname='freebsd freebsd.org 8.0-release freebsd 8.0-release #0: fri sep 18 03:01:27 pdt 2009 kris [at] freebsd.org:usrsrcsysmagickernelpath i386 '
config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.9/mach -Dprivlib=/usr/local/lib/perl5/5.8.9 -Dman3dir=/usr/local/lib/perl5/5.8.9/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.9/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.9 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.8.9/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -Doptimize=-O2 -pipe -fno-strict-aliasing -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include',
optimize='-O2 -pipe -fno-strict-aliasing',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -Wl,-E -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil
perllibs=-lm -lcrypt -lutil
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.8.9/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: MYMALLOC PERL_MALLOC_WRAP USE_64_BIT_INT
USE_FAST_STDIO USE_LARGE_FILES USE_PERLIO
Locally applied patches:
defined-or
Built under freebsd
Compiled at Sep 18 2009 10:03:02
%ENV:
PERL_LWP_USE_HTTP_10="1"
[at] INC:
/usr/local/lib/perl5/5.8.9/BSDPAN
/usr/local/lib/perl5/site_perl/5.8.9/mach
/usr/local/lib/perl5/site_perl/5.8.9
/usr/local/lib/perl5/5.8.9/mach
/usr/local/lib/perl5/5.8.9
.

*** Packages of interest status:

Apache2 : -
Apache2::Request : 2.12
CGI : 3.42
ExtUtils::MakeMaker: 6.48, undef
LWP : 5.834
mod_perl : -
mod_perl2 : 2.000004


3. This is the core dump trace: (if you get a core dump):

[CORE TRACE COMES HERE]

This report was generated by /usr/local/bin/mp2bug on Wed Dec 30 20:46:25 2009 GMT.


- Alex Aminoff
BaseSpace.net
National Bureau of Economic Research (nber.org)

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig761412AD7DDA0D65898F09A7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 09-12-30 16:28 , Alex Aminoff wrote:
>
> 1. Problem Description:
>
> Our production apache22/mod_perl2 web server segfaults regularly. I
> have tracked down this problem as much as I can, and could use some
> help. This is running on FreeBSD; the same problem occurs on FreeBSD
> 7.0, 7.2, and 8.0. apache22 is installed from ports, with WITH_DEBUG
> and WITH_EXECPTION_HOOK; mod_perl2 is installed from ports, my
> attempts to compile it with debug flags appear so far to be
> unsuccessful. (detailed version info below)
>
> For no reason I can fathom, I can not seem to get apache to dump core
> when it seg faults. I've set CoreDumpDirectory, I've set ulimit
> coredumpsize, nothing. No matter; I was able to install
> mod_whatkilledus and mod_backtrace.
>
> mod_whatkilledus shows that the segfaults happen on a wide variety of
> requests; most requests to our main web server are handled by
> mod_perl, either with a ResponseHandler or a FilterHandler.
>
> mod_backtrace shows that the seg fault is always happening in
> modperl_perl_global_request_save:
>
> [Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace backtrace for sig 11
> (thread "pid" 6955)
> [Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace main() is at 8063880
> 0x8085a26 <ap_run_fatal_exception+70> at /usr/local/sbin/httpd
> 0x808752e <ap_mpm_set_exception_hook+238> at /usr/local/sbin/httpd
> 0x8087569 <ap_mpm_set_exception_hook+297> at /usr/local/sbin/httpd
> 0xbfbfffb4
> 0x287a83c5 <modperl_perl_global_request_save+213> at
> /usr/local/libexec/apache22/mod_perl.so
> 0x2879982d <modperl_response_handler_cgi+141> at
> /usr/local/libexec/apache22/mod_perl.so
> 0x807bc66 <ap_run_handler+70> at /usr/local/sbin/httpd
> ...
>
> So, any general pointers for what direction to go next would be
> appreciated. I would like to have mod_backtrace tell me what in my
> perl code is triggering the error, in which case I could work around
> it. Should I continue to pursue recompiling mod_perl2 and possibly
> perl itself with more debugging turned on?

Most certainly, yes. A stack trace without debugging symbols is not
very useful in trying to pin down the source of a segfault bug.

--
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\ [at] (apache|cpan|ectoplasm)\.org/


--------------enig761412AD7DDA0D65898F09A7
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLO8i7yzKhB4jDpaURAot+AJ9Rgworfco/rLrNTV9qMh/rgnz51gCg ibzO
KDXdZ5T6B807ifEbeETPZmk=
=34/n
-----END PGP SIGNATURE-----

--------------enig761412AD7DDA0D65898F09A7--

> Most certainly, yes. A stack trace without debugging symbols is not
> very useful in trying to pin down the source of a segfault bug.

OK, I compiled mod_perl with MP_DEBUG and MP_TRACE, and
got apache to produce a core dump (kern.sugid_coredump=1 appears to
be needed on FreeBSD):

(gdb) bt
#0 0x28384eb4 in strncpy () from /lib/libc.so.7
#1 0x287af9ee in modperl_perl_global_avcv_clear ()
from /usr/local/libexec/apache22/mod_perl.so
#2 0x287afacf in modperl_perl_global_avcv_clear ()
from /usr/local/libexec/apache22/mod_perl.so
#3 0x287afbed in modperl_perl_global_request_save ()
from /usr/local/libexec/apache22/mod_perl.so
#4 0x2879bf7e in modperl_response_handler_cgi ()
from /usr/local/libexec/apache22/mod_perl.so
#5 0x0807bc66 in ap_run_handler (r=0x28eb6058) at config.c:157
#6 0x0807c42f in ap_invoke_handler (r=0x28eb6058) at config.c:372
#7 0x0808c503 in ap_process_request (r=0x28eb6058) at http_request.c:282
#8 0x080890c1 in ap_process_http_connection (c=0x28afe1f0) at
http_core.c:190
#9 0x080846a6 in ap_run_process_connection (c=0x28afe1f0) at
connection.c:43
#10 0x08084af8 in ap_process_connection (c=0x28afe1f0, csd=0x28afe058)
at connection.c:178
#11 0x08092b7f in child_main (child_num_arg=6) at prefork.c:662
#12 0x08092d6c in make_child (s=0x28410f10, slot=6) at prefork.c:758
#13 0x08092fc4 in perform_idle_server_maintenance (p=0x2840f018) at
prefork.c:893
#14 0x080934e8 in ap_mpm_run (_pconf=0x2840f018, plog=0x2843d018,
s=0x28410f10)
at prefork.c:1097
#15 0x08064389 in main (argc=0, argv=0x0) at main.c:740

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig896C3E85BDA12C39EE618767
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 09-12-31 12:52 , Alex Aminoff wrote:
>
>> Most certainly, yes. A stack trace without debugging symbols is not
>> very useful in trying to pin down the source of a segfault bug.
>
> OK, I compiled mod_perl with MP_DEBUG and MP_TRACE, and
> got apache to produce a core dump (kern.sugid_coredump=3D1 appears to b=
e
> needed on FreeBSD):
>
> (gdb) bt
> #0 0x28384eb4 in strncpy () from /lib/libc.so.7


> #1 0x287af9ee in modperl_perl_global_avcv_clear ()
> from /usr/local/libexec/apache22/mod_perl.so
> #2 0x287afacf in modperl_perl_global_avcv_clear ()
> from /usr/local/libexec/apache22/mod_perl.so


That's odd, modperl_perl_global_avcv_clear should not be able to call
itself. It looks like you don't have debugging symbols for mod_perl
itself, could you get that as well, as it would show me a little bit
more as to what's being double-freed, looks like.

Also, you can just try and run your test case with MOD_PERL_TRACE=3D'g' i=
n
your environment ? It turns on global handling debugging output.
error_log would contain information about what is going on in more detail=
s.

--
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\ [at] (apache|cpan|ectoplasm)\.org/


--------------enig896C3E85BDA12C39EE618767
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLPOrZyzKhB4jDpaURAiSXAJ9FlhYzaBBU/2/wwBngnX8qb8ch3gCe OEpi
BM2TQxAEGNPh373Q/H61V74=
=4RR7
-----END PGP SIGNATURE-----

--------------enig896C3E85BDA12C39EE618767--

On Thu, 31 Dec 2009, Philippe M. Chiasson wrote:

> On 09-12-31 12:52 , Alex Aminoff wrote:
>>
>>> Most certainly, yes. A stack trace without debugging symbols is not
>>> very useful in trying to pin down the source of a segfault bug.
>>
>> OK, I compiled mod_perl with MP_DEBUG and MP_TRACE, and
>> got apache to produce a core dump (kern.sugid_coredump=1 appears to be
>> needed on FreeBSD):
>>
>> (gdb) bt
>> #0 0x28384eb4 in strncpy () from /lib/libc.so.7
>
>
>> #1 0x287af9ee in modperl_perl_global_avcv_clear ()
>> from /usr/local/libexec/apache22/mod_perl.so
>> #2 0x287afacf in modperl_perl_global_avcv_clear ()
>> from /usr/local/libexec/apache22/mod_perl.so
>
>
> That's odd, modperl_perl_global_avcv_clear should not be able to call
> itself. It looks like you don't have debugging symbols for mod_perl
> itself, could you get that as well, as it would show me a little bit
> more as to what's being double-freed, looks like.
>
> Also, you can just try and run your test case with MOD_PERL_TRACE='g' in
> your environment ? It turns on global handling debugging output.
> error_log would contain information about what is going on in more details.

Well, we think we fixed our problem. I believe this is the same bug as
discussed in

http://www.gossamer-threads.com/lists/modperl/modperl/100066

we found a couple places where $/ was being assigned-to globally, fixed
those with local, and now no more seg faults. Thanks for your help!

- Alex Aminoff
BaseSpace.net
National Bureau of Economic Research (nber.org)