print throwing intermittent Segfaults

This is a multi-part message in MIME format.

------_=_NextPart_001_01CA6A8F.0EE5C327
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi Everybody,

I'm having big problems with mod_perl throwing intermittent Segmentation =
faults our production machines on RHEL 4 & 5.
To be able to produce a core dump on this segfaults I've installed =
mod_dumpcore from this tutorial:
http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-li nux/

gdb /usr/sbin/httpd core.1 produces following output:

#0 0x00b29f4b in XS_Apache__RequestRec_content_type () from =
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/aut o/Apache/Requ=
estRec/RequestRec.so

and very rarely ( 1 in 15 )

#0 0x003830b9 in apr_palloc () from /usr/lib/libapr-0.so.0


The content-type is set by
$r->content_type("text/html; charset=3Diso-8859-1") but this is not what =
is causing him to segfault...

By try and error I've figured out that the segfault happens when I do a
$r->print($mypagecontent);

I've even tried to do a
unless($r->connection->aborted) {
$r->print($mypagecontent);
}
but this didn't help either.

The segfault happens randomly, between 30 and 250 mod_perl requests. =
There is no specific request URL or script that causes him to segfault, =
it just happens after some time.
More load on the server means more segfaults.

From my Apache Config:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 15
</IfModule>

There are some additional Perl Modules that I've build from CPAN:
Compress-Zlib-2.004
Digest-MD5-2.39
Email-MIME-1.861
Email-MIME-ContentType-1.014
Email-MIME-Encodings-1.311
Email-Simple-2.004
Encode-Detect-1.01
ExtUtils-CBuilder-0.23
File-Slurp-9999.12
IO-Compress-Zlib-2.004
MIME-Base64-3.07
MIME-Types-1.24
Module-Build-0.2808
Pod-Escapes-1.04
Pod-Simple-3.07
String-Similarity-1.03
Template-Plugin-XML-Escape-0.02
Test-Pod-1.26
Test-Simple-0.80

Has anyone a hint where to start looking and what to do next to figure =
out why this segfault is happening?

Thanks

Denis


------_=_NextPart_001_01CA6A8F.0EE5C327
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.5726" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>Hi =
Everybody,</FONT></DIV>=0A=
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>I'm having big problems =
with mod_perl throwing intermittent Segmentation faults our production =
machines on RHEL 4 & 5.<BR>To be able to produce a core dump on this =
segfaults I've installed mod_dumpcore from this tutorial:</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2><A =
href=3D"http://mituzas.lt/2009/09/26/getting-apache-core-dum ps-in-linux/"=
>http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-l inux/</A></FO=
NT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>gdb /usr/sbin/httpd =
core.1 produces following output:</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>#0  0x00b29f4b in =
XS_Apache__RequestRec_content_type ()  from =
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/aut o/Apache/Requ=
estRec/RequestRec.so</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>and very rarely ( 1 in =
15 )</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>#0  0x003830b9 in =
apr_palloc () from /usr/lib/libapr-0.so.0</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>The content-type is set =
by<BR>$r->content_type("text/html; charset=3Diso-8859-1") but this is =
not what is causing him to segfault...</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>By try and error I've =
figured out that the segfault happens when I do =
a<BR>$r->print($mypagecontent);</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>I've even tried to do =
a<BR></FONT><FONT face=3DArial color=3D#000000 =
size=3D2>unless($r->connection->aborted) {<BR>    =
$r->print($mypagecontent);<BR>}<BR> but this didn't help =
either.</FONT></DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>The segfault happens =
randomly, between 30 and 250 mod_perl requests. There is no specific =
request URL or script that causes him to segfault, it just happens after =
some time.<BR>More load on the server means more segfaults. </FONT></DIV>=0A=
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>=0A=
<DIV><FONT face=3DArial color=3D#000000 size=3D2>From my Apache =
Config:<BR><IfModule =
prefork.c><BR>StartServers       =
8<BR>MinSpareServers    5<BR>MaxSpareServers   =
20<BR>ServerLimit      =
256<BR>MaxClients       =
200<BR>MaxRequestsPerChild  =
15<BR></IfModule></FONT></DIV><FONT face=3DArial color=3D#000000 =
size=3D2>=0A=
<DIV><BR>There are some additional Perl Modules that I've build from =
CPAN:</DIV>=0A=
<DIV>Compress-Zlib-2.004<BR>Digest-MD5-2.39<BR>Email-MIME-1.861<BR>Email-=
MIME-ContentType-1.014<BR>Email-MIME-Encodings-1.311<BR>Email-Simple-2.00=
4<BR>Encode-Detect-1.01<BR>ExtUtils-CBuilder-0.23<BR>File-Slurp-9999.12<B=
R>IO-Compress-Zlib-2.004<BR>MIME-Base64-3.07<BR>MIME-Types-1.24<BR>Module=
-Build-0.2808<BR>Pod-Escapes-1.04<BR>Pod-Simple-3.07<BR>String-Similarity=
-1.03<BR>Template-Plugin-XML-Escape-0.02<BR>Test-Pod-1.26<BR>Test-Simple-=
0.80</DIV>=0A=
<DIV> </DIV>=0A=
<DIV>Has anyone a hint where to start looking and what to do next to =
figure out why this segfault is happening?</DIV>=0A=
<DIV> </DIV>=0A=
<DIV>Thanks</DIV>=0A=
<DIV> </DIV>=0A=
<DIV>Denis</DIV>=0A=
<DIV></FONT> </DIV></BODY></HTML>
------_=_NextPart_001_01CA6A8F.0EE5C327--
Denis Banovic [ Sa, 21 November 2009 10:43 ] [ ID #2023606 ]

Re: print throwing intermittent Segfaults

This is the list of stuff I usually start with when I get a problem
that doesn't seem to be tied to a particular code path.

* code path - perhaps a particular code path is only being exercised
rarely, and it has a bug
* forking - when child dies, all open descriptors in it's name space
also get closed
* eval - always a good thing to look at when weird things happen
* persistancy - globals, closures, persistant objects,
serialized/restored objects, shared memory, shared objects (between
processes)

-wjt
William T [ Sa, 21 November 2009 19:27 ] [ ID #2023611 ]

AW: print throwing intermittent Segfaults

Hi Willian,

Thanks for your checklist, I've run through it, segfaults still there...
Right now it takes less then a minute from apache restart to the first =
segfault.
This is from the error_log from the RedHat 5 Production machine:

Apache2::RequestIO::print: (103) Software caused connection abort at

The guys from rackspace are saying that I should recompile all my perl =
modules installed directly from CPAN ( see above ) , do you think this =
would help?
Or has someone another hint?

Thanks

Denis

-----Ursprüngliche Nachricht-----
Von: William T [mailto:dietbuddha [at] gmail.com]
Gesendet: Samstag, 21. November 2009 19:28
An: Denis Banovic
Cc: modperl [at] perl.apache.org
Betreff: Re: print throwing intermittent Segfaults

This is the list of stuff I usually start with when I get a problem that =
doesn't seem to be tied to a particular code path.

* code path - perhaps a particular code path is only being exercised =
rarely, and it has a bug
* forking - when child dies, all open descriptors in it's name space =
also get closed
* eval - always a good thing to look at when weird things happen
* persistancy - globals, closures, persistant objects, =
serialized/restored objects, shared memory, shared objects (between
processes)

-wjt


------------------------------------------------------------ -------------=
-------
Von: Denis Banovic [mailto:denis.banovic [at] ncm.at]
Gesendet: Samstag, 21. November 2009 10:43
An: modperl [at] perl.apache.org
Betreff: print throwing intermittent Segfaults


Hi Everybody,

I'm having big problems with mod_perl throwing intermittent Segmentation =
faults our production machines on RHEL 4 & 5.
To be able to produce a core dump on this segfaults I've installed =
mod_dumpcore from this tutorial:
http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-li nux/

gdb /usr/sbin/httpd core.1 produces following output:

#0 0x00b29f4b in XS_Apache__RequestRec_content_type () from =
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/aut o/Apache/Requ=
estRec/RequestRec.so

and very rarely ( 1 in 15 )

#0 0x003830b9 in apr_palloc () from /usr/lib/libapr-0.so.0


The content-type is set by
$r->content_type("text/html; charset=3Diso-8859-1") but this is not what =
is causing him to segfault...

By try and error I've figured out that the segfault happens when I do a
$r->print($mypagecontent);

I've even tried to do a
unless($r->connection->aborted) {
$r->print($mypagecontent);
}
but this didn't help either.

The segfault happens randomly, between 30 and 250 mod_perl requests. =
There is no specific request URL or script that causes him to segfault, =
it just happens after some time.
More load on the server means more segfaults.

From my Apache Config:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 15
</IfModule>

There are some additional Perl Modules that I've build from CPAN:
Compress-Zlib-2.004
Digest-MD5-2.39
Email-MIME-1.861
Email-MIME-ContentType-1.014
Email-MIME-Encodings-1.311
Email-Simple-2.004
Encode-Detect-1.01
ExtUtils-CBuilder-0.23
File-Slurp-9999.12
IO-Compress-Zlib-2.004
MIME-Base64-3.07
MIME-Types-1.24
Module-Build-0.2808
Pod-Escapes-1.04
Pod-Simple-3.07
String-Similarity-1.03
Template-Plugin-XML-Escape-0.02
Test-Pod-1.26
Test-Simple-0.80

Has anyone a hint where to start looking and what to do next to figure =
out why this segfault is happening?

Thanks

Denis
Denis Banovic [ Mo, 23 November 2009 09:27 ] [ ID #2023754 ]

Re: AW: print throwing intermittent Segfaults

Denis Banovic wrote:
> Hi Willian,
>
> Thanks for your checklist, I've run through it, segfaults still there...
> Right now it takes less then a minute from apache restart to the first segfault.
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
> Or has someone another hint?
>
Just my grain of salt : in my own experience, 99% of the "segfault"
cases I have encountered, was when Apache or Perl tried to run a piece
of code not meant for this machine (such as a library meant for another
machine or another OS version).
Maybe one of the modules you are using installed a wrong library ?
In that sense, the guys from rackspace may be right, although I believe
that the CPAN modules don't generally contain object-code libraries, or
else they do compile them at installation.
So maybe it is a library from the RHEL repository which is wrong.
aw [ Mo, 23 November 2009 09:46 ] [ ID #2023755 ]

RE: AW: print throwing intermittent Segfaults

Hi

I've had a similar error I "fixed" it by adding an eval block around the
offending code which was tracked back to MASON.

http://rt.cpan.org/Public/Bug/Display.html?id=3D49031

We compile everything from scratch apache,perl,mod_perl,mason, all =
modules
by an automated build script.

Earlier when we run on mod_perl1.99 and the redhat stack it worked fine.
But then we had other worries :-)

--
Morten Bjoernsvik, Developer, Decision Analytics


-----Original Message-----
From: Andr=E9 Warnier [mailto:aw [at] ice-sa.com]
Sent: 23. november 2009 09:46
To: mod_perl list
Subject: Re: AW: print throwing intermittent Segfaults

Denis Banovic wrote:
> Hi Willian,
>
> Thanks for your checklist, I've run through it, segfaults still =
there...
> Right now it takes less then a minute from apache restart to the first =
segfault.
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl =
modules installed directly from CPAN ( see above ) , do you think this =
would help?
> Or has someone another hint?
>
Just my grain of salt : in my own experience, 99% of the "segfault"
cases I have encountered, was when Apache or Perl tried to run a piece
of code not meant for this machine (such as a library meant for another
machine or another OS version).
Maybe one of the modules you are using installed a wrong library ?
In that sense, the guys from rackspace may be right, although I believe
that the CPAN modules don't generally contain object-code libraries, or
else they do compile them at installation.
So maybe it is a library from the RHEL repository which is wrong.
morten.bjornsvik [ Mo, 23 November 2009 10:15 ] [ ID #2023756 ]

AW: print throwing intermittent Segfaults [ solved ]

Hi Morten,

Thanks a lot,

By putting an eval around the code I found out, that the segfault was =
produced by next request to the same child after the $r->print failed.
$r->print is still failing from time to time, but it's not producing =
segfaults anymore!

Thanks

Denis


-----Ursprüngliche Nachricht-----
Von: Morten Bj=F8rnsvik [mailto:morten.bjornsvik [at] experian-da.no]
Gesendet: Montag, 23. November 2009 10:16
An: mod_perl list
Betreff: RE: AW: print throwing intermittent Segfaults

Hi

I've had a similar error I "fixed" it by adding an eval block around the =
offending code which was tracked back to MASON.

http://rt.cpan.org/Public/Bug/Display.html?id=3D49031

We compile everything from scratch apache,perl,mod_perl,mason, all =
modules by an automated build script.

Earlier when we run on mod_perl1.99 and the redhat stack it worked fine.
But then we had other worries :-)

--
Morten Bjoernsvik, Developer, Decision Analytics


-----Original Message-----
From: Andr=E9 Warnier [mailto:aw [at] ice-sa.com]
Sent: 23. november 2009 09:46
To: mod_perl list
Subject: Re: AW: print throwing intermittent Segfaults

Denis Banovic wrote:
> Hi Willian,
>
> Thanks for your checklist, I've run through it, segfaults still =
there...
> Right now it takes less then a minute from apache restart to the first =
segfault.
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl =
modules installed directly from CPAN ( see above ) , do you think this =
would help?
> Or has someone another hint?
>
Just my grain of salt : in my own experience, 99% of the "segfault"
cases I have encountered, was when Apache or Perl tried to run a piece =
of code not meant for this machine (such as a library meant for another =
machine or another OS version).
Maybe one of the modules you are using installed a wrong library ?
In that sense, the guys from rackspace may be right, although I believe =
that the CPAN modules don't generally contain object-code libraries, or =
else they do compile them at installation.
So maybe it is a library from the RHEL repository which is wrong.
Denis Banovic [ Mo, 23 November 2009 11:24 ] [ ID #2023757 ]

Re: AW: print throwing intermittent Segfaults

Hello,

I started to see the following when I setup a new server on CentOS 5.4
and installed perl modules from CPAN on Dec. 4:

Denis Banovic wrote:
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
> Or has someone another hint?
>
>
I do not see this on another server with CentOS 5.2 and perl modules
primarily built Feb 09.

I think something has changed in either CentOS, Apache, or the perl
modules I use in the interim.

I think it has something to do with Apache children being killed either
because of size or request limit reached.

So, Denis, have you tried adjusting these:

> >From my Apache Config:
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 200
> MaxRequestsPerChild 15
> </IfModule>
>
>
The MaxRequetsPerChild especially seems awfully small.

Plus, you say you see the problem happening every 30 to 250 requests.
(StartServers)8 * (MaxRequestsPerChild)15 = 120. This sounds like
roughtly around the time you'd expect Apache children to die.

I had MaxRequests set at 2000 and saw the problem intermittently. I'm
going to set MaxRequests to 0 to turn it off and see if it goes away.

I wonder if something has changed in Apache recently? Or, mod_perl?

Both servers are running Apache 2.2.3. One is running mod_perl 2.0.2 and
the other one is 2.0.4. Judging on the version numbers...could this be a
bug in 2.0.4?

Take care,

Kurt Hansen
Kurt Hansen [ Fr, 11 Dezember 2009 20:08 ] [ ID #2025565 ]
Webserver » gmane.comp.apache.mod-perl » print throwing intermittent Segfaults

Vorheriges Thema: avoiding child death by size limit
Nächstes Thema: bytes malloc-ed less bytes free-ed

[ Startseite ] [ Administrator ] [ Suche ] [ Hinweise ] [ Impressum ]

Powered by FudForum