--Apple-Mail-1-126206485
Content-Type: text/plain;
charset=US-ASCII;
format=flowed;
delsp=yes
Content-Transfer-Encoding: 7bit

Running Perl programs in mod_perl in Apache (2.2) on RHEL:

> [10 artg [at] virtualplant:/etc]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
> [11 artg [at] virtualplant:/etc]$ uname -r
> 2.6.18-164.11.1.el5

Occasionally a process grows so large that it freezes the system:

> several of them will use so much memory that kswapd takes all the CPU:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
and


> from /etc/httpd/logs/error_log
> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing
> the http daemon from connecting to network port 3306 For complete
> SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.
> Feb 24 14:55:06 virtualplant last message repeated 6 times
> Feb 24 15:00:44 virtualplant last message repeated 3 times
> Feb 24 15:00:55 virtualplant last message repeated 5 times
> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to
> 128.122.128.24 port 67
> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked oom-
> killer: gfp_mask=0xd0, order=0, oomkilladj=0
> Feb 24 15:09:51 virtualplant kernel:
> Feb 24 15:09:51 virtualplant kernel: Call Trace:
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff800c6076>]
> out_of_memory+0x8e/0x2f3
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8000f487>]
> __alloc_pages+0x245/0x2ce
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80017812>] cache_grow
> +0x133/0x3c1
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8005c2e5>]
> cache_alloc_refill+0x136/0x186
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8000ac12>]
> kmem_cache_alloc+0x6c/0x76
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80012658>] getname
> +0x25/0x1c2
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80019cba>]
> do_sys_open+0x17/0xbe
> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8005d28d>] tracesys
> +0xd5/0xe0

Then I need to cycle the box's power.

I'm implementing a multi-layer defense against this.

1) Try to prevent input that might blow up a process. However, this
will be imperfect.

2) Kill apache httpd processes occasionally, to control the effect of
slow perl memory leaks. I'll do this by setting MPM Worker
MaxRequestsPerChild to some modest value. (I'll try 100.)

3) Kill processes that grow too big, which concerns this message.

In bash, ulimit sets user resource limits. With mod_perl on Apache
Apache2::Resource controls the size of httpd processes. Both
eventually call setrlimit(int resource, const struct rlimit *rlim).

With Apache2::Resource one can put this in the httpd.conf:

> PerlModule Apache2::Resource
> # set child memory limit in megabytes
> # RLIMIT_AS (address space) will work to limit the size of a process
> on Linux
> PerlSetEnv PERL_RLIMIT_AS 1000:1100
> # this loads Apache2::Resource for each new httpd; that will set the
> ulimits from the Perl environment variables
> PerlChildInitHandler Apache2::Resource

OK, that kills big processes. What happens next is that Perl runs out
of memory (outputs "Out of Memory!") and calls the __DIE__ signal
handler. So, my plan is to catch the signal, redirect the browser to
an error page, and finally kill the process. Before the http Request
handler is called I say:

> $SIG{__DIE__} = \&oversizedHandler;
>
Then when __DIE__ fires the code below runs.

> use CGI;
> use English;
> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>
> # SIG handler called when __DIE__ fires
> sub oversizedHandler{
> my $a = shift;
> chomp $a;
> print STDERR "handler in process $PID called with '$a'\n";
>
> # up the soft AS limit to the hard limit, so that we've some RAM
> to use; in this example we free up 100 MB, much more than needed
> my $success = setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100
> * 1024 * 1024);
> if( $success ) {
> print STDERR "set limits to 512*1024 * 1024\n";
> }
>
> $cgi = CGI->new;
> print $cgi->redirect( -location => 'http://website.com/program.cgi¶m1=value1¶m2=value2)
> ;
>
> CORE::exit();
> }
>
Here's the problem. Nothing goes to STDOUT, so I cannot write to the
browser.

Thus, my question is: how can one kill an oversized process and
provide feedback to the user at the browser?

One alternative seems to be to use the special variable $^M (see the
perlvar manpage for more details) as recommended by the modperlbook.
How does one determine whether -DPERL_EMERGENCY_SBRK is defined, and
if one does that does STDOUT still work?

BR

A

Arthur P. Goldberg

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences
www.cs.nyu.edu/artg

artg [at] cs.nyu.edu
New York University
212 995-4918
Coruzzi Lab
8th Floor Silver Building
1009 Silver Center
100 Washington Sq East
New York NY 10003-6688




--Apple-Mail-1-126206485
Content-Type: text/html;
charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; "><div><div>Running Perl programs =
in mod_perl in Apache (2.2) on =
RHEL:</div><div><br></div><div><blockquote type=3D"cite"><font =
class=3D"Apple-style-span" color=3D"#000000">[10 =
artg [at] virtualplant:/etc]$ cat /etc/redhat-release<br>Red Hat Enterprise =
Linux Server release 5.4 (Tikanga)<br>[11 artg [at] virtualplant:/etc]$ uname =
-r<br>2.6.18-164.11.1.el5</font><br></blockquote></div></div><div><div><di=
v><p align=3D"">Occasionally a process grows so large that it freezes =
the system:</p><p align=3D""><blockquote type=3D"cite"><font =
class=3D"Apple-style-span" color=3D"#000000">several of them will use so =
much memory that kswapd takes all the CPU:<br><font =
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">   PID =
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>   349 =
root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1 <br>   348 root 20 =
-5 0 0 0 R 35.8 0.0 5:57.67 kswapd0</span></font></font><font =
class=3D"Apple-style-span" face=3D"'Courier New'" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: =
12px;"> </span></font></blockquote><p align=3D"">and</p></p><p =
align=3D""><blockquote type=3D"cite"><font class=3D"Apple-style-span" =
color=3D"#000000">from /etc/httpd/logs/error_log<br>Feb
24 14:35:32 virtualplant setroubleshoot: SELinux is preventing the http
daemon from connecting to network port 3306 For complete SELinux
messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84<br>Feb 24 =
14:35:34 virtualplant avahi-daemon[3133]: Invalid query packet.<br>Feb =
24 14:55:06 virtualplant last message repeated 6 times<br>Feb 24 =
15:00:44 virtualplant last message repeated 3 times<br>Feb 24 15:00:55 =
virtualplant last message repeated 5 times<br>Feb 24 15:01:21 =
virtualplant dhclient: DHCPREQUEST on eth0 to 128.122.128.24 port =
67<br>Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0<br>Feb 24 =
15:09:51 virtualplant kernel: <br>Feb 24 15:09:51 virtualplant kernel: =
Call Trace:<br>Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff800c6076>] out_of_memory+0x8e/0x2f3<br>Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff8000f487>] =
__alloc_pages+0x245/0x2ce<br>Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff80017812>] cache_grow+0x133/0x3c1<br>Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff8005c2e5>] =
cache_alloc_refill+0x136/0x186<br>Feb 24 15:09:51 virtualplant =
kernel:  [<ffffffff8000ac12>] =
kmem_cache_alloc+0x6c/0x76<br>Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff80012658>] getname+0x25/0x1c2<br>Feb 24 15:09:51 =
virtualplant kernel:  [<ffffffff80019cba>] =
do_sys_open+0x17/0xbe<br>Feb 24 15:09:51 virtualplant kernel:  =
[<ffffffff8005d28d>] =
tracesys+0xd5/0xe0<br></font></blockquote></p><p align=3D""><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">Then I need to cycle the box's =
power.</span></font></p><p align=3D""><font class=3D"Apple-style-span" =
size=3D"3"><span class=3D"Apple-style-span" style=3D"font-size: =
12px;">I'm implementing a multi-layer defense against =
this. </span></font></p><p align=3D""><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">1) Try to prevent input that might blow up a =
process. However, this will be imperfect.</span></font></p><p =
align=3D""><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">2) Kill apache =
httpd processes occasionally, to control the effect of slow perl memory =
leaks. I'll do this by setting </span></font><a =
href=3D"http://httpd.apache.org/docs/2.0/mod/worker.html"><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"><font class=3D"Apple-style-span" =
color=3D"#000000">MPM Worker</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"> </span></font><a =
href=3D"http://httpd.apache.org/docs/2.0/mod/mpm_common.html #maxrequestspe=
rchild"><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;"><font =
class=3D"Apple-style-span" =
color=3D"#000000">MaxRequestsPerChild</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"> to some modest value. (I'll try =
100.)</span></font></p><p align=3D""><font class=3D"Apple-style-span" =
size=3D"3"><span class=3D"Apple-style-span" style=3D"font-size: =
12px;">3) Kill processes that grow too big, which concerns this =
message. </span></font></p><p align=3D""><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">In bash, ulimit sets user resource limits. =
With mod_perl on Apache </span></font><a =
href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod"><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;"><font =
class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"> controls the size of httpd processes. =
Both eventually call setrlimit(int resource, const struct rlimit =
*rlim). </span></font></p><p align=3D""><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">With  </span></font><a =
href=3D"http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ api/Apache2/Re=
source.pod"><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;"><font =
class=3D"Apple-style-span" =
color=3D"#000000">Apache2::Resource</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"> one can put this in the =
httpd.conf:</span></font></p><p align=3D""><pre><font =
class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;"><blockquote =
type=3D"cite"><span class=3D"Apple-style-span" style=3D"font-size: =
medium; white-space: normal; "><pre><font class=3D"Apple-style-span" =
face=3D"Helvetica" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px; ">PerlModule Apache2::Resource
# set child memory limit in megabytes
# RLIMIT_AS (address space) will work to limit the size of a process on =
Linux
PerlSetEnv PERL_RLIMIT_AS 1000:1100</span></font></pre><pre><font =
class=3D"Apple-style-span" face=3D"Helvetica" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px; "># this loads =
Apache2::Resource for each new httpd; that will set the ulimits from the =
Perl environment variables
PerlChildInitHandler Apache2::Resource =
</span></font></pre></span></blockquote></span></font></pre></p><p =
align=3D""><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">OK, that kills big =
processes. What happens next is that Perl runs out of memory =
(outputs "Out of Memory!") and calls the __DIE__ signal handler. So, my =
plan is to catch the signal, redirect the browser to an error page, and =
finally kill the process. Before the http Request handler is called I =
say:</span></font></p><blockquote type=3D"cite"><p =
align=3D"">$SIG{__DIE__} =3D =
\&oversizedHandler;<br></p></blockquote><p align=3D"">Then =
when __DIE__ fires the code below runs.</p><blockquote =
type=3D"cite">use CGI; <br>use English;<br>use BSD::Resource =
qw(setrlimit getrlimit get_rlimits getrusage);<br><br># SIG handler =
called when __DIE__ fires<br><p align=3D"">sub =
oversizedHandler{<br>    my $a =3D shift;<br>   =
 chomp $a;<br>    print STDERR "handler in process =
$PID called with '$a'\n";<br><br>    # up the soft AS =
limit to the hard limit, so that we've some RAM to use; in this example =
we free up 100 MB, much more than =
needed<br>        my $success =3D =
setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 * 1024 * =
1024);<br>    if( $success ) {<br>   =
     print STDERR "set limits to 512*1024 * =
1024\n";<br>    }</p><p align=3D"">   =
 $cgi =3D CGI->new;<br>   =
 print $cgi->redirect( -location =3D> '<a =
href=3D"http://website.com/program.cgi&param1=3Dvalue1&a mp;param2=3Dva=
lue2)">http://website.com/program.cgi&param1=3Dvalue1&am p;param2=3Dval=
ue2)</a>;</p><p align=3D"">   =
 CORE::exit();<br>}</p></blockquote><p align=3D""><span =
class=3D"Apple-style-span" style=3D"background-color: transparent; =
"><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">Here's the =
problem. Nothing goes to STDOUT, so I cannot write to the =
browser. </span></font></span></p><p align=3D""><span =
class=3D"Apple-style-span" style=3D"background-color: =
transparent;"><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">Thus, my question =
is: how can one kill an oversized process and provide feedback to the =
user at the browser?</span></font></span></p><p align=3D""><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">One alternative seems to be to use the =
special variable $^M (</span></font><a =
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml"><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"><font class=3D"Apple-style-span" =
color=3D"#000000">see the </font></span></font></a><a =
href=3D"http://perldoc.perl.org/perlvar.html"><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"><font class=3D"Apple-style-span" =
color=3D"#000000">perlvar</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"> manpage for more =
details</span></font><font class=3D"Apple-style-span" size=3D"3"><span =
class=3D"Apple-style-span" style=3D"font-size: 12px;">) as recommended =
by the </span></font><a =
href=3D"http://modperlbook.org/html/22-3-10-Out-of-memory.ht ml"><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;"><font class=3D"Apple-style-span" =
color=3D"#000000">modperlbook</font></span></font></a><font =
class=3D"Apple-style-span" size=3D"3"><span class=3D"Apple-style-span" =
style=3D"font-size: 12px;">. How does one determine =
whether -DPERL_EMERGENCY_SBRK is defined, and if one does that does =
STDOUT still work?</span></font></p><p align=3D""><font =
class=3D"Apple-style-span" color=3D"#00000000"><span =
class=3D"Apple-style-span" style=3D"background-color: transparent; =
">BR</span></font></p><p align=3D""><font class=3D"Apple-style-span" =
color=3D"#00000000"><span class=3D"Apple-style-span" =
style=3D"background-color: =
transparent;">A</span></font></p></div></div></div><div =
apple-content-edited=3D"true"><div><div><div><div align=3D"">Arthur P. =
Goldberg</div><div align=3D""><br></div><div align=3D"">Research =
Scientist in Bioinformatics</div><div align=3D"">Plant Systems Biology =
Laboratory</div><div align=3D""><a =
href=3D"http://www.virtualplant.org">www.virtualplant.org</a></div><div =
align=3D""><br></div><div align=3D"">Visiting Academic</div><div =
align=3D"">Computer Science Department</div><div align=3D"">Courant =
Institute of Mathematical Sciences</div><div align=3D""><a =
href=3D"http://www.cs.nyu.edu/artg">www.cs.nyu.edu/artg</a></div><div =
align=3D""><br></div><div align=3D""><a =
href=3D"mailto:artg [at] cs.nyu.edu">artg [at] cs.nyu.edu</a></div><div =
align=3D"">New York University</div><div align=3D"">212 =
995-4918</div><div align=3D"">Coruzzi Lab</div><div align=3D"">8th Floor =
Silver Building</div><div align=3D""><div align=3D"">1009 Silver =
Center</div><div align=3D"">100 Washington Sq East<br></div><div =
align=3D"">New York NY =
10003-6688</div><br></div></div></div></div><br></div><br></body></html>=

--Apple-Mail-1-126206485--

On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG <artg [at] cs.nyu.edu> wrote:
> 2) Kill apache httpd processes occasionally, to control the effect of slo=
w
> perl memory leaks. I'll do this by setting=A0MPM Worker=A0MaxRequestsPerC=
hild=A0to
> some modest value. (I'll try 100.)

You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.

> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process.

A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.

Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.

> How
> does one determine whether=A0-DPERL_EMERGENCY_SBRK is defined, and if one=
does
> that does STDOUT still work?

It's really not much different from what you're already doing, but you
can check your compile options with perl -V.

- Perrin

On Thu, Mar 11, 2010 at 1:41 PM, ARTHUR GOLDBERG <artg [at] cs.nyu.edu> wrote:
>
> OK, that kills big processes.=A0What happens next is that Perl runs out o=
f
> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. S=
o,
> my plan is to catch the signal, redirect the browser to an error page, an=
d
> finally kill the process. Before the http Request handler is called I say=
:

Too complicated. =A0Use Apache2::SizeLimit. =A0It runs as a cleanup
handler which means your user will have gotten served, and then the
child will die. =A0You can also write your own cleanup handler to
monitor sizes and if you want without having to do anything like
catching a __DIE__.

> Thus, my question is: how can one kill an oversized process and provide
> feedback to the user at the browser?

In general kill during cleanup phase. =A0Apache2::SizeLimit does just that.

I would also argue that you should probably fix your memory leaks.
Sometimes all it takes is a few days of analysis. =A0That's what
happened with me.

You can also segment out functionality to different Apache servers so
that every Apache/mod_perl child does not have to pull in the entire
code base for the website.

-wjt

Am 11.03.2010 um 22:41 schrieb ARTHUR GOLDBERG:

> Running Perl programs in mod_perl in Apache (2.2) on RHEL:
>
>> [10 artg [at] virtualplant:/etc]$ cat /etc/redhat-release
>> Red Hat Enterprise Linux Server release 5.4 (Tikanga)
>> [11 artg [at] virtualplant:/etc]$ uname -r
>> 2.6.18-164.11.1.el5
> Occasionally a process grows so large that it freezes the system:

I had a similar problem, ending with the kernel message:
no more processes to kill, giving up.

so I installed
Apache2::Resource
and set
# set limit to 500MB
PerlSetEnv PERL_RLIMIT_AS 500
PerlChildInitHandler Apache2::Resource

and now the problem dossn't appear again.

And I don't think that Apache2::SizeLimit
can handle this issue, since it checks the size after the request, which =
may help if you have a memory leak,
but my problem was caused by a single request and the process grew very =
fast during this request, eating up all memory.
Setting RLIMIT is handled by the OS , so that process get's killed when =
it grows to large.

>
>> several of them will use so much memory that kswapd takes all the =
CPU:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 349 root 10 -5 0 0 0 R 37.7 0.0 5:53.56 kswapd1
>> 348 root 20 -5 0 0 0 R 35.8 0.0 5:57.67 kswapd0
> and
>
>
>
>> from /etc/httpd/logs/error_log
>> Feb 24 14:35:32 virtualplant setroubleshoot: SELinux is preventing =
the http daemon from connecting to network port 3306 For complete =
SELinux messages. run sealert -l 0afcfa46-07b8-48eb-aec3-e7dda9872b84
>> Feb 24 14:35:34 virtualplant avahi-daemon[3133]: Invalid query =
packet.
>> Feb 24 14:55:06 virtualplant last message repeated 6 times
>> Feb 24 15:00:44 virtualplant last message repeated 3 times
>> Feb 24 15:00:55 virtualplant last message repeated 5 times
>> Feb 24 15:01:21 virtualplant dhclient: DHCPREQUEST on eth0 to =
128.122.128.24 port 67
>> Feb 24 15:09:51 virtualplant kernel: hald-addon-stor invoked =
oom-killer: gfp_mask=3D0xd0, order=3D0, oomkilladj=3D0
>> Feb 24 15:09:51 virtualplant kernel:
>> Feb 24 15:09:51 virtualplant kernel: Call Trace:
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff800c6076>] =
out_of_memory+0x8e/0x2f3
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8000f487>] =
__alloc_pages+0x245/0x2ce
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80017812>] =
cache_grow+0x133/0x3c1
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8005c2e5>] =
cache_alloc_refill+0x136/0x186
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8000ac12>] =
kmem_cache_alloc+0x6c/0x76
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80012658>] =
getname+0x25/0x1c2
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff80019cba>] =
do_sys_open+0x17/0xbe
>> Feb 24 15:09:51 virtualplant kernel: [<ffffffff8005d28d>] =
tracesys+0xd5/0xe0
>
> Then I need to cycle the box's power.
>
> I'm implementing a multi-layer defense against this.
>
> 1) Try to prevent input that might blow up a process. However, this =
will be imperfect.
>
> 2) Kill apache httpd processes occasionally, to control the effect of =
slow perl memory leaks. I'll do this by setting MPM Worker =
MaxRequestsPerChild to some modest value. (I'll try 100.)
>
> 3) Kill processes that grow too big, which concerns this message.
>
> In bash, ulimit sets user resource limits. With mod_perl on Apache =
Apache2::Resource controls the size of httpd processes. Both eventually =
call setrlimit(int resource, const struct rlimit *rlim).
>
> With Apache2::Resource one can put this in the httpd.conf:
>
>
>> PerlModule Apache2::Resource
>> # set child memory limit in megabytes
>> # RLIMIT_AS (address space) will work to limit the size of a process =
on Linux
>> PerlSetEnv PERL_RLIMIT_AS 1000:1100
>>
>> # this loads Apache2::Resource for each new httpd; that will set the =
ulimits from the Perl environment variables
>> PerlChildInitHandler Apache2::Resource
>>
>
> OK, that kills big processes. What happens next is that Perl runs out =
of memory (outputs "Out of Memory!") and calls the __DIE__ signal =
handler. So, my plan is to catch the signal, redirect the browser to an =
error page, and finally kill the process. Before the http Request =
handler is called I say:
>
>> $SIG{__DIE__} =3D \&oversizedHandler;
>>
> Then when __DIE__ fires the code below runs.
>
>> use CGI;
>> use English;
>> use BSD::Resource qw(setrlimit getrlimit get_rlimits getrusage);
>>
>> # SIG handler called when __DIE__ fires
>> sub oversizedHandler{
>> my $a =3D shift;
>> chomp $a;
>> print STDERR "handler in process $PID called with '$a'\n";
>>
>> # up the soft AS limit to the hard limit, so that we've some RAM =
to use; in this example we free up 100 MB, much more than needed
>> my $success =3D setrlimit('RLIMIT_AS', 1100*1024 * 1024, 1100 =
* 1024 * 1024);
>> if( $success ) {
>> print STDERR "set limits to 512*1024 * 1024\n";
>> }
>>
>> $cgi =3D CGI->new;
>> print $cgi->redirect( -location =3D> =
'http://website.com/program.cgi¶m1=3Dvalue1¶m2=3Dval ue2);
>>
>> CORE::exit();
>> }
>>
> Here's the problem. Nothing goes to STDOUT, so I cannot write to the =
browser.
>
> Thus, my question is: how can one kill an oversized process and =
provide feedback to the user at the browser?
>
> One alternative seems to be to use the special variable $^M (see the =
perlvar manpage for more details) as recommended by the modperlbook. How =
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one =
does that does STDOUT still work?
>
> BR
>
> A
>
> Arthur P. Goldberg
>
> Research Scientist in Bioinformatics
> Plant Systems Biology Laboratory
> www.virtualplant.org
>
> Visiting Academic
> Computer Science Department
> Courant Institute of Mathematical Sciences
> www.cs.nyu.edu/artg
>
> artg [at] cs.nyu.edu
> New York University
> 212 995-4918
> Coruzzi Lab
> 8th Floor Silver Building
> 1009 Silver Center
> 100 Washington Sq East
> New York NY 10003-6688
>
>
>

Mit freundlichen Grüßen
Rolf Schaufelberger
Geschäftsführer

plusW GmbH
Vorstadtstr. 61 -67 Tel. 07181 47 47 305
73614 Schorndorf Fax. 07181 47 45 344

www.plusw.de
www.mypixler.com
www.calendrino.de

This is a multi-part message in MIME format.
--------------040807000909090007080907
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Perrin

Thanks all for your responses. I'd like to try your suggestion, but
cannot find documentation on "apache error page handlers". Presumably
that's code that Apache runs if an httpd dies.
mod_perl: HTTP
<http://perl.apache.org/docs/2.0/user/handlers/http.html>/Handlers
<http://perl.apache.org/docs/2.0/user/handlers/http.html> describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors./
Perhaps you're referring to Apache's ErrorDocument
<http://httpd.apache.org/docs/2.0/mod/core.html#errordocument> (also
described in Custom Error Response
<http://httpd.apache.org/docs/2.0/custom-error.html>). While that can
run local Perl (in a new process of course) as in the example
| | ErrorDocument 500 /cgi-bin/crash-recover.pl||
it isn't clear to me how to access $r (which I assume means the Request).
However, I could write a little info to a file which that
crash-recover.pl reads. It will only run occasionally.
And given your observation
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in crash-recover.pl. Alternatively, I could
write a handler that did

$original_handler = $SIG{__DIE__};

sub myHandler {
if( out_of_memory ) {
# increase RLIMIT_AS a little so memory can be allocated
# write some info (especially user identity) to a file, that
crash-recover.pl will use
} else {
$original_handler( [at] _ );
}
}

BR
A


Perrin Harkins wrote:
> On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG <artg [at] cs.nyu.edu> wrote:
>
>> 2) Kill apache httpd processes occasionally, to control the effect of slow
>> perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
>> some modest value. (I'll try 100.)
>>
>
> You definitely should be doing that, and possibly running
> Apache2::SizeLimit as well. There's plenty of documentation about
> this on the site.
>
>
>> OK, that kills big processes. What happens next is that Perl runs out of
>> memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
>> my plan is to catch the signal, redirect the browser to an error page, and
>> finally kill the process.
>>
>
> A simpler solution would be to set something in $r->notes from your
> DIE handler and use that in an apache error page handler to determine
> what to show.
>
> Also, be careful with using a universal DIE handler, since it kills
> exception handling in code that uses eval {} blocks.
>
>
>> How
>> does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
>> that does STDOUT still work?
>>
>
> It's really not much different from what you're already doing, but you
> can check your compile options with perl -V.
>
> - Perrin
>
>

--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg

artg [at] cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building



--------------040807000909090007080907
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Perrin<br>
<div><br>
</div>
<div>Thanks all for your responses. I'd like to try
your suggestion, but cannot find documentation on "apache error page
handlers". Presumably that's code that Apache runs if an httpd dies.</div>
<div><span class="Apple-style-span"
style="font-size: 14px; font-weight: bold;"><a
href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">mod_perl:
HTTP </a><em><a
href="http://perl.apache.org/docs/2.0/user/handlers/http.htm l"
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNEW2Gk7TPYzMXspQfhW1RyR0_7XgA ','','0CAYQFjAA')">Handlers</a><span
class="Apple-style-span"
style="font-size: medium; font-style: normal; font-weight: normal;"> describes
mod_perl's complete HTTP Request processing, but doesn't handle fatal
Perl errors.</span></em></span></div>
<div>Perhaps you're referring to Apache's <a
href="http://httpd.apache.org/docs/2.0/mod/core.html#errordo cument">ErrorDocument</a>
(also described in <a
href="http://httpd.apache.org/docs/2.0/custom-error.html">Cu stom Error
Response</a>). While that can run local Perl (in a new process of
course) as in the example</div>
<div><font size="+1"><code>       </code></font>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">ErrorDocument
500 /cgi-bin/crash-recover.pl</span></span><font size="+1"><code></code></font></div>
<div>it isn't clear to me how to access $r (which I assume means the
Request).</div>
<div>However, I could write a little info to a file which that <span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl</span></span>
reads. It will only run occasionally.<br>
And given your observation<br>
<blockquote type="cite">
<pre wrap="">Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.
</pre>
</blockquote>
it might be better to not assign a special handler to $SIG{__DIE__} but
rather to just handle stuff in <span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl.
</span></span>Alternatively, I could write a handler that did<br>
<br>
$original_handler = $SIG{__DIE__};<br>
<br>
sub myHandler {<br>
if( out_of_memory ) {<br>
   # increase RLIMIT_AS a little so memory can be allocated<br>
   # write some info (especially user identity) to a file, that <span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 20px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="font-family: 'Courier New',Courier,monospace; font-size: 19px; line-height: 20px;">crash-recover.pl</span></span>
will use<br>
} else {<br>
  $original_handler( [at] _ );<br>
}<br>
}<br>
<br>
BR</div>
<div apple-content-edited="true">
<div>
<div>
<div>
<div align=""><font class="Apple-style-span"><span
class="Apple-style-span" style="white-space: normal;">A</span></font></div>
</div>
</div>
</div>
</div>
<br>
<br>
Perrin Harkins wrote:
<blockquote
cite="mid:66887a3d1003111404m2127564dq1ce498ddd769498 [at] mail.g mail.com"
type="cite">
<pre wrap="">On Thu, Mar 11, 2010 at 4:41 PM, ARTHUR GOLDBERG <a class="moz-txt-link-rfc2396E" href="mailto:artg [at] cs.nyu.edu"><artg [at] cs.nyu.edu></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">2) Kill apache httpd processes occasionally, to control the effect of slow
perl memory leaks. I'll do this by setting MPM Worker MaxRequestsPerChild to
some modest value. (I'll try 100.)
</pre>
</blockquote>
<pre wrap=""><!---->
You definitely should be doing that, and possibly running
Apache2::SizeLimit as well. There's plenty of documentation about
this on the site.

</pre>
<blockquote type="cite">
<pre wrap="">OK, that kills big processes. What happens next is that Perl runs out of
memory (outputs "Out of Memory!") and calls the __DIE__ signal handler. So,
my plan is to catch the signal, redirect the browser to an error page, and
finally kill the process.
</pre>
</blockquote>
<pre wrap=""><!---->
A simpler solution would be to set something in $r->notes from your
DIE handler and use that in an apache error page handler to determine
what to show.

Also, be careful with using a universal DIE handler, since it kills
exception handling in code that uses eval {} blocks.

</pre>
<blockquote type="cite">
<pre wrap="">How
does one determine whether -DPERL_EMERGENCY_SBRK is defined, and if one does
that does STDOUT still work?
</pre>
</blockquote>
<pre wrap=""><!---->
It's really not much different from what you're already doing, but you
can check your compile options with perl -V.

- Perrin

</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
<a class="moz-txt-link-abbreviated" href="http://www.virtualplant.org">www.virtualplant.org</a>

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences <a class="moz-txt-link-abbreviated" href="http://www.cs.nyu.edu/artg">www.cs.nyu.edu/artg</a>

<a class="moz-txt-link-abbreviated" href="mailto:artg [at] cs.nyu.edu">artg [at] cs.nyu.edu</a>
New York University
100 Washington Sq East
8th Floor Silver Building

</pre>
</body>
</html>

--------------040807000909090007080907--

On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg <artg [at] cs.nyu.edu> wrote:
> Perhaps you're referring to Apache's=A0ErrorDocument (also described in C=
ustom
> Error Response).

Yes, that's what I meant.

> While that can run local Perl (in a new process of course)
> as in the example
> =A0=A0 =A0 =A0 ErrorDocument 500 /cgi-bin/crash-recover.pl
> it isn't clear to me how to access $r (which I assume means the Request).

You can send it to any URL you like, including one handled by mod_perl.

- Perrin

This is a multi-part message in MIME format.
--------------000202040908030509030209
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Thanks Perrin

Unfortunately, this doesn't work.
First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache
ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.

Second, if Apache detects a server error and redirects to a script, that
script runs in a new process and has no access to the request object $r.
It can access the REDIRECT environment variables as discussed in Custom
Error Response <http://httpd.apache.org/docs/2.0/custom-error.html> .

At this point I think that the best thing to do is use
MaxRequestsPerChild
<http://httpd.apache.org/docs/2.0/mod/mpm_common.html#maxrequestsperchild>
and Apache2::SizeLimit
<http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/api/Apache2/SizeLimit.pod>
to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.

BR
A

Perrin Harkins wrote:
> On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg <artg [at] cs.nyu.edu> wrote:
>
>> Perhaps you're referring to Apache's ErrorDocument (also described in Custom
>> Error Response).
>>
>
> Yes, that's what I meant.
>
>
>> While that can run local Perl (in a new process of course)
>> as in the example
>> ErrorDocument 500 /cgi-bin/crash-recover.pl
>> it isn't clear to me how to access $r (which I assume means the Request).
>>
>
> You can send it to any URL you like, including one handled by mod_perl.
>
> - Perrin
>
>

--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
www.virtualplant.org

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg

artg [at] cs.nyu.edu
New York University
100 Washington Sq East
8th Floor Silver Building



--------------000202040908030509030209
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Thanks Perrin<br>
<br>
Unfortunately, this doesn't work. <br>
First, failure of a mod_perl process with "Out of Memory!", as occurs
when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache
<br>
ErrorDocument 500. A mod_perl process that exits (actually CORE::exit()
must be called), that doesn't trigger an ErrorDocument 500 either.<br>
<br>
Second, if Apache detects a server error and redirects to a script,
that script runs in a new process and has no access to the request
object $r. It can access the REDIRECT environment variables as
discussed in <span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"><span
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"><a
href="http://httpd.apache.org/docs/2.0/custom-error.html"
style="color: rgb(85, 26, 139);">Custom Error Response</a></span></span>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
..<br>
<br>
At this point I think that the best thing to do is use
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"><span
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"><span
class="Apple-converted-space"> </span><a
href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#m axrequestsperchild"
id="hmps" title="MaxRequestsPerChild">MaxRequestsPerChild</a></span></span>
and
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"><span
class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"><a
href="http://search.cpan.org/%7Egozer/mod_perl-2.0.4/docs/ap i/Apache2/SizeLimit.pod"
id="j:1-" title="Apache2::SizeLimit">Apache2::SizeLimit</a></span></span>
to handle most memory problems, and simply processes that blow up die
without feedback to users. Not ideal, but they should be extremely rare
events.<br>
<br>
BR<br>
A<br>
<br>
Perrin Harkins wrote:
<blockquote
cite="mid:66887a3d1003151555i4704ea31w8100fbb20042e266 [at] mail. gmail.com"
type="cite">
<pre wrap="">On Sun, Mar 14, 2010 at 11:22 AM, Arthur Goldberg <a class="moz-txt-link-rfc2396E" href="mailto:artg [at] cs.nyu.edu"><artg [at] cs.nyu.edu></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Perhaps you're referring to Apache's ErrorDocument (also described in Custom
Error Response).
</pre>
</blockquote>
<pre wrap=""><!---->
Yes, that's what I meant.

</pre>
<blockquote type="cite">
<pre wrap="">While that can run local Perl (in a new process of course)
as in the example
       ErrorDocument 500 /cgi-bin/crash-recover.pl
it isn't clear to me how to access $r (which I assume means the Request).
</pre>
</blockquote>
<pre wrap=""><!---->
You can send it to any URL you like, including one handled by mod_perl.

- Perrin

</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Arthur P. Goldberg, PhD

Research Scientist in Bioinformatics
Plant Systems Biology Laboratory
<a class="moz-txt-link-abbreviated" href="http://www.virtualplant.org">www.virtualplant.org</a>

Visiting Academic
Computer Science Department
Courant Institute of Mathematical Sciences <a class="moz-txt-link-abbreviated" href="http://www.cs.nyu.edu/artg">www.cs.nyu.edu/artg</a>

<a class="moz-txt-link-abbreviated" href="mailto:artg [at] cs.nyu.edu">artg [at] cs.nyu.edu</a>
New York University
100 Washington Sq East
8th Floor Silver Building

</pre>
</body>
</html>

--------------000202040908030509030209--