Strange behaviour with Pseudo-Proxy script

This is a multi-part message in MIME format.

------=_NextPart_000_004A_01CB49DC.ADF15C20
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi,



I am having a strange issue with a mod_perl handler which I've written
lately.



A little background. we are using a mod_perl script for our self-developed
MS .NET application.

The application connects to the frontend server, where the mod_perl
"proxy" is running. The script

does some kind of load_balancing and then proxies the request to the
backend and providest the

answer from the backend back to the application. This concept is working
fine, but the initial version

of the proxy script is poorly written. It first collects all data from the
client in memory (which might be

a pretty high amount of data (up to 100MB)), then send the stuff to the
backend and then provides

the answer. This causes bad memory consumption- especially if there are
more than 100 concurrent

users. Also the script is no real mod_perl script, but uses
ModPerl::Registry instead.



So I've re-written the whole script from scratch. I've written it as
"real" mod_perl handler module.

It's now working with chunks of data to be sent and received. Everything
seems to working fine-

except of one thing. The .NET app has the ability to upload a file to the
backend. With the original

script the upload usually has at least about 1-2MB/sec bandwidth. but with
the new script that

I've written, it only provides a throughput of about 300kb/sec. I did
couple of profiling already,

but wasn't able to find the cause of this.



So I am wondering if someone of the mod_perl community has some advices or
hints, how to resolve

this issue. Every help or thought-provoking impulse is highly appreciated.




Thanks

Winni


------=_NextPart_000_004A_01CB49DC.ADF15C20
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
[at] font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
[at] font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
..MsoChpDefault
{mso-style-type:export-only;}
[at] page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>

<body lang=3DDE link=3Dblue vlink=3Dpurple>

<div class=3DWordSection1>

<p class=3DMsoNormal>Hi,<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal><span lang=3DEN-US>I am having a strange issue with =
a mod_perl
handler which I’ve written lately.<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>A little background… we =
are using a
mod_perl script for our self-developed MS .NET =
application.<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>The application connects to the =
frontend
server, where the mod_perl “proxy” is running. The =
script<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>does some kind of load_balancing =
and then proxies
the request to the backend and providest the<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>answer from the backend back to =
the
application. This concept is working fine, but the initial =
version<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>of the proxy script is poorly =
written. It
first collects all data from the client in memory (which might =
be<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>a pretty high amount of data (up =
to
100MB)), then send the stuff to the backend and then provides =
<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>the answer… This causes =
bad memory
consumption- especially if there are more than 100 =
concurrent<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>users. Also the script is no =
real mod_perl
script, but uses ModPerl::Registry instead.<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>So I’ve re-written the =
whole script
from scratch. I’ve written it as “real” mod_perl =
handler
module.<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>It’s now working with =
chunks of data
to be sent and received. Everything seems to working fine- =
<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>except of one thing… The =
..NET app has
the ability to upload a file to the backend. With the original =
<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>script the upload usually has at =
least
about 1-2MB/sec bandwidth… but with the new script that =
<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>I’ve written, it only =
provides a
throughput of about 300kb/sec. I did couple of profiling =
already,<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>but wasn’t able to find =
the cause of
this. <o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>So I am wondering if someone of =
the
mod_perl community has some advices or hints, how to =
resolve<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>this issue. Every help or =
thought-provoking
impulse is highly appreciated.<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US><br>
Thanks<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US>Winni<o:p></o:p></span></p>

</div>

</body>

</html>

------=_NextPart_000_004A_01CB49DC.ADF15C20--
Winfried Neessen [ Mi, 01 September 2010 13:50 ] [ ID #2046991 ]

Re: Strange behaviour with Pseudo-Proxy script

Aside from posting the source code so that we can peruse and say "That
might be it", you might try putting a non blocking reverse proxy such
as Perlbal (my favorite, and darn tootin' fast), Nginx, or Varnish in
front of your mod_perl instance so that it does the dirty work of
connection handling and mod_perl isn't stuck spoon feeding slow
clients.


On Wed, Sep 1, 2010 at 4:50 AM, Winfried Neessen
<neessen [at] cleverbridge.com> wrote:
> Hi,
>
>
>
> I am having a strange issue with a mod_perl handler which I=92ve written
> lately.
>
>
>
> A little background=85 we are using a mod_perl script for our self-develo=
ped
> MS .NET application.
>
> The application connects to the frontend server, where the mod_perl =93pr=
oxy=94
> is running. The script
>
> does some kind of load_balancing and then proxies the request to the back=
end
> and providest the
>
> answer from the backend back to the application. This concept is working
> fine, but the initial version
>
> of the proxy script is poorly written. It first collects all data from th=
e
> client in memory (which might be
>
> a pretty high amount of data (up to 100MB)), then send the stuff to the
> backend and then provides
>
> the answer=85 This causes bad memory consumption- especially if there are=
more
> than 100 concurrent
>
> users. Also the script is no real mod_perl script, but uses
> ModPerl::Registry instead.
>
>
>
> So I=92ve re-written the whole script from scratch. I=92ve written it as =
=93real=94
> mod_perl handler module.
>
> It=92s now working with chunks of data to be sent and received. Everythin=
g
> seems to working fine-
>
> except of one thing=85 The .NET app has the ability to upload a file to t=
he
> backend. With the original
>
> script the upload usually has at least about 1-2MB/sec bandwidth=85 but w=
ith
> the new script that
>
> I=92ve written, it only provides a throughput of about 300kb/sec. I did c=
ouple
> of profiling already,
>
> but wasn=92t able to find the cause of this.
>
>
>
> So I am wondering if someone of the mod_perl community has some advices o=
r
> hints, how to resolve
>
> this issue. Every help or thought-provoking impulse is highly appreciated=
..
>
>
>
> Thanks
>
> Winni
Fred Moyer [ Mi, 01 September 2010 20:21 ] [ ID #2046992 ]

RE: Strange behaviour with Pseudo-Proxy script

Hi Fred,

here is the code:
Proxy.pm: http://dokuleser.privatepaste.com/dd9f56cb09/vu0bxLIICY
Proxy/AppServer.pm:
http://dokuleser.privatepaste.com/36b60ba9e2/byzjGiLki0
Proxy/Header.pm: http://dokuleser.privatepaste.com/e306920a14/rhxc8kRUFB

Thanks
Winni

-----Original Message-----
From: Fred Moyer [mailto:fred [at] redhotpenguin.com]
Sent: Wednesday, September 01, 2010 8:21 PM
To: Winfried Neessen
Cc: modperl [at] perl.apache.org
Subject: Re: Strange behaviour with Pseudo-Proxy script

Aside from posting the source code so that we can peruse and say "That
might be it", you might try putting a non blocking reverse proxy such
as Perlbal (my favorite, and darn tootin' fast), Nginx, or Varnish in
front of your mod_perl instance so that it does the dirty work of
connection handling and mod_perl isn't stuck spoon feeding slow
clients.


On Wed, Sep 1, 2010 at 4:50 AM, Winfried Neessen
<neessen [at] cleverbridge.com> wrote:
> Hi,
>
>
>
> I am having a strange issue with a mod_perl handler which I've written
> lately.
>
>
>
> A little background. we are using a mod_perl script for our
self-developed
> MS .NET application.
>
> The application connects to the frontend server, where the mod_perl
"proxy"
> is running. The script
>
> does some kind of load_balancing and then proxies the request to the
backend
> and providest the
>
> answer from the backend back to the application. This concept is working
> fine, but the initial version
>
> of the proxy script is poorly written. It first collects all data from
the
> client in memory (which might be
>
> a pretty high amount of data (up to 100MB)), then send the stuff to the
> backend and then provides
>
> the answer. This causes bad memory consumption- especially if there are
more
> than 100 concurrent
>
> users. Also the script is no real mod_perl script, but uses
> ModPerl::Registry instead.
>
>
>
> So I've re-written the whole script from scratch. I've written it as
"real"
> mod_perl handler module.
>
> It's now working with chunks of data to be sent and received. Everything
> seems to working fine-
>
> except of one thing. The .NET app has the ability to upload a file to
the
> backend. With the original
>
> script the upload usually has at least about 1-2MB/sec bandwidth. but
with
> the new script that
>
> I've written, it only provides a throughput of about 300kb/sec. I did
couple
> of profiling already,
>
> but wasn't able to find the cause of this.
>
>
>
> So I am wondering if someone of the mod_perl community has some advices
or
> hints, how to resolve
>
> this issue. Every help or thought-provoking impulse is highly
appreciated.
>
>
>
> Thanks
>
> Winni
Winfried Neessen [ Do, 09 September 2010 11:07 ] [ ID #2047412 ]

RE: Strange behaviour with Pseudo-Proxy script

Hi again,

sorry forgot Proxy/Sender.pm in my last mail.
http://dokuleser.privatepaste.com/05e9b4f124/QHvZO0RMoi


Winni

-----Original Message-----
From: Winfried Neessen [mailto:neessen [at] cleverbridge.com]
Sent: Thursday, September 09, 2010 11:07 AM
To: modperl [at] perl.apache.org
Subject: RE: Strange behaviour with Pseudo-Proxy script

Hi Fred,

here is the code:
Proxy.pm: http://dokuleser.privatepaste.com/dd9f56cb09/vu0bxLIICY
Proxy/AppServer.pm:
http://dokuleser.privatepaste.com/36b60ba9e2/byzjGiLki0
Proxy/Header.pm: http://dokuleser.privatepaste.com/e306920a14/rhxc8kRUFB

Thanks
Winni

-----Original Message-----
From: Fred Moyer [mailto:fred [at] redhotpenguin.com]
Sent: Wednesday, September 01, 2010 8:21 PM
To: Winfried Neessen
Cc: modperl [at] perl.apache.org
Subject: Re: Strange behaviour with Pseudo-Proxy script

Aside from posting the source code so that we can peruse and say "That
might be it", you might try putting a non blocking reverse proxy such
as Perlbal (my favorite, and darn tootin' fast), Nginx, or Varnish in
front of your mod_perl instance so that it does the dirty work of
connection handling and mod_perl isn't stuck spoon feeding slow
clients.


On Wed, Sep 1, 2010 at 4:50 AM, Winfried Neessen
<neessen [at] cleverbridge.com> wrote:
> Hi,
>
>
>
> I am having a strange issue with a mod_perl handler which I've written
> lately.
>
>
>
> A little background. we are using a mod_perl script for our
self-developed
> MS .NET application.
>
> The application connects to the frontend server, where the mod_perl
"proxy"
> is running. The script
>
> does some kind of load_balancing and then proxies the request to the
backend
> and providest the
>
> answer from the backend back to the application. This concept is working
> fine, but the initial version
>
> of the proxy script is poorly written. It first collects all data from
the
> client in memory (which might be
>
> a pretty high amount of data (up to 100MB)), then send the stuff to the
> backend and then provides
>
> the answer. This causes bad memory consumption- especially if there are
more
> than 100 concurrent
>
> users. Also the script is no real mod_perl script, but uses
> ModPerl::Registry instead.
>
>
>
> So I've re-written the whole script from scratch. I've written it as
"real"
> mod_perl handler module.
>
> It's now working with chunks of data to be sent and received. Everything
> seems to working fine-
>
> except of one thing. The .NET app has the ability to upload a file to
the
> backend. With the original
>
> script the upload usually has at least about 1-2MB/sec bandwidth. but
with
> the new script that
>
> I've written, it only provides a throughput of about 300kb/sec. I did
couple
> of profiling already,
>
> but wasn't able to find the cause of this.
>
>
>
> So I am wondering if someone of the mod_perl community has some advices
or
> hints, how to resolve
>
> this issue. Every help or thought-provoking impulse is highly
appreciated.
>
>
>
> Thanks
>
> Winni
Winfried Neessen [ Do, 09 September 2010 11:22 ] [ ID #2047413 ]

RE: Strange behaviour with Pseudo-Proxy script

Hi again,

I was able to figure out the problem on my own. After re-reading the
mod_perl2
documentation of Apache2::RequestIO I figured that I forgot to enable
autoflush
on STDOUT, which causes the $r->print() call to be buffered.

Regards
Winni


-----Original Message-----
From: Winfried Neessen [mailto:neessen [at] cleverbridge.com]
Sent: Thursday, September 09, 2010 11:22 AM
To: modperl [at] perl.apache.org
Subject: RE: Strange behaviour with Pseudo-Proxy script

Hi again,

sorry forgot Proxy/Sender.pm in my last mail.
http://dokuleser.privatepaste.com/05e9b4f124/QHvZO0RMoi


Winni

-----Original Message-----
From: Winfried Neessen [mailto:neessen [at] cleverbridge.com]
Sent: Thursday, September 09, 2010 11:07 AM
To: modperl [at] perl.apache.org
Subject: RE: Strange behaviour with Pseudo-Proxy script

Hi Fred,

here is the code:
Proxy.pm: http://dokuleser.privatepaste.com/dd9f56cb09/vu0bxLIICY
Proxy/AppServer.pm:
http://dokuleser.privatepaste.com/36b60ba9e2/byzjGiLki0
Proxy/Header.pm: http://dokuleser.privatepaste.com/e306920a14/rhxc8kRUFB

Thanks
Winni

-----Original Message-----
From: Fred Moyer [mailto:fred [at] redhotpenguin.com]
Sent: Wednesday, September 01, 2010 8:21 PM
To: Winfried Neessen
Cc: modperl [at] perl.apache.org
Subject: Re: Strange behaviour with Pseudo-Proxy script

Aside from posting the source code so that we can peruse and say "That
might be it", you might try putting a non blocking reverse proxy such
as Perlbal (my favorite, and darn tootin' fast), Nginx, or Varnish in
front of your mod_perl instance so that it does the dirty work of
connection handling and mod_perl isn't stuck spoon feeding slow
clients.


On Wed, Sep 1, 2010 at 4:50 AM, Winfried Neessen
<neessen [at] cleverbridge.com> wrote:
> Hi,
>
>
>
> I am having a strange issue with a mod_perl handler which I've written
> lately.
>
>
>
> A little background. we are using a mod_perl script for our
self-developed
> MS .NET application.
>
> The application connects to the frontend server, where the mod_perl
"proxy"
> is running. The script
>
> does some kind of load_balancing and then proxies the request to the
backend
> and providest the
>
> answer from the backend back to the application. This concept is working
> fine, but the initial version
>
> of the proxy script is poorly written. It first collects all data from
the
> client in memory (which might be
>
> a pretty high amount of data (up to 100MB)), then send the stuff to the
> backend and then provides
>
> the answer. This causes bad memory consumption- especially if there are
more
> than 100 concurrent
>
> users. Also the script is no real mod_perl script, but uses
> ModPerl::Registry instead.
>
>
>
> So I've re-written the whole script from scratch. I've written it as
"real"
> mod_perl handler module.
>
> It's now working with chunks of data to be sent and received. Everything
> seems to working fine-
>
> except of one thing. The .NET app has the ability to upload a file to
the
> backend. With the original
>
> script the upload usually has at least about 1-2MB/sec bandwidth. but
with
> the new script that
>
> I've written, it only provides a throughput of about 300kb/sec. I did
couple
> of profiling already,
>
> but wasn't able to find the cause of this.
>
>
>
> So I am wondering if someone of the mod_perl community has some advices
or
> hints, how to resolve
>
> this issue. Every help or thought-provoking impulse is highly
appreciated.
>
>
>
> Thanks
>
> Winni
Winfried Neessen [ Do, 09 September 2010 12:39 ] [ ID #2047414 ]

Re: Strange behaviour with Pseudo-Proxy script

Winfried Neessen schrieb am 09.09.2010 um 12:39 (+0200):
> After re-reading the mod_perl2 documentation of Apache2::RequestIO I
> figured that I forgot to enable autoflush on STDOUT, which causes the
> $r->print() call to be buffered.

So to sum it up, no autoflush, hence no buffering, hence the performance
drop you mentioned in your original mail. Correct?

--
Michael Ludwig
Michael Ludwig [ Do, 09 September 2010 20:41 ] [ ID #2047415 ]

RE: Strange behaviour with Pseudo-Proxy script

No, not exactly...

No autoflush -> buffered output -> weird performance issues.


Winni


-----Original Message-----
From: Michael Ludwig [mailto:milu71 [at] gmx.de]
Sent: Thursday, September 09, 2010 8:41 PM
To: modperl [at] perl.apache.org
Subject: Re: Strange behaviour with Pseudo-Proxy script

Winfried Neessen schrieb am 09.09.2010 um 12:39 (+0200):
> After re-reading the mod_perl2 documentation of Apache2::RequestIO I
> figured that I forgot to enable autoflush on STDOUT, which causes the
> $r->print() call to be buffered.

So to sum it up, no autoflush, hence no buffering, hence the performance
drop you mentioned in your original mail. Correct?

--
Michael Ludwig
Winfried Neessen [ Fr, 10 September 2010 14:00 ] [ ID #2047480 ]
Webserver » gmane.comp.apache.mod-perl » Strange behaviour with Pseudo-Proxy script

Vorheriges Thema: [RELEASE CANDIDATE] Apache-Test-1.33 RC1
Nächstes Thema: Very special handler