There appears to be a major memory leak in mod

This is a multi-part message in MIME format.

------_=_NextPart_001_01C3E752.CA1743A6
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I have been tracking this down for a couple of weeks and thought it was
in the code my company is developing and it appears that is not the
case. In order to eliminate our code from the mix and isolate the
problem here is what I did:
This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for
Linux use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for
the second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ..... -p pid(httpN)
watching the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all
verified that it leaks like a sieve.
We were all trying to figure out why no one else has complained about
such a huge leak so we ran another test. We tried using the prefork MPM
and it turns out that worked fine. Based on the results it appears the
OS is cleaning up memory for the prefork module and the threaded model
never gets its memory freed. I have used a debugger on Windows and set
break points on the CRYPTO_malloc and CRYPTO_free functions and have
seen gobs of memory CRYPTO_malloc(ed) and not one time have I seen
CRYPTO_free called. I was not sure if having the OS cleanup memory was
part of the design (if indeed that is what is happening) or if there is
potentially a problem in the OpenSSL memory management code.
With all this said, I am by no means an expert on this code and could
really use some help understanding what is going on here?
Any and all help is appreciated,
Ken

------_=_NextPart_001_01C3E752.CA1743A6
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.0.6487.1">
<TITLE>There appears to be a major memory leak in =
mod_ssl/OpenSSL</TITLE>
</HEAD>
<BODY>


I have been tracking this down for a =
couple of weeks and thought it was in the code my company is developing =
and it appears that is not the case. In order to eliminate our code from =
the mix and isolate the problem here is what I did:

This was done on Windows and =
Linux: 
1. Download the latest Apache =
from <A HREF=3D"file://www.apache.org">www.apache.org</A>. 
2. Download the latest OpenSSL =
from <A HREF=3D"file://www.openssl.org">www.openssl.org</A>. 
3. Build them both, with apache add =
the mod_ssl option and also for Linux use the MPM worker =
module. 
4. Install and modify the ssl.conf =
file ServerName value. 
5. Run Apache (httpd) 
6a. Run the Performance monitor on =
Windows and look at Private Bytes for the second Apache =
process. 
6b. On Linux run top -p pid(httpd1) =
-p pid(http2) ….. -p pid(httpN) watching the size of the =
processes 
7. Set you browser to not cache =
requests and check for a new page every time. 
8. Start fetching a page from =
<A HREF=3D"https://localhost">https://localhost</A> and keep refreshing the page. 

 So far 3 other engineers have =
reproduced this test because they did not believe the problem could be =
in Apache mod_ssl/OpenSSL, they all verified that it leaks like a =
sieve.

We were all trying to figure out why no =
one else has complained about such a huge leak so we ran another test. =
We tried using the prefork MPM and it turns out that worked fine. Based =
on the results it appears the OS is cleaning up memory for the prefork =
module and the threaded model never gets its memory freed. I have used a =
debugger on Windows and set break points on the CRYPTO_malloc and =
CRYPTO_free functions and have seen gobs of memory CRYPTO_malloc(ed) and =
not one time have I seen CRYPTO_free called. I was not sure if having =
the OS cleanup memory was part of the design (if indeed that is what is =
happening) or if there is potentially a problem in the OpenSSL memory =
management code.

With all this said, I am by no means an =
expert on this code and could really use some help understanding what is =
going on here?

Any and all help is =
appreciated, 
Ken 


</BODY>
</HTML>
------_=_NextPart_001_01C3E752.CA1743A6--
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it was
> in the code my company is developing and it appears that is not the
> case. In order to eliminate our code from the mix and isolate the
> problem here is what I did:
> This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.
> 2. Download the latest OpenSSL from www.openssl.org.
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

There appears to be a major memory leak in mod_ssl/OpenSSLHello,

I encountered many memory leak trouble with OpenSSL. I used quite old
versions (from 0.9.6c), so I don't know if this is relevant or not for
you... Anyway I figured out that nobody seemed to ever call the
CRYPTO_thread_cleanup(). I just added a call to that function, and the
memory heap became clean. I hope the solution will be that simple in your
case.

The OpenSSL code did not look robust to me on that matter... But I think
that OpenSSL people are trying to (or did) improve the cleanup code.

Adrien

----- Original Message -----
From: Avery, Ken
To: modssl-users [at] modssl.org
Sent: Friday, January 30, 2004 6:02 PM
Subject: There appears to be a major memory leak in mod_ssl/OpenSSL

I have been tracking this down for a couple of weeks and thought it was in
the code my company is developing and it appears that is not the case. In
order to eliminate our code from the mix and isolate the problem here is
what I did:
This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for Linux
use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for the
second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN) watching
the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all verified
that it leaks like a sieve.
We were all trying to figure out why no one else has complained about such a
huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.
With all this said, I am by no means an expert on this code and could really
use some help understanding what is going on here?
Any and all help is appreciated,
Ken

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.574 / Virus Database: 364 - Release Date: 29/01/2004

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

This is a multi-part message in MIME format.

------=_NextPart_000_0056_01C3E757.657A7100
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

There appears to be a major memory leak in mod_ssl/OpenSSLAre u using:

> On linux you really should be using a shared memory session cache - like
> SSLSessionCache shmcb:logs/ssl_gcache_data(512000)

> SSLSessionCacheTimeout 300

and not the dbm cache

I posted some email about this just before XMAS where I had found a "memory
leak" - and Mads Toftum suggested the use of shmcb. I then ran tests for
nearly a week - without a hint of a memory leak

John
-----Original Mess
age-----
From: owner-modssl-users [at] modssl.org
[mailto:owner-modssl-users [at] modssl.org]On Behalf Of Avery, Ken
Sent: 30 January 2004 17:02
To: modssl-users [at] modssl.org
Subject: There appears to be a major memory leak in mod_ssl/OpenSSL

I have been tracking this down for a couple of weeks and thought it was in
the code my company is developing and it appears that is not the case. In
order to eliminate our code from the mix and isolate the problem here is
what I did:

This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for Linux
use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for
the second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN)
watching the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all verified
that it leaks like a sieve.

We were all trying to figure out why no one else has complained about such
a huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.

With all this said, I am by no means an expert on this code and could
really use some help understanding what is going on here?

Any and all help is appreciated,
Ken

------=_NextPart_000_0056_01C3E757.657A7100
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>There appears to be a major memory leak in =
mod_ssl/OpenSSL</TITLE>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.2800.1170" name=3DGENERATOR></HEAD>
<BODY>
<DIV>Are u
using:</DIV>
<DIV> </DIV>
<DIV>> On linux you =
really should
be using a shared memory session cache - like</DIV>
> SSLSessionCache shmcb:logs/ssl_gcache_data(512000)
> SSLSessionCacheTimeout 300
<DIV> </DIV>
<DIV>and not the dbm
cache </DIV>
<DIV> </DIV>
<DIV>I
posted some email about this just before XMAS where I had found a =
"memory leak"
- and Mads Toftum suggested the use of shmcb. I =
then ran
tests for nearly a week - without a hint of a memory
leak</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>John</DIV>
<BLOCKQUOTE dir=3Dltr
style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px =
solid; MARGIN-RIGHT: 0px">
<DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft>-----Original Mess </DIV>
<DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft> age----- From:
owner-modssl-users [at] modssl.org =
[mailto:owner-modssl-users [at] modssl.org]On
Behalf Of Avery, Ken Sent: 30 January 2004 =
17:02 To:
modssl-users [at] modssl.org Subject: There appears to be a major =
memory
leak in mod_ssl/OpenSSL </DIV>
I have been tracking this down for a =
couple of
weeks and thought it was in the code my company is developing and it =
appears
that is not the case. In order to eliminate our code from the mix and =
isolate
the problem here is what I did:
This was done on Windows and =
Linux: 1. =
Download the
latest Apache from <A
href=3D"file://www.apache.org">www.apache.org</A>. 2. =
Download the
latest OpenSSL from <A
href=3D"file://www.openssl.org">www.openssl.org</A>. 3. =
Build them both,
with apache add the mod_ssl option and also for Linux use the MPM =
worker
module. 4. Install and modify the ssl.conf file ServerName =
value. 5. Run =
Apache
(httpd) 6a. Run the Performance monitor on Windows and look at =
Private Bytes
for the second Apache process. 6b. On =
Linux run top
-p pid(httpd1) -p pid(http2) ….. -p pid(httpN) watching the size =
of the
processes 7. Set you browser to not cache requests and check for a new =
page every
time. 8.
Start fetching a page from <A =
href=3D"https://localhost">https://localhost</A> and keep refreshing the page. So =
far 3 other
engineers have reproduced this test because they did not believe the =
problem
could be in Apache mod_ssl/OpenSSL, they all verified that it leaks =
like a
sieve.
We were all trying to figure out why no =
one else
has complained about such a huge leak so we ran another test. We tried =
using
the prefork MPM and it turns out that worked fine. Based on the =
results it
appears the OS is cleaning up memory for the prefork module and the =
threaded
model never gets its memory freed. I have used a debugger on Windows =
and set
break points on the CRYPTO_malloc and CRYPTO_free functions and have =
seen gobs
of memory CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free =
called. I
was not sure if having the OS cleanup memory was part of the design =
(if indeed
that is what is happening) or if there is potentially a problem in the =
OpenSSL
memory management code.
With all this said, I am by no means an =
expert on
this code and could really use some help understanding what is going =
on
here?
Any and all help is =
appreciated, Ken </BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_0056_01C3E757.657A7100--

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

I have tried the shmcb and that does not help, I think the real issue
has to do with worker MPM verses prefork MPM. Note: Linux worker MPM and
Windows mpm_winnt MPM are threaded verses prefork MPM which has its own
memory space.

I will try the patches Joe recommended and see what happens.

Thanks Joe,
Ken

BTW - I am using 2.0.48, I just download the latest and see the problem.

-----Original Message-----
From: Joe Orton [mailto:jorton [at] redhat.com]
Sent: Friday, January 30, 2004 11:13 AM
To: Avery, Ken
Cc: modssl-users [at] modssl.org
Subject: Re: There appears to be a major memory leak in mod_ssl/OpenSSL

On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it
> was in the code my company is developing and it appears that is not
> the case. In order to eliminate our code from the mix and isolate the
> problem here is what I did: This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.
> 2. Download the latest OpenSSL from www.openssl.org.
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

I just tried the 2 patches listed below and they did not make a
difference, using mpm_winnt and worker MPM.

-----Original Message-----
From: Joe Orton [mailto:jorton [at] redhat.com]
Sent: Friday, January 30, 2004 11:13 AM
To: Avery, Ken
Cc: modssl-users [at] modssl.org
Subject: Re: There appears to be a major memory leak in mod_ssl/OpenSSL

On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it
> was in the code my company is developing and it appears that is not
> the case. In order to eliminate our code from the mix and isolate the
> problem here is what I did: This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.
> 2. Download the latest OpenSSL from www.openssl.org.
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users [at] modssl.org
Automated List Manager majordomo [at] modssl.org

There appears to be a major memory leak in mod_ssl/OpenSSL

Re: There appears to be a major memory leak in mod_ssl/OpenSSL

Re: There appears to be a major memory leak in mod_ssl/OpenSSL

RE: There appears to be a major memory leak in mod_ssl/OpenSSL

RE: There appears to be a major memory leak in mod_ssl/OpenSSL

RE: There appears to be a major memory leak in mod_ssl/OpenSSL