Apache mutex error??

I'm upgrading my Apache install from 2.0.17 to 2.0.52. With the
upgrade, the server runs fine unless I run mod_ssl. When I use
mod_ssl, I get the following in my error log on startup:

[Mon Oct 11 08:30:20 2004] [notice] Digest: generating secret for
digest authentication ...
[Mon Oct 11 08:30:20 2004] [notice] Digest: done
[Mon Oct 11 08:30:21 2004] [notice] Apache/2.0.52 (Unix) DAV/2
PHP/4.3.9 mod_ssl/2.0.52 OpenSSL/0.9.7b configured -- resuming normal
operations
[Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Mon Oct 11 08:30:22 2004] [alert] Child 2031 returned a Fatal
error...\nApache is exiting!

And the server fails to start.

Has anybody seen this before? The Apache docs were of no help.

Thanks for your assistance.

Brent

Re: Apache mutex error??

Brent Miller wrote in message news:...
> I'm upgrading my Apache install from 2.0.17 to 2.0.52. With the
> upgrade, the server runs fine unless I run mod_ssl. When I use
> mod_ssl, I get the following in my error log on startup:
>
> [Mon Oct 11 08:30:20 2004] [notice] Digest: generating secret for
> digest authentication ...
> [Mon Oct 11 08:30:20 2004] [notice] Digest: done
> [Mon Oct 11 08:30:21 2004] [notice] Apache/2.0.52 (Unix) DAV/2
> PHP/4.3.9 mod_ssl/2.0.52 OpenSSL/0.9.7b configured -- resuming normal
> operations
> [Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
> the accept mutex
> [Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
> the accept mutex
> [Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
> the accept mutex
> [Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
> the accept mutex
> [Mon Oct 11 08:30:21 2004] [emerg] (13)Permission denied: couldn't grab
> the accept mutex
> [Mon Oct 11 08:30:22 2004] [alert] Child 2031 returned a Fatal
> error...\nApache is exiting!
>
> And the server fails to start.
>
> Has anybody seen this before? The Apache docs were of no help.

no.

but it looks like there's a permission problem while opening a mutex
(aka semaphore) - the obvious choice would be the AcceptMutex defined
by the directive of the same name used by about any apache2
multiprocessing module (mpm) there is to serialize children while
accepting requests - see

http://httpd.apache.org/docs-2.0/mod/mpm_common.html#acceptm utex

and for further reading on that the paragraph on accept serialization
in

http://httpd.apache.org/docs-2.0/misc/perf-tuning.html

the fact that this problem only occurs when using mod_ssl is a bit
strange but could be explained by your server having been compiled
with the SINGLE_LISTEN_UNSERIALIZED_ACCEPT option so no mutex is used
in case of a single socket server (just one ip and one port 80 - so no
problems) and when running with the additional ssl vhost the fact that
apache worker processes have no permissions to use whatever mutex is
configured bites you.

Or this is depite the error messages calling it accept mutex about the
SSLMutex used by mod_ssl to coordinate access to the ssl session
cache. same thing here though - if defined, apache workers need to be
able to access it.

Things to check (and maybe post here):

- os and platform
- mpm and compile options (httpd -V)
- AcceptMutex directive (apache worker needs to be allowed to use the
semaphore defined here, if any)
- LockFile directive (if AcceptMutex is flock or fcntl apache worker
needs to be able to write to this file).
- SSLMutexDirective (same here: apache worker needs to be allowed to
write the file in case of file/flock/fcntl or access the semaphore
otherwise)

joachim

Re: Apache mutex error??

Brent Miller wrote in message news:...
> I'm upgrading my Apache install from 2.0.17 to 2.0.52. With the
> upgrade, the server runs fine unless I run mod_ssl. When I use
> mod_ssl, I get the following in my error log on startup:
>
> [Mon Oct 11 08:30:20 2004] [notice] Digest: generating secret for
> digest authentication ...
> [Mon Oct 11 08:30:20 2004] [notice] Digest: done
> [Mon Oct 11 08:30:21 2004] [notice] Apache/2.0.52 (Unix) DAV/2
> PHP/4.3.9 mod_ssl/2.0.52 OpenSSL/0.9.7b configured -- resuming normal

oh, and while you're at it, upgrading OpenSSL to 0.9.7d might be a
good idea too, since there was some security problems in between.

joachim

Re: Apache mutex error??

After playing a bit, here's how the config stands:

> Things to check (and maybe post here):
>
> - os and platform

Mac OS X.3.5

> - mpm and compile options (httpd -V)

Server version: Apache/2.0.52
Server built: Sep 30 2004 18:20:43
Server's Module Magic Number: 20020903:9
Architecture: 32-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_MMAP
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT="/Library/Apache2"
-D SUEXEC_BIN="/Library/Apache2/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

> - AcceptMutex directive (apache worker needs to be allowed to use the
> semaphore defined here, if any)

AcceptMutex Default

> - LockFile directive (if AcceptMutex is flock or fcntl apache worker
> needs to be able to write to this file).

LockFile logs/accept.lock

> - SSLMutexDirective (same here: apache worker needs to be allowed to
> write the file in case of file/flock/fcntl or access the semaphore
> otherwise)

SSLMutex file:/Library/Apache2/logs/ssl_mutex


Neither the lock file nor the mutex file currently exist. Should they?

The behavior now is that apache will start, and serve properly off of
port 80, but still won't serve SSL (port 443) pages. The port isn't
being listened to. My error logs now read:

[Tue Oct 12 17:35:30 2004] [notice] Graceful restart requested, doing restart
[Tue Oct 12 17:35:30 2004] [notice] Digest: generating secret for
digest authentication ...
[Tue Oct 12 17:35:30 2004] [notice] Digest: done
[Tue Oct 12 17:35:30 2004] [notice] Apache/2.0.52 (Unix) PHP/4.3.9
DAV/2 configured -- resuming normal operations

Which is to say, they don't say much.

Brent

Re: Apache mutex error??

> Mac OS X.3.5

ok, default for Accept-Mutext on MacOS seems to be flock.

> > - mpm and compile options (httpd -V)
>
> Server version: Apache/2.0.52
> Server built: Sep 30 2004 18:20:43
> Server's Module Magic Number: 20020903:9
> Architecture: 32-bit
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_MMAP
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

that about explains why the errors only came when using more than one
port (ie. 80 plus 443 for ssl). when there was only port 80 used
(plain HTTP), no mutex is used at all.

> > - LockFile directive (if AcceptMutex is flock or fcntl apache worker
> > needs to be able to write to this file).
>
> LockFile logs/accept.lock

can the apache-worker user (the one specified with the User directive)
write to that directory (that logs directory is relative to your
ServerRoot - most probably it's /Library/Apache2/logs/)? Also it
_must_ be on a local filesytem.

> > - SSLMutexDirective (same here: apache worker needs to be allowed to
> > write the file in case of file/flock/fcntl or access the semaphore
> > otherwise)
>
> SSLMutex file:/Library/Apache2/logs/ssl_mutex
>
>
> Neither the lock file nor the mutex file currently exist. Should they?

SSLMutex should be present when mod_ssl is used. as it isn't currently
(nobody there on 443) it's ok for it to be missing. accept.lock should
be there with your settings when apache listens to more than one port.

Also there _should_ be logfiles in this directory unless you
configured them to be elsewhere with Error/Transfer/CustomLog
directives...

> The behavior now is that apache will start, and serve properly off of
> port 80, but still won't serve SSL (port 443) pages. The port isn't
> being listened to. My error logs now read:
>
> [Tue Oct 12 17:35:30 2004] [notice] Graceful restart requested, doing restart
> [Tue Oct 12 17:35:30 2004] [notice] Digest: generating secret for
> digest authentication ...
> [Tue Oct 12 17:35:30 2004] [notice] Digest: done
> [Tue Oct 12 17:35:30 2004] [notice] Apache/2.0.52 (Unix) PHP/4.3.9
> DAV/2 configured -- resuming normal operations

so you probably don't load mod_ssl at all or don't Listen to 443.

make shure that apache worker user can write to
/Library/Apache2/logs/, load mod_ssl again and check that there is a
Listen 443 in the config.

joachim

Re: Apache mutex error??

> can the apache-worker user (the one specified with the User directive)
> write to that directory (that logs directory is relative to your
> ServerRoot - most probably it's /Library/Apache2/logs/)? Also it
> _must_ be on a local filesytem.
>

Yes. I've verified the permissions, and the apache user has rwx perms
on those directories, which are on a local filesystem. Which makes
this problem even weirder.

>
> Also there _should_ be logfiles in this directory unless you
> configured them to be elsewhere with Error/Transfer/CustomLog
> directives...

There are.

I've figured out why my log files weren't showing that the SSL module
was trying to load. I've got the config such that if -DSSL is used
when starting apache, then all of the SSL module configuration
directives get processed. Apparently if I edit the apachectl file to
include -DSSL while Apache is running, and then issue a graceful, the
system uses the old version of the apachectl file and fails to load the
SSL directives. Issuing a stop and a start force it to load correctly,
which in this case means

[Wed Oct 20 15:25:59 2004] [notice] Apache/2.0.52 (Unix) DAV/2
PHP/4.3.9 mod_ssl/2.0.52 OpenSSL/0.9.7b configured -- resuming normal
operations
[Wed Oct 20 15:25:59 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Wed Oct 20 15:25:59 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Wed Oct 20 15:25:59 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Wed Oct 20 15:25:59 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Wed Oct 20 15:25:59 2004] [emerg] (13)Permission denied: couldn't grab
the accept mutex
[Wed Oct 20 15:26:00 2004] [alert] Child 6089 returned a Fatal
error...\nApache is exiting!

Thanks so much for your help so far. If you have any other suggestions
I'd love to have them.

Brent

Re: Apache mutex error??

Ignore all that. I just found a solution on another board. It seems
that changing the AcceptMutex directive to

AcceptMutex flock

explicitly, instead of "Default" fixes the issue. Problem solved.
Thanks for your help Joachim. :+)

Brent