I have a very simple sendmail MTA setup on my Redhat 2.6.20 system. It
handles only local requests, and sends out to a smart host on my ISP.

I have a problem though with floods of mails. I use fetchmail to get
my users' mail off POP3, then it directs locally. Problem is, it goes
through fetchmail->sendmail->spamd->clamd->procmail->destination. This
is fine with a few mails at a time (lets say under 10) but sometimes
it happens that many more mails will come in at once resulting in
everything choking itself. I have tried to read the doc on
MaxDaemonChildren and MaxQueueChildren, and tried using them, but it's
not obvious... and I have trouble telling which is responsible for
what.

I have fetchmail capped at only 5 messsages/user/fetch.
Spamd has maxchildren of 10
I want sendmail to receive the messages from fetchmail but queue them
unprocessed while more than 10 requests are currently being serviced.
Then process them locally nice and slowly.

What tweaks can I use to do this? I don't want sendmail refusing
connections, just queuing the messages. Can this be done for local
delivery?

Thanks!
Pat

On 08/16/07 15:33, pzaloum [at] gmail.com wrote:
> I have a problem though with floods of mails. I use fetchmail to get
> my users' mail off POP3, then it directs locally. Problem is, it goes
> through fetchmail->sendmail->spamd->clamd->procmail->destination.
> This is fine with a few mails at a time (lets say under 10) but
> sometimes it happens that many more mails will come in at once
> resulting in everything choking itself. I have tried to read the doc
> on MaxDaemonChildren and MaxQueueChildren, and tried using them, but
> it's not obvious... and I have trouble telling which is responsible
> for what.

Are you interfacing with spamd and clamd with milters or are you calling
them somehow out side of Sendmail? If you are interfacing with them as
milters Sendmail will be processing the messages as they are coming in
until the queue load average is reached.

Speaking of the queue load average, you could possibly lower it to the
point that Sendmail will accept messages and queue them for later
delivery once the load average drops below the queue load average level.
However I'm not entirely sure how this works when milters are involved
in the message accepting process. Can someone else help with this?

> I have fetchmail capped at only 5 messsages/user/fetch. Spamd has
> maxchildren of 10 I want sendmail to receive the messages from
> fetchmail but queue them unprocessed while more than 10 requests are
> currently being serviced. Then process them locally nice and slowly.

Look in to the queue load average setting for Sendmail.

> What tweaks can I use to do this? I don't want sendmail refusing
> connections, just queuing the messages. Can this be done for local
> delivery?

You could probably set a limit on how many messages fetchmail pulls in
at one time and then pause in between fetch runs. A small bit of shell
scripting should make this possible.



Grant. . . .

pzaloum [at] gmail.com unleashed the infinite monkeys on 16/08/2007 21:33
producing:
> I have a very simple sendmail MTA setup on my Redhat 2.6.20 system. It
> handles only local requests, and sends out to a smart host on my ISP.
>
> I have a problem though with floods of mails. I use fetchmail to get
> my users' mail off POP3, then it directs locally. Problem is, it goes
> through fetchmail->sendmail->spamd->clamd->procmail->destination. This
> is fine with a few mails at a time (lets say under 10) but sometimes
> it happens that many more mails will come in at once resulting in
> everything choking itself. I have tried to read the doc on
> MaxDaemonChildren and MaxQueueChildren, and tried using them, but it's
> not obvious... and I have trouble telling which is responsible for
> what.

It might help to know what you mean by choking - what is the error/problem.

I've got a similar setup, though using MIMEDefang to handle deal with
spam and virus scanning via the milter interface. This copes well even
with large bursts despite my very low end hardware.

As for fetchmail, you may also want to look to the batchlimit option.

--
Rob MacGregor (BOFH)

Rule 37: "There is no 'overkill'. There is only 'open fire'
and 'I need to reload.'"

On Aug 17, 11:58 am, Rob MacGregor <m... [at] privacy.net> wrote:
> pzal... [at] gmail.com unleashed the infinite monkeys on 16/08/2007 21:33
> producing:
>
> > I have a very simple sendmail MTA setup on my Redhat 2.6.20 system. It
> > handles only local requests, and sends out to a smart host on my ISP.
>
> > I have a problem though with floods of mails. I use fetchmail to get
> > my users' mail off POP3, then it directs locally. Problem is, it goes
> > through fetchmail->sendmail->spamd->clamd->procmail->destination. This
> > is fine with a few mails at a time (lets say under 10) but sometimes
> > it happens that many more mails will come in at once resulting in
> > everything choking itself. I have tried to read the doc on
> > MaxDaemonChildren and MaxQueueChildren, and tried using them, but it's
> > not obvious... and I have trouble telling which is responsible for
> > what.
>
> It might help to know what you mean by choking - what is the error/problem.
>
> I've got a similar setup, though using MIMEDefang to handle deal with
> spam and virus scanning via the milter interface. This copes well even
> with large bursts despite my very low end hardware.
>
> As for fetchmail, you may also want to look to the batchlimit option.
>
> --
> Rob MacGregor (BOFH)
>
> Rule 37: "There is no 'overkill'. There is only 'open fire'
> and 'I need to reload.'"

What i mean by choking is that sendmail will continue to accept
messages from fetchmail but the system can't cope with the load -
between spamassassin and clamav scanning each message (through
procmail) the ressources become saturated as the message count reaches
numbers greater than 10-15 emails at any instant. Spamassassin starts
to complain about BayesDB locking errors, the system CPU reaches 20-30
(enormous for a linux box of this size), sendmail times out after 2
minutes and the mails finally just go through unprocessed

pzaloum [at] gmail.com unleashed the infinite monkeys on 17/08/2007 19:32
producing:
<---SNIP--->
> What i mean by choking is that sendmail will continue to accept
> messages from fetchmail but the system can't cope with the load -
> between spamassassin and clamav scanning each message (through
> procmail) the ressources become saturated as the message count reaches
> numbers greater than 10-15 emails at any instant. Spamassassin starts
> to complain about BayesDB locking errors, the system CPU reaches 20-30
> (enormous for a linux box of this size), sendmail times out after 2
> minutes and the mails finally just go through unprocessed

What hardware have you got? I'm running something similar (as I said),
but calling spamassassin and clamd through MIMEDefang. On a 1.2 GHz box
with 512 MB of RAM it can handle streams of 20+ emails without major
problems.

The advantage of MD is that because it's a milter the processing happens
during the message acceptance, not after. This helps ensure that your
system doesn't become overloaded by processing emails after you've
accepted them.

--
Rob MacGregor (BOFH)

Rule 37: "There is no 'overkill'. There is only 'open fire'
and 'I need to reload.'"

"pzaloum [at] gmail.com" <pzaloum [at] gmail.com> writes:

> I have a very simple sendmail MTA setup on my Redhat 2.6.20 system. It
> handles only local requests, and sends out to a smart host on my ISP.
>
> I have a problem though with floods of mails. I use fetchmail to get
> my users' mail off POP3, then it directs locally. Problem is, it goes
> through fetchmail->sendmail->spamd->clamd->procmail->destination. This
> is fine with a few mails at a time (lets say under 10) but sometimes
> it happens that many more mails will come in at once resulting in
> everything choking itself. I have tried to read the doc on
> MaxDaemonChildren and MaxQueueChildren, and tried using them, but it's
> not obvious... and I have trouble telling which is responsible for
> what.
>
> I have fetchmail capped at only 5 messsages/user/fetch.
> Spamd has maxchildren of 10
> I want sendmail to receive the messages from fetchmail but queue them
> unprocessed while more than 10 requests are currently being serviced.
> Then process them locally nice and slowly.
>
> What tweaks can I use to do this? I don't want sendmail refusing
> connections, just queuing the messages. Can this be done for local
> delivery?

Have you considered removing sendmail from the delivery path?
fetchmail->procmail(spamd,clamd)->destination
or even
fetchmail->procmail->maildir
maildir->custom_script->procmail(spamd,clamd)->destination

--
[pl>en: Andrew] Andrzej Adam Filip : anfi [at] priv.onet.pl : anfi [at] xl.wp.pl
Speak softly and own a big, mean Doberman.
-- Dave Millman