Sendmail LDAP access milter

I've just uploaded an updated version of my LDAP milter to
www.symas.com, available for free download. For those of you who have
or are thinking about migrating your access_db into LDAP, you should
realize that Sendmail's map lookup algorithm is extremely ill-suited
for use with LDAP. I documented the problem in an earlier post to this
newsgroup:
http://groups.google.com/group/comp.mail.sendmail/msg/b45f87 9876a96280?hl=en&

With the update my ldapmap milter now supports the ClientRate and
ClientConn keywords, so you can do rate control without needing the
access_db feature. It also supports the GreetPause feature, but you
need to patch Sendmail to support it. The patch is included with the
ldapmap source code. It adds a new smfi_setsymval function to
libmilter, which allows a milter to set a macro in the Sendmail
session. It also patches the Sendmail server to check for a
{greet_pause} macro and use it directly, bypassing the ruleset lookup
(and the horrid access_db lookup) if set. The milter sets the
{greet_pause} macro and everything goes on from there.

For an explanation of why Sendmail's map algorithm is so terrible, and
why mine is several orders of magnitude more efficient:

Sendmail's map lookup is designed to work with embedded databases,
using simple key/value pair lookups. This approach by its very nature
embodies a flat namespace. When Sendmail tries to lookup a hostname
like host.sub.domain it searches for "host.sub.domain", "sub.domain",
and "domain" all as separate lookups, until it finds a match. It also
does a lookup on the IP address, in the same manner, looking for
127.0.0.1, 127.0.0, 127.0, and 127 until it finds a match. On top of
that it also parses the usernames, so if an email has the address
"joe.blog@my.big.network.com" Sendmail will have to look for "com",
"network.com", "big.network.com", "my.big.network.com",
"blog@my.big.network.com" and "joe.blog@my.big.network.com" until it
finds a match.

This is extremely inefficient for a number of reasons, not least of
which is that hostnames and IP addresses are inherently hierarchical
namespaces, not flat. And furthermore, LDAP is a hierarchical
namespace, and there's a natural, efficient mapping of these hostname
and IP address hierarchies onto LDAP, which Sendmail does not use.

Keep in mind that Sendmail does all of those lookups for every possible
keyword that you enable in accessdb. So you're not just talking about 8
or 10 queries for one sender address and another 8 or 10 queries for
one recipient address. You're talking about 10 queries per possible
keyword - Connect, From, To, Spam, ClientRate, ClientConn, GreetPause -
every feature you enable magnifies the query load.

My milter can get all of the needed information out of an LDAP server
in 2 queries, tops. It does this by using LDAP the way it was intended
- mapping hierarchical names into hierarchical DNs. For example, if you
had these two rules:
Connect:goodhost.some.big.network.com OK
Connect:network.com REJECT
GreetPause:network.com 2000
GreetPause:goodhost.some.big.network.com 0

Sendmail would turn those into four separate LDAP entries. With my code
they map to only two entries:

dn: dc=goodhost,dc=some,dc=big,dc=network,dc=com,
lmConnect: OK
lmGreetPause: 0

dn: dc=network,dc=com,
lmConnect: REJECT
lmGreetPause: 1000

An email coming in from "goodhost" will get all of its relevant rules
in a single LDAP lookup. Sendmail's access map would take at least 2
searches. An email coming in from badhost.some.big.network.com would
take two lookups with my code, one for
"dc=badhost,dc=some,dc=big,dc=network,dc=com" which will fail, but will
return a matchedDN of "dc=network,dc=com". The second lookup using the
matchedDN will return the desired result. With Sendmail, it would take
at least 8 searches. (And that assumes that Sendmail is *only* looking
for the Connect and GreetPause rules, but most likely it will also look
for From and various other rules too.) The more features you turn on in
your accessdb, the more queries Sendmail's accessdb will need. Hosts
that aren't even in the map will always result in the worst case - a
separate lookup for each component of the hostname, plus a separate
lookup for each component of the IP address, multiplied by the number
of enabled keywords.

My milter will always require no more than two queries per hostname.

By the way - this milter is also more efficient than Sendmail's DNSBL
implementation, for the same reasons. I've been using it to filter spam
on my highlandsun.com server for 3-4 years, it definitely makes life
easier.

--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

Re: Sendmail LDAP access milter

hyc wrote:
> I've just uploaded an updated version of my LDAP milter to
> www.symas.com, available for free download.

I have added your milter to the non-open source section of this list
kept here

http://www.jmaimon.com/sendmail/milters

Let me know if I got it right.


> ldapmap source code. It adds a new smfi_setsymval function to
> libmilter, which allows a milter to set a macro in the Sendmail
> session.

milter-rrres patch includes this funcitonality also, controllable by
the sendmail.

Sendmail8.14 also appears to be including this feature (even though
IMHO it is far to trusting to allow milters to configure "random" macro
values in certain proprietary outsourcing models)


> For an explanation of why Sendmail's map algorithm is so terrible, and
> why mine is several orders of magnitude more efficient:

As you mention, the real problem is that the algorithm doesnt scale
well to LDAP where queries are individually more expensive than queries
with in process memory berkely db.

>
> Sendmail's map lookup is designed to work with embedded databases,
> using simple key/value pair lookups. This approach by its very nature
> embodies a flat namespace.

Re: Sendmail LDAP access milter

jmaimon@ttec.com wrote:
> hyc wrote:
> > I've just uploaded an updated version of my LDAP milter to
> > www.symas.com, available for free download.
>
> I have added your milter to the non-open source section of this list
> kept here
>
> http://www.jmaimon.com/sendmail/milters
>
> Let me know if I got it right.

It is in fact Free Software, licensed under GPL and distributed in
source form. It doesn't get more Open than that.

Thanks for maintaining your list, that's a nice service you provide.

> > ldapmap source code. It adds a new smfi_setsymval function to
> > libmilter, which allows a milter to set a macro in the Sendmail
> > session.
>
> milter-rrres patch includes this funcitonality also, controllable by
> the sendmail.

I discovered that after I posted... I didn't look at your patch, but
from your release notes it sounds like it allows a milter to invoke
rulesets and it allows a milter to be invoked by rulesets. That would
probably be a better approach for my milter, instead of having to
introduce new macros to trigger existing functionality. But your patch
also sounds pretty complicated, so I'll leave things as-is for now.

> Sendmail8.14 also appears to be including this feature (even though
> IMHO it is far to trusting to allow milters to configure "random" macro
> values in certain proprietary outsourcing models)

Yes, in a closed-source environment that would make me worry...

> > For an explanation of why Sendmail's map algorithm is so terrible, and
> > why mine is several orders of magnitude more efficient:

> As you mention, the real problem is that the algorithm doesnt scale
> well to LDAP where queries are individually more expensive than queries
> with in process memory berkely db.

Yes. Personally I'm amazed that anyone would release code that
obviously behaves so poorly. Hopefully my posting will help educate
people about how to properly use LDAP in their own applications, and
illustrate how bad things can get when you take a naive approach.

> > Sendmail's map lookup is designed to work with embedded databases,
> > using simple key/value pair lookups. This approach by its very nature
> > embodies a flat namespace.

--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

Re: Sendmail LDAP access milter

jmaimon@ttec.com wrote:
> hyc wrote:

> > ldapmap source code. It adds a new smfi_setsymval function to
> > libmilter, which allows a milter to set a macro in the Sendmail
> > session.

> Sendmail8.14 also appears to be including this feature (even though
> IMHO it is far to trusting to allow milters to configure "random" macro
> values in certain proprietary outsourcing models)

Hm, there's no such feature in the beta that I just downloaded. The
closest I can see is the smfi_setsymlist function, which just lets the
milter tell Sendmail which macros to make available. That seems like a
nice feature though, because you don't have to keep track of which
macros are relevant in your sendmail.mc file.

Re: Sendmail LDAP access milter

In article <1169154820.433992.157920@m58g2000cwm.googlegroups.com> "hyc"
writes:
>
>jmaimon@ttec.com wrote:
>
>> As you mention, the real problem is that the algorithm doesnt scale
>> well to LDAP where queries are individually more expensive than queries
>> with in process memory berkely db.
>
>Yes. Personally I'm amazed that anyone would release code that
>obviously behaves so poorly. Hopefully my posting will help educate
>people about how to properly use LDAP in their own applications, and
>illustrate how bad things can get when you take a naive approach.

As far as sendmail is concerned, it's not a naive approach but pretty
much inherent in the architecture, where LDAP is a comparatively late
arrival. The fundamental principle in sendmail (not alwayas adhered to
these days, but mostly) is that sendmail.cf, which is strictly speaking
a user-written program, defines the logic. Thus (as I'm sure you know)
the cumbersome algorithms you describe are actually implemented as
rewrite rules in sendmail.cf.

Furhermore the choice of map types is secondary to the logic of
sendmail.cf - you should be able to substitute any map type for any
function just by changing the K line (these days normally via .mc, of
course) - the rules don't (and shouldn't) need to change based on the
map type, in fact they are ignorant about the characteristics of
different map types. So, this all, unfortunately for LDAP, leads to a
"least common denominator" behaviour of simple key-value lookups, rather
than the more complex searches that LDAP could support.

Of course, since sendmail.cf can be modified by the user, it would in
principle be possible to deviate from the above by running different
rules depending on the map type. It would require some source changes
though, a) the ability for rules to find out the map type, and b) the
ability to specify searches in the rules, and not only key-value lookups
- the latter is already possible to some extent for LDAP, given that you
can specify a search filter and that it is possibe for it to return
multiple values, but I'm not sure if that's enough for the case of the
"hiearchical" lookups.

Bottom line, fitting more efficient use of LDAP into the sendmail
achitecture would require a fair amount of work, and given the small
demand for this and the limited set of people doing serious work on
sendmail development, it's not likely to happen. But this is where
milters fit the bill perfectly - "third parties" can, outside the
sendmail architecture, implement logic that can't easily be fitted into
the sendmail architecture (and/or is unlikely to be, for whatever
reason).

--Per Hedeland
per@hedeland.org

Re: Sendmail LDAP access milter

hyc wrote:
> jmaimon@ttec.com wrote:
> > hyc wrote:
>
> > > ldapmap source code. It adds a new smfi_setsymval function to
> > > libmilter, which allows a milter to set a macro in the Sendmail
> > > session.
>
> > Sendmail8.14 also appears to be including this feature (even though
> > IMHO it is far to trusting to allow milters to configure "random" macro
> > values in certain proprietary outsourcing models)
>
> Hm, there's no such feature in the beta that I just downloaded. The
> closest I can see is the smfi_setsymlist function, which just lets the
> milter tell Sendmail which macros to make available. That seems like a
> nice feature though, because you don't have to keep track of which
> macros are relevant in your sendmail.mc file.

Post in haste, repent at leisure. Whoops, your right. However, my
comment still applies....

Re: Sendmail LDAP access milter

hyc wrote:
> jmaimon@ttec.com wrote:
> > hyc wrote:
> > > I've just uploaded an updated version of my LDAP milter to
> > > www.symas.com, available for free download.
> >
> > I have added your milter to the non-open source section of this list
> > kept here
> >
> > http://www.jmaimon.com/sendmail/milters
> >
> > Let me know if I got it right.
>
> It is in fact Free Software, licensed under GPL and distributed in
> source form. It doesn't get more Open than that.

My apologies.

I followed the link you posted and all I found was a download area
protected with a fairly restrictive non-open source EULA

The open source list of milters are of milters I either know or have
strong reason to believe are open source.

Everything else is non-open source.

> Thanks for maintaining your list, that's a nice service you provide.

Nothing compared to the good folks who donate code.

>
> > > ldapmap source code. It adds a new smfi_setsymval function to
> > > libmilter, which allows a milter to set a macro in the Sendmail
> > > session.
> >
> > milter-rrres patch includes this funcitonality also, controllable by
> > the sendmail.
>
> I discovered that after I posted... I didn't look at your patch, but
> from your release notes it sounds like it allows a milter to invoke
> rulesets and it allows a milter to be invoked by rulesets. That would
> probably be a better approach for my milter, instead of having to
> introduce new macros to trigger existing functionality. But your patch
> also sounds pretty complicated, so I'll leave things as-is for now.

It allows milter invocation and results to be controlled by rulesets.

Among the many and varied operations allowed on rulesets, macros,
classes, maps, sendmail -bt mode, you can directly set macro's values.

This is controllable inside sendmail with requiring explicit turning on
of features for milters and reviewable at runtime with yet more
rulesets.

Re: Sendmail LDAP access milter

Per Hedeland wrote:
> As far as sendmail is concerned, it's not a naive approach but pretty
> much inherent in the architecture, where LDAP is a comparatively late
> arrival. The fundamental principle in sendmail (not alwayas adhered to
> these days, but mostly) is that sendmail.cf, which is strictly speaking
> a user-written program, defines the logic. Thus (as I'm sure you know)
> the cumbersome algorithms you describe are actually implemented as
> rewrite rules in sendmail.cf.

Yes, and changing proto.m4 to do "the right thing" here is even more
daunting.

> Of course, since sendmail.cf can be modified by the user, it would in
> principle be possible to deviate from the above by running different
> rules depending on the map type. It would require some source changes
> though, a) the ability for rules to find out the map type, and b) the
> ability to specify searches in the rules, and not only key-value lookups
> - the latter is already possible to some extent for LDAP, given that you
> can specify a search filter and that it is possibe for it to return
> multiple values, but I'm not sure if that's enough for the case of the
> "hiearchical" lookups.

Since the main problem is the way the access map is used, it might be
sufficient to alter
the behavior of the FEATURE(`access_db'), if an optional "LDAP"
argument was supplied. In all
the other cases the map lookups tend to be direct, not
component-by-component.

Fixing Sendmail's LDAP map code to do proper hierarchical lookups would
be easy enough, though you'd need to change the schema too, and provide
a tool to convert from the existing flat layout to hierarchical. The
other side of it is that you can obtain all of the information needed
to process a message in one or two LDAP queries, but Sendmail asks for
it one item at a time, so you either need a mechanism to cache some
query results for the current session, or waste time repeating the same
queries.

> Bottom line, fitting more efficient use of LDAP into the sendmail
> achitecture would require a fair amount of work, and given the small
> demand for this and the limited set of people doing serious work on
> sendmail development, it's not likely to happen. But this is where
> milters fit the bill perfectly - "third parties" can, outside the
> sendmail architecture, implement logic that can't easily be fitted into
> the sendmail architecture (and/or is unlikely to be, for whatever
> reason).

Being able to administer all of your mail access and routing for a farm
of mail servers from a single LDAP repository is a pretty compelling
advantage. I suspect it hasn't gained much popularity thus far because
the prevalent implementation is so abysmal.

Perhaps keeping the functionality in a milter is the best way forward.

PS: I'm currently testing a patch to allow a milter to be invoked as a
map. This will provide a way to plug into new rulesets like
greet_pause and queue_group without needing to define new macros for
the purpose. Claus tells me it's too late to get this into 8.14, but it
may turn out to be a better way forward.

> --Per Hedeland
> per@hedeland.org

Re: Sendmail LDAP access milter

jmaimon@ttec.com wrote:
> hyc wrote:
> > It is in fact Free Software, licensed under GPL and distributed in
> > source form. It doesn't get more Open than that.
>
> My apologies.
>
> I followed the link you posted and all I found was a download area
> protected with a fairly restrictive non-open source EULA

Hm, I didn't think it was so restrictive when I wrote it. It just
disclaims liability, and requires you to agree to the terms of the GPL
before letting you thru.

> It allows milter invocation and results to be controlled by rulesets.
>
> Among the many and varied operations allowed on rulesets, macros,
> classes, maps, sendmail -bt mode, you can directly set macro's values.

I completely missed that when reading thru your patch. I'll have to
look again...

> This is controllable inside sendmail with requiring explicit turning on
> of features for milters and reviewable at runtime with yet more
> rulesets.

It sounds great, but again it sounds very complex, and your disclaimer
about not using it in production kinda makes it a moot point.
-- Howard

Re: Sendmail LDAP access milter

In article <1169185215.699511.283210@s34g2000cwa.googlegroups.com> "hyc"
writes:
>
>Per Hedeland wrote:
>> Of course, since sendmail.cf can be modified by the user, it would in
>> principle be possible to deviate from the above by running different
>> rules depending on the map type. It would require some source changes
>> though, a) the ability for rules to find out the map type, and b) the
>> ability to specify searches in the rules, and not only key-value lookups
>> - the latter is already possible to some extent for LDAP, given that you
>> can specify a search filter and that it is possibe for it to return
>> multiple values, but I'm not sure if that's enough for the case of the
>> "hiearchical" lookups.
>
>Since the main problem is the way the access map is used, it might be
>sufficient to alter
>the behavior of the FEATURE(`access_db'), if an optional "LDAP"
>argument was supplied. In all
>the other cases the map lookups tend to be direct, not
>component-by-component.

Well, access_db may the "worst" case and the only one where domain names
and IP addresses are split up into their components, but many/most of
the maps have various aother forms of "wildcarding" that requires
multiple key-value lookups to implement, e.g. in virtusertable look for
user@domain, user+@domain, @domain, etc.

>Fixing Sendmail's LDAP map code to do proper hierarchical lookups would
>be easy enough, though you'd need to change the schema too, and provide
>a tool to convert from the existing flat layout to hierarchical.

I feel that you're still missing my main point - in principle
"sendmail's LDAP code" doesn't do any particular lookups or implement
some particular schema, it just does what sendmail.cf tells it to do,
via the K line and the rules. There is recently some "builtin" support
for certain cases, but those are the cases where the binary has
"hardwired" knowledge anyway (macros, classes, aliases) - in principle
sendmail the binary has no idea what access_db is or what it is used for
(though ISTR that this principle has been marginally "violated" too
recently).

> The
>other side of it is that you can obtain all of the information needed
>to process a message in one or two LDAP queries, but Sendmail asks for
>it one item at a time, so you either need a mechanism to cache some
>query results for the current session, or waste time repeating the same
>queries.

Likewise - this would require either much more functionality in
sendmail.cf, or a binary that "knows" (or guesses) what the rules are up
to. The latter would be a definite break-away from the fundamental
architecture.

>Being able to administer all of your mail access and routing for a farm
>of mail servers from a single LDAP repository is a pretty compelling
>advantage. I suspect it hasn't gained much popularity thus far because
>the prevalent implementation is so abysmal.

Based on discussions here (which may not be representative), the problem
is rather that the potential users have no clue about what LDAP is and
how it could be used. Or possibly that there is no significant problem,
and there is just very little interest in using LDAP for anything other
than user verification and/or bare-bones mail routing a la ldap_routing.

--Per Hedeland
per@hedeland.org

Re: Sendmail LDAP access milter

> Being able to administer all of your mail access and routing for a farm
> of mail servers from a single LDAP repository is a pretty compelling
> advantage. I suspect it hasn't gained much popularity thus far because
> the prevalent implementation is so abysmal.

Actually, I would like to see all information in LDAP, too.
I tried to convince somebody some time ago on this list, that to have
..forward information in LDAP is a GoodThing(tm), however, it was
annihilated. The con was that sendmail would rely on the ownership
information of the .forward file, my pro was that sendmail already has
the local user (and its uid/gid) in order to find the home directory,
hence, it can use that information.

Re: Sendmail LDAP access milter

Per Hedeland wrote:
> Likewise - this would require either much more functionality in
> sendmail.cf, or a binary that "knows" (or guesses) what the rules are up
> to. The latter would be a definite break-away from the fundamental
> architecture.

Actually it would require *less* in sendmail.cf. With my milter map
patch, the rulesets get a lot simpler.

The old rule for greet_pause:

SLocal_greet_pause
Sgreet_pause
R$* $: <$1> $| $>"Local_greet_pause" $1
R<$*> $| $#$* $#$2
R<$*> $| $* $: $1
R$+ $| $+ $: $>D < $1 > < $2 >
R $| $+ $: $>A < $1 > <> empty client_name
R <$+> $: $>A < $1 > <> no: another lookup
R <$*> $@
R<$* > <$*> $@
R<$+> <$*> $# $1

The rule for my milter:

Sgreet_pause
R$* $# $(ldapmap GreetPause $: 0 $)

The only tedium remaining is the time it takes to rewrite all of the
relevant rulesets.

Re: Sendmail LDAP access milter

hyc wrote:
> Per Hedeland wrote:
> > Likewise - this would require either much more functionality in
> > sendmail.cf, or a binary that "knows" (or guesses) what the rules are up
> > to. The latter would be a definite break-away from the fundamental
> > architecture.
>
> Actually it would require *less* in sendmail.cf. With my milter map
> patch, the rulesets get a lot simpler.
>
> The old rule for greet_pause:
>
> SLocal_greet_pause
> Sgreet_pause
> R$* $: <$1> $| $>"Local_greet_pause" $1
> R<$*> $| $#$* $#$2
> R<$*> $| $* $: $1
> R$+ $| $+ $: $>D < $1 > < $2 >
> R $| $+ $: $>A < $1 > <> empty client_name
> R <$+> $: $>A < $1 > <> no: another lookup
> R <$*> $@
> R<$* > <$*> $@
> R<$+> <$*> $# $1
>
> The rule for my milter:
>
> Sgreet_pause
> R$* $# $(ldapmap GreetPause $: 0 $)
>

This is a not a good example

In the above ruleset, there is only one real obvious lookup that will
normally get performed. The rest enables local hooks (which you should
have used instead of rewriting the entire thing) and further processing

The "redundant" lookups occur in the the lookup routines. A D

> The only tedium remaining is the time it takes to rewrite all of the
> relevant rulesets.

Re: Sendmail LDAP access milter

Per Hedeland wrote:
> In article <1169185215.699511.283210@s34g2000cwa.googlegroups.com> "hyc"
> writes:
> >
>
> Well, access_db may the "worst" case and the only one where domain names
> and IP addresses are split up into their components, but many/most of
> the maps have various aother forms of "wildcarding" that requires
> multiple key-value lookups to implement, e.g. in virtusertable look for
> user@domain, user+@domain, @domain, etc.
>
> >Fixing Sendmail's LDAP map code to do proper hierarchical lookups would
> >be easy enough, though you'd need to change the schema too, and provide
> >a tool to convert from the existing flat layout to hierarchical.
>
> I feel that you're still missing my main point - in principle
> "sendmail's LDAP code" doesn't do any particular lookups or implement
> some particular schema, it just does what sendmail.cf tells it to do,
> via the K line and the rules. There is recently some "builtin" support
> for certain cases, but those are the cases where the binary has
> "hardwired" knowledge anyway (macros, classes, aliases) - in principle
> sendmail the binary has no idea what access_db is or what it is used for
> (though ISTR that this principle has been marginally "violated" too
> recently).

It sounds like it would prove usefull for ldap maps to have the ability
to "predict" query patterns and cache accordingly. Win/Win.

Patches anyone?

Re: Sendmail LDAP access milter

jmaimon@ttec.com wrote:
> This is a not a good example
>
> In the above ruleset, there is only one real obvious lookup that will
> normally get performed. The rest enables local hooks (which you should
> have used instead of rewriting the entire thing) and further processing

You're missing my point - using my milter means that you don't enable
FEATURE(`access_db') at all, nor do you use any of the FEATUREs that
depend on it. In which case the original greet_pause ruleset doesn't
even get generated. The resulting sendmail.cf is significantly simpler.

The one thing that bothers me with this approach is that there's still
the overhead of the MTA asking the milter for information one item at a
time. In my previous approach I could have just had the milter send a
batch of macros and values to the MTA. So when there was any data to
send, it would just take one IPC exchange, and if there was no relevant
data, there'd be no extra IPC at all. With this approach, it still
takes one invocation per item, even if there is no item to return.

Hmmmmmmm. Perhaps that's the answer - the milter should send everything
it knows in one batch, which the map_lookup routine caches. That would
work...

> The "redundant" lookups occur in the the lookup routines. A D

> > The only tedium remaining is the time it takes to rewrite all of the
> > relevant rulesets.

Re: Sendmail LDAP access milter

hyc wrote:

> Hmmmmmmm. Perhaps that's the answer - the milter should send everything
> it knows in one batch, which the map_lookup routine caches. That would
> work...


Couldnt this also be done as a patch to ldap map (or maybe a new map
type of ldapcache) ?

Re: Sendmail LDAP access milter

jmaimon@ttec.com wrote:
> hyc wrote:
> > Hmmmmmmm. Perhaps that's the answer - the milter should send everything
> > it knows in one batch, which the map_lookup routine caches. That would
> > work...

> Couldnt this also be done as a patch to ldap map (or maybe a new map
> type of ldapcache) ?

Yes, it could. I hadn't really considered that route before because
just using the milter approach worked well enough and didn't require
coordination with sendmail.org. But now that we're talking about
patches to the core milter engine I guess that's no longer true.

That leaves open the question about whether it's still useful to extend
the milter API as I have. It now feels to me like adding the milter
mapclass was unnecessary, because you could just as easily have the
milter set a batch of macros that are then expanded in each of the
relevant rulesets. Probably I should have just focused on my
smfi_setsymval patch instead.

Re: Sendmail LDAP access milter

In article <1169197967.829548.60690@l53g2000cwa.googlegroups.com> "hyc"
writes:
>
>Per Hedeland wrote:
>> Likewise - this would require either much more functionality in
>> sendmail.cf, or a binary that "knows" (or guesses) what the rules are up
>> to. The latter would be a definite break-away from the fundamental
>> architecture.
>
>Actually it would require *less* in sendmail.cf. With my milter map
>patch, the rulesets get a lot simpler.

Well, I'm sorry, but I can only say that your harsh criticism of
sendmail can be simply dismissed, since you don't understand the
philosophy and architecture of the program at all, nor care to be
enlightened when it is explained to you at length.

Of course all of FEATURE(access_db) and lots of other features could
have been implemented in the binary itself, with in many/most cases
significantly better performance. But that's not The sendmail Way(tm) -
the binary provides mechanisms, the rules decide logic and policy, and
maps are there to support *the rules*.

Frankly I'm not sure why you're interested in sendmail at all, since as
far as I know all other significant MTAs are just like you would like
sendmail to be - a hardwired logic with a set of hardwired policies,
where you get to choose between them and set their parameters in a
config file that is just that - a config file, as opposed to the
infinitely flexible logic engine that sendmail.cf is.

--Per Hedeland
per@hedeland.org

Sendmail-8 philosophy [Was: Sendmail LDAP access milter]

per@hedeland.org (Per Hedeland) writes:
> [...]
> Of course all of FEATURE(access_db) and lots of other features could
> have been implemented in the binary itself, with in many/most cases
> significantly better performance. But that's not The sendmail Way(tm) -
> the binary provides mechanisms, the rules decide logic and policy, and
> maps are there to support *the rules*.

You are mostly right ;-)

Even the above philosophy could be implemented in a way giving "smart"
maps *a chance* to perform better that "for any map" procedures
implemented in sendmail.cf allow.
e.g.
* virtusertable may require 7 lookups for processing a plussed address
* smart socket map daemon (e.g.) in perl may "predict" reply to
remaining 6 queries when replying the first query
* it may "cheat" and give "predicted match" to a later virtusertable
query but there is no *official* way (for "any"table) to return
"no match" for all queries in first reply.


> [...]

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/

Re: Sendmail LDAP access milter

Per Hedeland wrote:
> In article <1169197967.829548.60690@l53g2000cwa.googlegroups.com> "hyc"
> writes:
> >
> >Per Hedeland wrote:
> >> Likewise - this would require either much more functionality in
> >> sendmail.cf, or a binary that "knows" (or guesses) what the rules are up
> >> to. The latter would be a definite break-away from the fundamental
> >> architecture.

> >Actually it would require *less* in sendmail.cf. With my milter map
> >patch, the rulesets get a lot simpler.

> Well, I'm sorry, but I can only say that your harsh criticism of
> sendmail can be simply dismissed, since you don't understand the
> philosophy and architecture of the program at all, nor care to be
> enlightened when it is explained to you at length.

Don't confuse disagreement for misunderstanding. I understand
perfectly. I first started using sendmail more than 20 years ago, as a
postmaster @ umich.edu. I've been writing rewrite rulesets for a long
time.

You seem content to say "LDAP arrived late on the scene and the fact
that it's not supported well by the current model is just too bad." In
fact it could easily be fixed at the core. I never attacked the problem
from that angle before because I have enough projects on my plate
already and didn't want to get deeply involved in this code base. Using
an external plugin was a good solution given the time I had available.

> Of course all of FEATURE(access_db) and lots of other features could
> have been implemented in the binary itself, with in many/most cases
> significantly better performance. But that's not The sendmail Way(tm) -
> the binary provides mechanisms, the rules decide logic and policy, and
> maps are there to support *the rules*.

I'm not asking for all of the features to be implemented in the
sendmail binary. The binary provides hooks for external programs to be
used to make decisions. I'm looking for one more hook to allow those
external programs one more degree of influence.

Configurability is a two-way street. It is just as valid to write
simple rulesets that delegate all decisions to an external program, as
it is to write complex rulesets that just get bits of help from
external programs. There's nothing sacred about the pre-packaged
rulesets or the way they do things.
-- Howard

SOCKETMAP (was Re: Sendmail LDAP access milter)

hyc wrote:

> PS: I'm currently testing a patch to allow a milter to be invoked as a
> map.

There's no need; Sendmail already has the SOCKETMAP map type that can
talk to a daemon of your choice.

You still need to modify the .cf file, though, because the default
..cf file treats SOCKETMAP as a dumb look-up-a-key map just like all
the other map types.

--
David.

Re: SOCKETMAP (was Re: Sendmail LDAP access milter)

David F. Skoll wrote:
> hyc wrote:
>
>> PS: I'm currently testing a patch to allow a milter to be invoked as a
>> map.
>
> There's no need; Sendmail already has the SOCKETMAP map type that can
> talk to a daemon of your choice.
>
> You still need to modify the .cf file, though, because the default
> .cf file treats SOCKETMAP as a dumb look-up-a-key map just like all
> the other map types.

The problem with that is that it requires a separate daemon, you can't
multiplex its communication on top of the existing milter framework. I
suppose you could spawn your own thread to handle that traffic, but it
doesn't seem like a particularly clean approach. In particular the separate
thread wouldn't have access to the milter context.

--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

Re: SOCKETMAP

"David F. Skoll" writes:

> hyc wrote:
>
>> PS: I'm currently testing a patch to allow a milter to be invoked as a
>> map.
>
> There's no need; Sendmail already has the SOCKETMAP map ty pe that can
> talk to a daemon of your choice.

It would be nice to create variant of "socket map" capable to work over
connection-less sockets. I would help to use resources strictly on
"as needed and exactly when needed" basis.

> You still need to modify the .cf file, though, because the default
> .cf file treats SOCKETMAP as a dumb look-up-a-key map just like all
> the other map types.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/

Re: SOCKETMAP (was Re: Sendmail LDAP access milter)

Howard Chu wrote:

> The problem with that is that it requires a separate daemon, you can't
> multiplex its communication on top of the existing milter framework.

So how is that a problem?

> In particular the separate thread wouldn't have access to the milter
> context.

There is not necessarily any milter context for a map lookup anyway.
For example, when you're sending mail from the queue and Sendmail does a
mailertable lookup, there is no milter context.

Regards,

David.

Re: SOCKETMAP (was Re: Sendmail LDAP access milter)

David F. Skoll wrote:
> Howard Chu wrote:
>
> > The problem with that is that it requires a separate daemon, you can't
> > multiplex its communication on top of the existing milter framework.
>
> So how is that a problem?
>
> > In particular the separate thread wouldn't have access to the milter
> > context.
>
> There is not necessarily any milter context for a map lookup anyway.
> For example, when you're sending mail from the queue and Sendmail does a
> mailertable lookup, there is no milter context.

Good point. I've only been looking at lookups used to decide whether to
accept a connection or a given email; once it's accepted it was no
longer my concern. Perhaps I was looking too narrowly. Still, that's
the only situation that presents an actual resource/efficiency problem,
so it made sense to focus solely on it.

Re: SOCKETMAP (was Re: Sendmail LDAP access milter)

hyc wrote:

[David Skoll]
>> There is not necessarily any milter context for a map lookup anyway.
>> For example, when you're sending mail from the queue and Sendmail does a
>> mailertable lookup, there is no milter context.

[hyc]
> Good point. I've only been looking at lookups used to decide whether to
> accept a connection or a given email; once it's accepted it was no
> longer my concern. Perhaps I was looking too narrowly. Still, that's
> the only situation that presents an actual resource/efficiency problem,
> so it made sense to focus solely on it.

I don't think that's right. I can definitely see a use for LDAP routing.
The LDAP map would be consulted not only during an SMTP session, but any
time Sendmail wants to route mail. This means no milter context in some
cases.

Depending on what you're trying to do, it might make sense to put
accept/reject policy in a milter, but I don't see how with Sendmail's
current architecture it is possible to put routing decisions in a milter.

Regards,

David.

Re: SOCKETMAP (was Re: Sendmail LDAP access milter)

David F. Skoll wrote:
> [hyc]
> > Good point.

> I don't think that's right. I can definitely see a use for LDAP routing.
> The LDAP map would be consulted not only during an SMTP session, but any
> time Sendmail wants to route mail. This means no milter context in some
> cases.

You made your point already.

> Depending on what you're trying to do, it might make sense to put
> accept/reject policy in a milter, but I don't see how with Sendmail's
> current architecture it is possible to put routing decisions in a milter.

My ongoing conversation on the sendmail support mailing list has
refocused on fixing the LDAP map, instead of the milter. However, back
to my point - sure, LDAP routing is useful. What I said was that it is
not a resource/efficiency problem, in the same way that access control
is. My sendmail server gets a few thousand connection attempts per day
(it's not exactly a popular email domain). 96% of those are attempts to
deliver spam, which my milter rejects. As such, the demand on the LDAP
server for routing is inconsequential, down in the noise, compared to
the demand for accept/reject decisions.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

Re: SOCKETMAP

On Jan 20, 5:17 am, Andrzej Adam Filip wrote:
> It would be nice to create variant of "socket map" capable to work over
> connection-less sockets. I would help to use resources strictly on
> "as needed and exactly when needed" basis.

That's only feasible if you know your lookups will never exceed the
size of a datagram. For a large alias lookup, that might be a problem.

With Unix domain sockets I don't think there's enough overhead to make
it worth worrying about. On my 2GHz dual core AMD system (X2 3800+) I
can run 30-40,000 queries per second, one query per connection,
between a local client and slapd over ldapi://. In fact local TCP
isn't terrible either, but you run out of ports and hit the 2MSL limit
in only a few seconds. (Recall that every local connection uses up 2
ports, and there's only 65535 of them on a given interface. You have
to let the TIME_WAITs expire before you can reuse them.)
-- Howard

Re: SOCKETMAP

"hyc" writes:

> On Jan 20, 5:17 am, Andrzej Adam Filip wrote:
>> It would be nice to create variant of "socket map" capable to work over
>> connection-less sockets. I would help to use resources strictly on
>> "as needed and exactly when needed" basis.
>
> That's only feasible if you know your lookups will never exceed the
> size of a datagram. For a large alias lookup, that might be a problem.

500 bytes for *one* reply is sufficient for most situations
except big aliases.

> With Unix domain sockets I don't think there's enough overhead to make
> it worth worrying about. On my 2GHz dual core AMD system (X2 3800+) I
> can run 30-40,000 queries per second, one query per connection,
> between a local client and slapd over ldapi://. In fact local TCP
> isn't terrible either, but you run out of ports and hit the 2MSL limit
> in only a few seconds. (Recall that every local connection uses up 2
> ports, and there's only 65535 of them on a given interface. You have
> to let the TIME_WAITs expire before you can reuse them.)

With UDP (connection less) there would no one TCP session per every SMTP
session accepted by sendmail requiring monitoring by full SMTP session
lifetime. It works strictly on base what is needed only when it is needed.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

On Jan 28, 2:11 am, Andrzej Adam Filip wrote:
> > That's only feasible if you know your lookups will never exceed the
> > size of a datagram. For a large alias lookup, that might be a problem.

> 500 bytes for *one* reply is sufficient for most situations
> except big aliases.

Yes, agreed. It would be a simple change to the map code, and yes, in
general it would be low overhead.

This diff against the 8.14beta4 will do the trick. Just add a "-d"
option to your map definition to get datagram mode.

The diff could actually be made a little smaller; there's no actual
need for the stream and dgram code paths to be different.
============================================================ =======
RCS file: RCS/conf.c,v
retrieving revision 1.1
diff -wu -r1.1 conf.c
--- conf.c 2007/01/28 10:48:04 1.1
+++ conf.c 2007/01/28 11:43:58
@@ -662,7 +662,7 @@
#if SOCKETMAP
/* arbitrary daemons */
MAPDEF("socket", NULL, MCF_ALIASOK,
- map_parseargs, socket_map_open, socket_map_close,
+ socket_map_init, socket_map_open, socket_map_close,
socket_map_lookup, null_map_store);
#endif /* SOCKETMAP */

============================================================ =======
RCS file: RCS/map.c,v
retrieving revision 1.1
diff -wu -r1.1 map.c
--- map.c 2007/01/28 10:32:07 1.1
+++ map.c 2007/01/28 11:43:10
@@ -19,6 +19,10 @@
# include
#endif /* LDAPMAP */

+#if SOCKETMAP
+# include
+#endif /* SOCKETMAP */
+
#if NDBM
# include
# ifdef R_FIRST
@@ -7330,6 +7334,29 @@

# define socket_map_next map_stack[0]

+
+/*
+** SOCKET_MAP_INIT -- parse flags
+*/
+bool
+socket_map_init(map, args)
+ MAP *map;
+ char *args;
+{
+ register *p;
+
+ /* There really ought to be an easier way to check for custom
+ * flags in addition to the default flags...
+ */
+ p = strstr(args, "-d");
+ if (p)
+ {
+ strcpy(p, p+2);
+ map->map_db2 = p;
+ }
+ return map_parseargs(map, args);
+}
+
/*
** SOCKET_MAP_OPEN -- open socket table
*/
@@ -7647,7 +7674,8 @@
/* nope, actually connecting */
for (;;)
{
- sock = socket(addr.sa.sa_family, SOCK_STREAM, 0);
+ sock = socket(addr.sa.sa_family,
+ map->map_db2 ? SOCK_DGRAM : SOCK_STREAM, 0);
if (sock < 0)
{
save_errno = errno;
@@ -7837,6 +7865,8 @@
char *replybuf, *rval, *value, *status, *key;
SM_FILE_T *f;
char keybuf[MAXNAME + 1];
+ char lenbuf[11]; /* SOCKETMAP_MAXL */
+ int lenlen;

replybuf = NULL;
rval = NULL;
@@ -7872,6 +7902,32 @@
goto errcl;
}

+ /* datagram sockets need special treatment */
+ if (map->map_db2)
+ {
+ lenlen = recv(f->f_file, lenbuf, sizeof(lenbuf),
MSG_PEEK);
+ if (lenlen < 0)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
failed to read length parameter of reply",
+ map->map_mname);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ if (sm_io_sscanf(lenbuf, SM_TIME_DEFAULT, "%9u:",
&replylen) != 1)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
failed to read length parameter of reply",
+ map->map_mname);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ if (replylen > SOCKETMAP_MAXL)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s): reply
too long: %u",
+ map->map_mname, replylen);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ } else {
if (sm_io_fscanf(f, SM_TIME_DEFAULT, "%9u", &replylen) != 1)
{
syserr("451 4.3.0 socket_map_lookup(%s): failed to
read length parameter of reply",
@@ -7893,6 +7949,7 @@
*statp = EX_TEMPFAIL;
goto error;
}
+ }

replybuf = (char *) sm_malloc(replylen + 1);
if (replybuf == NULL)
@@ -7903,15 +7960,29 @@
goto error;
}

- recvlen = sm_io_read(f, SM_TIME_DEFAULT, replybuf, replylen);
- if (recvlen < replylen)
+ if (map->map_db2)
+ {
+ struct iovec vec[2] = {{lenbuf, lenlen}, {replybuf,
replylen+1}};
+ recvlen = readv(f->f_file, vec, 2);
+ if (recvlen < lenlen + replylen + 1)
{
syserr("451 4.3.0 socket_map_lookup(%s): received only
%u of %u reply characters",
map->map_mname, recvlen, replylen);
*statp = EX_TEMPFAIL;
goto errcl;
}
- if (sm_io_getc(f, SM_TIME_DEFAULT) != ',')
+ } else
+ {
+ recvlen = sm_io_read(f, SM_TIME_DEFAULT, replybuf,
replylen+1);
+ if (recvlen < replylen+1)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
received only %u of %u reply characters",
+ map->map_mname, recvlen, replylen);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ }
+ if (replybuf[replylen] != ',')
{
syserr("451 4.3.0 socket_map_lookup(%s): missing ','
in reply",
map->map_mname);

Re: SOCKETMAP

On Jan 28, 2:11 am, Andrzej Adam Filip wrote:
> "hyc" writes:
> > That's only feasible if you know your lookups will never exceed the
> > size of a datagram. For a large alias lookup, that might be a problem.

> 500 bytes for *one* reply is sufficient for most situations
> except big aliases.

True. This patch against 8.14beta4 will do the trick. (Sorry for the
typos earlier.)
Add a "-d" option to your map definition to use datagram mode.
============================================================ =======
RCS file: RCS/conf.c,v
retrieving revision 1.1
diff -u -r1.1 conf.c
--- conf.c 2007/01/28 10:48:04 1.1
+++ conf.c 2007/01/28 11:43:58
@@ -662,7 +662,7 @@
#if SOCKETMAP
/* arbitrary daemons */
MAPDEF("socket", NULL, MCF_ALIASOK,
- map_parseargs, socket_map_open, socket_map_close,
+ socket_map_init, socket_map_open, socket_map_close,
socket_map_lookup, null_map_store);
#endif /* SOCKETMAP */

============================================================ =======
RCS file: RCS/map.c,v
retrieving revision 1.1
diff -wu -r1.1 map.c
--- map.c 2007/01/28 10:32:07 1.1
+++ map.c 2007/01/28 11:58:03
@@ -19,6 +19,10 @@
# include
#endif /* LDAPMAP */

+#if SOCKETMAP
+# include
+#endif /* SOCKETMAP */
+
#if NDBM
# include
# ifdef R_FIRST
@@ -7330,6 +7334,29 @@

# define socket_map_next map_stack[0]

+
+/*
+** SOCKET_MAP_INIT -- parse flags
+*/
+bool
+socket_map_init(map, args)
+ MAP *map;
+ char *args;
+{
+ register char *p;
+
+ /* There really ought to be an easier way to check for custom
+ * flags in addition to the default flags...
+ */
+ p = strstr(args, "-d");
+ if (p)
+ {
+ strcpy(p, p+2);
+ map->map_db2 = p;
+ }
+ return map_parseargs(map, args);
+}
+
/*
** SOCKET_MAP_OPEN -- open socket table
*/
@@ -7647,7 +7674,8 @@
/* nope, actually connecting */
for (;;)
{
- sock = socket(addr.sa.sa_family, SOCK_STREAM, 0);
+ sock = socket(addr.sa.sa_family,
+ map->map_db2 ? SOCK_DGRAM : SOCK_STREAM, 0);
if (sock < 0)
{
save_errno = errno;
@@ -7837,6 +7865,8 @@
char *replybuf, *rval, *value, *status, *key;
SM_FILE_T *f;
char keybuf[MAXNAME + 1];
+ char lenbuf[11]; /* SOCKETMAP_MAXL */
+ int lenlen;

replybuf = NULL;
rval = NULL;
@@ -7872,6 +7902,32 @@
goto errcl;
}

+ /* datagram sockets need special treatment */
+ if (map->map_db2)
+ {
+ lenlen = recv(f->f_file, lenbuf, sizeof(lenbuf),
MSG_PEEK);
+ if (lenlen < 0)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
failed to read length parameter of reply",
+ map->map_mname);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ if (sm_io_sscanf(lenbuf, "%9u:", &replylen) != 1)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
failed to read length parameter of reply",
+ map->map_mname);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ if (replylen > SOCKETMAP_MAXL)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s): reply
too long: %u",
+ map->map_mname, replylen);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ } else {
if (sm_io_fscanf(f, SM_TIME_DEFAULT, "%9u", &replylen) != 1)
{
syserr("451 4.3.0 socket_map_lookup(%s): failed to
read length parameter of reply",
@@ -7893,6 +7949,7 @@
*statp = EX_TEMPFAIL;
goto error;
}
+ }

replybuf = (char *) sm_malloc(replylen + 1);
if (replybuf == NULL)
@@ -7903,15 +7960,29 @@
goto error;
}

- recvlen = sm_io_read(f, SM_TIME_DEFAULT, replybuf, replylen);
- if (recvlen < replylen)
+ if (map->map_db2)
+ {
+ struct iovec vec[2] = {{lenbuf, lenlen}, {replybuf,
replylen+1}};
+ recvlen = readv(f->f_file, vec, 2);
+ if (recvlen < lenlen + replylen + 1)
{
syserr("451 4.3.0 socket_map_lookup(%s): received only
%u of %u reply characters",
map->map_mname, recvlen, replylen);
*statp = EX_TEMPFAIL;
goto errcl;
}
- if (sm_io_getc(f, SM_TIME_DEFAULT) != ',')
+ } else
+ {
+ recvlen = sm_io_read(f, SM_TIME_DEFAULT, replybuf,
replylen+1);
+ if (recvlen < replylen+1)
+ {
+ syserr("451 4.3.0 socket_map_lookup(%s):
received only %u of %u reply characters",
+ map->map_mname, recvlen, replylen);
+ *statp = EX_TEMPFAIL;
+ goto errcl;
+ }
+ }
+ if (replybuf[replylen] != ',')
{
syserr("451 4.3.0 socket_map_lookup(%s): missing ','
in reply",
map->map_mname);

Re: SOCKETMAP

"hyc" writes:

> On Jan 28, 2:11 am, Andrzej Adam Filip wrote:
>> > That's only feasible if you know your lookups will never exceed the
>> > size of a datagram. For a large alias lookup, that might be a problem.
>
>> 500 bytes for *one* reply is sufficient for most situations
>> except big aliases.
>
> Yes, agreed. It would be a simple change to the map code, and yes, in
> general it would be low overhead.
>
> This diff against the 8.14beta4 will do the trick. Just add a "-d"
> option to your map definition to get datagram mode.
>
> The diff could actually be made a little smaller; there's no actual
> need for the stream and dgram code paths to be different.
> ============================================================ =======
> RCS file: RCS/conf.c,v
> retrieving revision 1.1
> diff -wu -r1.1 conf.c
> [...]

Have you submitted the patch to sendmail.org?

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

Andrzej Adam Filip wrote:
> "hyc" writes:

>> ============================================================ =======
>> RCS file: RCS/conf.c,v
>> retrieving revision 1.1
>> diff -wu -r1.1 conf.c
>> [...]
>
> Have you submitted the patch to sendmail.org?
>
There's not much point at the moment. Since 8.14 is in beta it's
feature-frozen, patches aren't being accepted right now.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

Re: SOCKETMAP

Howard Chu writes:

> Andrzej Adam Filip wrote:
>> "hyc" writes:
>
>>> ============================================================ =======
>>> RCS file: RCS/conf.c,v
>>> retrieving revision 1.1
>>> diff -wu -r1.1 conf.c
>>> [...]
>>
>> Have you submitted the patch to sendmail.org?
>
> There's not much point at the moment. Since 8.14 is in beta it's
> feature-frozen, patches aren't being accepted right now.

It has no chance to be included into officially blessed parts of
source code sendmail-8.14.0. It has some chance to be included as
FFR (For Future Release) code part - alpha/beta quality parts of
source code compiled only upon direct request during compilation.

I have seen a few (TWO+) very nice set of features never included into
sendmail.org distribution.
* "wide" patch (included selecting another mailer as fall-back e.g. uucp
as backup for smtp)
* check_local m4 files delivering extended "per user" anti-spam checks

I would like to avoid seeing another idea lost to sendmail's postmaster
community.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

Andrzej Adam Filip wrote:
> "hyc" writes:

> >> 500 bytes for *one* reply is sufficient for most situations
> >> except big aliases.

So it "may" work, but maybe it fails? Does the patch handle those
errors in a sensible way?

> Have you submitted the patch to sendmail.org?

Where is your performance analysis? Please post reproduceable data that
demonstrates a non-neglible performance increase for the MTA. Or is
there another good reason for this additional code?

Re: SOCKETMAP

On Jan 28, 10:09 am, Claus Aßmann please)@mine.informatik.uni-kiel.de> wrote:
> Andrzej Adam Filip wrote:
>
> > "hyc" writes:
> > >> 500 bytes for *one* reply is sufficient for most situations
> > >> except big aliases.
> So it "may" work, but maybe it fails? Does the patch handle those
> errors in a sensible way?

This error can't get handled on the receiving end. If a set of data is=20
too large, the sender will receive an error. It's up to the sender to=20
do something sane about it, e.g. try again with a smaller message,=20
most likely one indicating a failure. Given that the maximum size of a=20
datagram is 65507 bytes, it would be pretty rare for the sender to=20
fail but when it does there are very few choices for recovery.

It would be pointless to try any other type of recovery (such as=20
splitting the message into multiple datagrams) since then you would=20
need to do sequence numbers and other such features that TCP already=20
implements. At any rate, the patch doesn't address it because that's=20
not the receiver's responsibility.

> > Have you submitted the patch to sendmail.org?
> Where is your performance analysis? Please post reproduceable data that
> demonstrates a non-neglible performance increase for the MTA. Or is
> there another good reason for this additional code?

Re: SOCKETMAP

On Jan 28, 10:09 am, Claus Aßmann please)@mine.informatik.uni-kiel.de> wrote:
> Where is your performance analysis? Please post reproduceable data that
> demonstrates a non-neglible performance increase for the MTA. Or is
> there another good reason for this additional code?

Since Andrzej requested this I'll leave it to him to benchmark it,=20
because if he has a genuine interest in it, he'll do it. Anyone can=20
say "it would be nice if..." Let's see if he backs up his words with=20
action.
-- Howard

Re: SOCKETMAP

Claus Aßmann
writes:

> Andrzej Adam Filip wrote:
>> "hyc" writes:
>
>> >> 500 bytes for *one* reply is sufficient for most situations
>> >> except big aliases.
>
> So it "may" work, but maybe it fails? Does the patch handle those
> errors in a sensible way?
>
>> Have you submitted the patch to sendmail.org?
>
> Where is your performance analysis? Please post reproduceable data that
> demonstrates a non-neglible performance increase for the MTA. Or is
> there another good reason for this additional code?

I can agree that your objections/usefulness queries are perfectly OK for
*non FFR* code. I have not "inspected code" of this particular patch.

I can assure you that *writing* UDP (connection less) socket map
*server* would be *much simpler*, especially load balancing would be
*much easier* to implement. *For me* it is even more important than
scale of the overall performance gains.

UDP services in general are easier to write than TCP services but harder
to debug in case of under heavy load problems.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

On Jan 28, 4:03 am, "hyc" wrote:
> On Jan 28, 2:11 am, Andrzej Adam Filip wrote:
>
> > "hyc" writes:
> > > That's only feasible if you know your lookups will never exceed the
> > > size of a datagram. For a large alias lookup, that might be a problem.
> > 500 bytes for *one* reply is sufficient for most situations
> > except big aliases.True. This patch against 8.14beta4 will do the trick. (Sorry for the
> typos earlier.)
> Add a "-d" option to your map definition to use datagram mode.

One problem with this patch is that it doesn't check for read
timeouts. It probably also ought to use a unique messageID in each
transaction, since it's possible that a delayed reply may still arrive
at a later time.

Re: SOCKETMAP

"hyc" writes:

> On Jan 28, 10:09 am, Claus Aßmann > please)@mine.informatik.uni-kiel.de> wrote:
>> Andrzej Adam Filip wrote:
>>
>> > "hyc" writes:
>> > >> 500 bytes for *one* reply is sufficient for most situations
>> > >> except big aliases.
>> So it "may" work, but maybe it fails? Does the patch handle those
>> errors in a sensible way?
>
> This error can't get handled on the receiving end. If a set of data is
> too large, the sender will receive an error. It's up to the sender to
> do something sane about it, e.g. try again with a smaller message,
> most likely one indicating a failure. Given that the maximum size of a
> datagram is 65507 bytes, it would be pretty rare for the sender to
> fail but when it does there are very few choices for recovery.

AFAIR for WAN UDP services it recommended to limit payload to 536 *BUT*
would be socket UDP servers will usually work over unix sockets or LAN.

> It would be pointless to try any other type of recovery (such as
> splitting the message into multiple datagrams) since then you would
> need to do sequence numbers and other such features that TCP already
> implements. At any rate, the patch doesn't address it because that's
> not the receiver's responsibility.

Even with conservative 536 bytes limit on UDP queries/replies the UDP
socket map would be useful for all map except aliases with big alias
expansions.

> [...]

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

On Jan 28, 12:48 pm, Andrzej Adam Filip wrote:
> I can agree that your objections/usefulness queries are perfectly OK for
> *non FFR* code. I have not "inspected code" of this particular patch.
>
> I can assure you that *writing* UDP (connection less) socket map
> *server* would be *much simpler*, especially load balancing would be
> *much easier* to implement. *For me* it is even more important than
> scale of the overall performance gains.

I don't understand how it simplifies any of this.

> UDP services in general are easier to write than TCP services but harder
> to debug in case of under heavy load problems.

Indeed, you pretty much have to reinvent TCP. E.g., you need to
implement your own retry timeouts if you want to guarantee message
delivery. I recall there was an RDP protocol (Reliable Datagram
Protocol) being worked on many years ago, but it never seemed to catch
on. I wonder why not...

Re: SOCKETMAP

"hyc" writes:

> On Jan 28, 12:48 pm, Andrzej Adam Filip wrote:
>> I can agree that your objections/usefulness queries are perfectly OK for
>> *non FFR* code. I have not "inspected code" of this particular patch.
>>
>> I can assure you that *writing* UDP (connection less) socket map
>> *server* would be *much simpler*, especially load balancing would be
>> *much easier* to implement. *For me* it is even more important than
>> scale of the overall performance gains.
>
> I don't understand how it simplifies any of this.

In case of UDP it is not possible to a lot of TCP connections
with very low overall queries per second rate.

Classic (and the most simple to implement) variant of TCP servers uses
one process per one TCP connection.

>> UDP services in general are easier to write than TCP services but
>> harder to debug in case of under heavy load problems.
>
> Indeed, you pretty much have to reinvent TCP. E.g., you need to
> implement your own retry timeouts if you want to guarantee message
> delivery. I recall there was an RDP protocol (Reliable Datagram
> Protocol) being worked on many years ago, but it never seemed to catch
> on. I wonder why not...

In case of UDP socket map (IMHO) single retry would be sufficient
[ after fixed + variable delays e.g. between 2 and 5s after first query]
In first implementation intended mostly for unix socket the retries may
be skipped.

My intention is to make socket maps server queried by sendmail as simple
as possible => no resends whatsoever by socket map server.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

"hyc" writes:

> On Jan 28, 10:09 am, Claus Aßmann > please)@mine.informatik.uni-kiel.de> wrote:
>> Where is your performance analysis? Please post reproduceable data that
>> demonstrates a non-neglible performance increase for the MTA. Or is
>> there another good reason for this additional code?
>
> Since Andrzej requested this I'll leave it to him to benchmark it,
> because if he has a genuine interest in it, he'll do it. Anyone can
> say "it would be nice if..." Let's see if he backs up his words with
> action.
> -- Howard

It is something I can live without :-)
Do not count I am going to "push it" very hard.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

On Jan 28, 2:09 pm, Andrzej Adam Filip wrote:
> It is something I can live without :-)
> Do not count I am going to "push it" very hard.

I don't think it's something I particularly need either.

By the way, I left out these lines in the previous patch. But since
nobody is going to actually use the patch it probably doesn't matter.

if (replylen > SOCKETMAP_MAXL)
{
syserr("451 4.3.0 socket_map_lookup(%s): reply too long:
%u",
map->map_mname, replylen);
*statp = EX_TEMPFAIL;
goto errcl;
}
+ colon = strchr(lenbuf, ':');
+ lenlen = colon - lenbuf;

Re: SOCKETMAP

Andrzej Adam Filip wrote:

> It is something I can live without :-)

Great... so you are asking several times for it and when finally
someone writes a patch you aren't even helping to demonstrate that it
is useful, but you like to push it onto "sendmail.org" and complain if
"they" don't add features.

Bye.

Re: SOCKETMAP

Claus Aßmann
writes:

> Andrzej Adam Filip wrote:
>
>> It is something I can live without :-)
>
> Great... so you are asking several times for it and when finally
> someone writes a patch you aren't even helping to demonstrate that it
> is useful, but you like to push it onto "sendmail.org" and complain if
> "they" don't add features.
>
> Bye.

Claus,

1) I think functionality provided this specific patch would be useful.
Yes, I would like somebody else to do *sendmail part*.
Yes, I do seriously consider switching sendmail to another (GPL) MTA.
I have tried at least twice to do "read and understand all sendmail-8
source code and twice came to conclusion would not be time effective
effort. Keeping good documentation for developers does pay in
attracting new "code submitters for free".

2) I have "complained" about *two other specific patches*:
a) "wide" patch
[debian distributes "alternative" sendmail package with the path]
b) check_local files for sendmail.cf generation

Could you tell us why the above patches have not "reached"
sendmail.org distribution?
Could you tell us where can I find with reasonable ease answers to
such questions to stop irritate *you*?

I do not expect most patches not included into "main trunk"
to survive more than a few years. It makes me ask the standard
question about attempts to submitting code patches to main
code trunk (*currently* sendmail.org) every time.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Before You Ask: http://anfi.homeunix.net/sendmail/B4UAsk-Sendmail.html
http://anfi.homeunix.net/sendmail/ [orkut,linkedin,xing]

Re: SOCKETMAP

On Jan 28, 10:54 pm, Andrzej Adam Filip wrote:
> Keeping good documentation for developers does pay in
> attracting new "code submitters for free".

Good point. Claus, when are you going to post the coding standards/
developers guidelines?

Re: SOCKETMAP

On Jan 29, 2:29 am, "hyc" wrote:
> On Jan 28, 10:54 pm, Andrzej Adam Filip wrote:
>
> > Keeping good documentation for developers does pay in
> > attracting new "code submitters for free".Good point. Claus, when are you going to post the coding standards/
> developers guidelines?

I dont know what Claus considers proper coding standards and my code
hasnt reached sendmail trunk either, but here is the principles I keep
in mind while patching sendmail.

Make your code look like the rest of the code, use sendmail provided
facilities such as debug tracing, memory allocation, (exceptions,
resource pools?), keep it self contained. Try not to depend on other
patches.

Patches should be self encapsulating with _FFR definitions that record
themselves in conf.c

This means that everything should work as if unpatched if the _FFR is
not defined.

If the _FFR is defined, configuration should not be further required
to get reasonable behavior (this way 3rd parties will feel much safer
compiling with patch turned on).

Having said all that, there are some things that are really difficult
to do as self contained _FFR's.

Re: Coding standadrd (was: SOCKETMAP)

hyc wrote:

> Good point. Claus, when are you going to post the coding standards/
> developers guidelines?

http://www.sendmail.org/doc/CodingStandard.html