Hi,

I have a proposal for a new Milter flag to be negotiated upon filter
startup:

SMFIF_TERMLF

If this flag is provided, then Sendmail terminates body lines with
a single LF character (\n) rather than a CRLF pair (\r\n). If the flag
is omitted, current behaviour is maintained.

The reasoning is that Sendmail is pulling the data out of the queue file
and very carefully converting all the \n to \r\n. I bet that 99% of
the milters out there then just as carefully convert \r\n back to \n so
the body can be processed more easily with normal UNIX text-processing tools
that expect lines to be terminated with \n.

All this insertion and deletion of \r's seems like an awful waste.

Comments?

(This came up after a discussion on the the MIMEDefang list about
"bare" CR's in the SMTP body. It turns out that with Sendmail, it is
impossible for a milter to distinguish between a "bare" LF transmitted
over SMTP and a CRLF terminator; the milter sees CRLF for both. A
milter can, however, see a bare CR character. I'm not sure if the
implicit conversion of LF to CRLF by Sendmail could have security
ramifications for MUAs, but it seems to me it could.)

Regards,

David.

Once upon a time, David F. Skoll <dfs [at] roaringpenguin.com> said:
>The reasoning is that Sendmail is pulling the data out of the queue file
>and very carefully converting all the \n to \r\n.

Does sendmail write the data to the queue file before passing it to
milters? I thought it copied it from the network buffer (which would
have the \r already as that is the network protocol).

>I bet that 99% of
>the milters out there then just as carefully convert \r\n back to \n so
>the body can be processed more easily with normal UNIX text-processing tools
>that expect lines to be terminated with \n.

I bet that 99% of the milters out there don't use normal UNIX
text-processing tools to parse messages. Most milters are either C or
perl and probably don't care about end-of-line characters.
--
Chris Adams <cmadams [at] hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Chris Adams wrote:

> Does sendmail write the data to the queue file before passing it to
> milters?

Yup.

> I thought it copied it from the network buffer (which would
> have the \r already as that is the network protocol).

Nope.

> I bet that 99% of the milters out there don't use normal UNIX
> text-processing tools to parse messages. Most milters are either C or
> perl and probably don't care about end-of-line characters.

I know some Perl modules don't handle embedded CR's too intelligently.

Anyway, a survey: How many milter authors write code to convert CRLF to LF?

--
David.

Once upon a time, David F. Skoll <dfs [at] roaringpenguin.com> said:
>Anyway, a survey: How many milter authors write code to convert CRLF to LF?

Not me (one milter).
--
Chris Adams <cmadams [at] hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Not me (currently writing one milter)

David F. Skoll wrote:
> Chris Adams wrote:
>
>
>>Does sendmail write the data to the queue file before passing it to
>>milters?
>
>
> Yup.
>
>
>>I thought it copied it from the network buffer (which would
>>have the \r already as that is the network protocol).
>
>
> Nope.
>
>
>>I bet that 99% of the milters out there don't use normal UNIX
>>text-processing tools to parse messages. Most milters are either C or
>>perl and probably don't care about end-of-line characters.
>
>
> I know some Perl modules don't handle embedded CR's too intelligently.
>
> Anyway, a survey: How many milter authors write code to convert CRLF to LF?
>
> --
> David.

The effort to arrange for the flag to be negotiable is greater than the
actual test of the flag.

I had to do plenty of that as part of the milter-rrres patch.
(apologies to all those to whom this is old hat)

IIRC you have to update

include/libmilter/mfdef.h
include/libmilter/mfapi.h
sendmail/milter.c milter_body()

right inside the while loop, by the line

/* Change LF to CRLF */
if (c == '\n')

Make it

if (c == '\n' && !bitset(SMFIF_TERMLF, m->mf_fflags))

Surround it with _FFR_MILTER_TERMLF and update sendmail/conf.c _FFR
array to describe the new _FFR

Add the definition to site.config.m4, rebuild libmilter (probaly best
to "rm -rf obj*" first) and share the patch.

Watch out for future realeases changing the way feature flags are
negotiated just, like 8.13.5 did to protocol flags (the new way DOES
appear to be easier and better, but still took a rereading of the diff
to catch)

Joe

David F. Skoll wrote:

> I have a proposal for a new Milter flag to be negotiated upon filter
> startup:

> SMFIF_TERMLF

> If this flag is provided, then Sendmail terminates body lines with
> a single LF character (\n) rather than a CRLF pair (\r\n). If the flag
> is omitted, current behaviour is maintained.

This would work for sendmail 8 (which has a lot of code to do that
conversion), but sendmail X just streams the data to the (p)milter as
it comes in from the net.

Hence I'm not sure whether it worth it as milter authors would
have to write additional code to deal with the two line endings.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

Claus Aßmann wrote:
> David F. Skoll wrote:

>
> This would work for sendmail 8 (which has a lot of code to do that
> conversion), but sendmail X just streams the data to the (p)milter as
> it comes in from the net.

Current behaviour is fine for me : receiving the stuff as it was received from
the net.

Jose Marcio Martins da Cruz wrote:

> Current behaviour is fine for me : receiving the stuff as it was received
> from the net.

Except that's not what SM8 does. It reads the queue file and then
converts LF to CRLF.

I guess I should rephrase it: How many Milter authors *care* about
line terminators? If milters don't care, then SM8 probably doesn't
need to do all the work to convert LF to CRLF, but can just leave line
endings as LF.

I wasn't aware that SMX took data straight off the network. That makes
the proposal much less attractive.

Regards,

David.

Claus =?iso-8859-1?Q?Aßmann?= (ca+sendmail(-no-copies-please) [at] mine.informatik.uni-kiel.de) wrote:
: David F. Skoll wrote:

: > I have a proposal for a new Milter flag to be negotiated upon filter
: > startup:

: > SMFIF_TERMLF

: > If this flag is provided, then Sendmail terminates body lines with
: > a single LF character (\n) rather than a CRLF pair (\r\n). If the flag
: > is omitted, current behaviour is maintained.

: This would work for sendmail 8 (which has a lot of code to do that
: conversion), but sendmail X just streams the data to the (p)milter as
: it comes in from the net.

Can sendmail X chain (p)milters like sendmail 8 can?

John Nemeth wrote:

> Can sendmail X chain (p)milters like sendmail 8 can?

No. I prefer to have this functionality "outside" the SMTP server for
more flexibility. sm8 has a fixed way the milters are called which
might not always be what the user needs. Hence a "milter multiplexor"
should be written that can interface with multiple (p)milters.
--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.