Mail-tuning

From PTAGISWiki

From brad.knowles@skynet.be Thu Oct 25 07:43:39 2001 Date: Tue, 23 Oct 2001 23:31:33 +0200 From: Brad Knowles <brad.knowles@skynet.be> To: sage-members@usenix.org Subject: Re: [SAGE] how to minimize the evil?

At 2:20 PM -0500 10/18/01, Amos Gouaux wrote:

> I'm not sure if this ever made it out to the list, or at least I > haven't seen it come by. So I was wondering if I could pose this > question to you directly since in fact you were one of the folks on > this list that I had hoped would see it.

I've already responded to this message privately, but I'd like to also respond publicly.

> I don't know, maybe lists of this size are now common place on the > net. Still, I'd hate for us to end up doing something that's going > to piss off a lot of folks.

Large mailing lists are not uncommon. The first suggestion I'd make is that you read the two papers that have already been written on this subject:

R. Kolstad, "Tuning Sendmail for Large Mailing Lists", USENIX, LISA XI Proceedings, October 1997. <http://www.usenix.org/publications/library/proceedings/lisa97/full_papers/21.kolstad/21_html/main.html>

S. R. Chalup, C. Hogan, G. Kulosa, B. McDonald, and B. Stansell, "Drinking from the Fire(walls) Hose: Another Approach to Very Large Mailing Lists", USENIX, LISA XI Proceedings, December 1998. <http://www.usenix.org/events/lisa98/full_papers/chalup/chalup_html/chalup.html>

Of course, another SAGE member has also pointed out a third paper:

V. Skahan, Jr. and R. Katz, "MJDLM: Majordomo based Distribution List Management", USENIX, LISA XIII Proceedings, November 1999 <http://www.usenix.org/publications/library/proceedings/lisa99/skahan.html>

Unfortunately, in this third paper, they did no tuning of either Majordomo or sendmail for performance, unlike the earlier papers. Indeed, while they seem to be aware of the earlier papers, they don't really seem to have learned anything from them. So far as I can tell, they don't even seem to be aware at all of the papers that have been presented on the tuning of sendmail:

B. Knowles, "Sendmail Performance Tuning for Large Systems" SANE '98, November 1998 <http://www.shub-internet.org/brad/papers/sendmail-tuning/>

N. Christenson, "Performance Tuning Your sendmail System" O'Reilly Open Source Conference, August 1999 <http://www.jetcafe.org/~npc/doc/performance_tuning.pdf>

I believe that you will find that more modern mailing list management packages plus more modern MTAs will automatically do many of the right things -- sorting the list of recipients and grouping them into reasonable "chunks" by domain, making parallel connections to multiple recipient sites (to keep the flow going as fast as possible, while also handling slow sites), etc....

I believe that Listserv, Listproc, and mailman probably qualify as "modern mailing list managers", but I don't know about Listar. Postfix certainly qualifies as a modern MTA.

Beyond this, the only two things I can imagine that would be likely to further improve the situation would be:

1) Have your MLM and your MTA use VERPs (Variable Envelope Return Paths) so that you have a guaranteed unique address that is encoded in every e-mail message that is sent out (this makes it a lot easier to debug problems with bouncing addresses and generally keeping your list clean, although it does increase your outbound volume and send one message per recipient).

2) Track your average delivery time on a per-recipient basis, and keep your recipient list sorted by this average, thus delivering soonest to the addresses which you know will be delivered fastest, and reducing your overall average time-to-deliver for all recipients.

-- Brad Knowles, <brad.knowles@skynet.be>

H4sICIFgXzsCA2RtYS1zaWcAPVHLbsMwDDvXX0H0kkvbfxiwVw8FCmzAzqqj1F4dy7CdBfn7 Kc6wmyGRFEnvvxiWQoCvqI7RSWTcfGXQNqCUAnfIU+AT8OZ/GCNjRVlH0bKpguJkxiITZqes MxwpSucyDJzXxQEUe/ihgXqJXUXwD9ajB6NHonLmNrUSK9nacHQnH097szO74xFXqtlbT3il wMsBz5cnfCR5cEmci0Rj9u/jqBbPeES1I4PeFBXPUIT1XDSOuutFXylzrQvGyboWstCoQZyP dxX4dLx0eauFe1x9puhoi0Ao1omEJo+BZ6XLVNaVpWiKekxN0VK2VMpmAy+Bk7ZV4SO+p1L/ uErNRS/qH2iFU+iNOtbcmVt9N16lfF7tLv9FXNj8AiyNcOi1AQAA