source: trunk/third/sendmail/sendmail/TUNING @ 19204

# Copyright (c) 2001-2003 Sendmail, Inc. and its suppliers.
#       All rights reserved.
#
# By using this file, you agree to the terms and conditions set
# forth in the LICENSE file which can be found at the top level of
# the sendmail distribution.
#
#       $Id: TUNING,v 1.1.1.1 2003-04-08 15:08:52 zacheiss Exp $
#

********************************************
** This is a DRAFT, comments are welcome! **
********************************************


If the default configuration of sendmail does not achieve the
required performance, there are several configuration options that
can be changed to accomplish higher performance.  However, before
those options are changed it is necessary to understand why the
performance is not as good as desired.  This may also involve hardware
and software (OS) configurations which are not extensively explored
in this document.  We assume that your system is not limited by
network bandwidth because optimizing for this situation is beyond
the scope of this guide.  In almost all other cases performance will
be limited by disk I/O.


This text assumes that all options which are mentioned here are
familiar to the reader, they are explained in the Sendmail Installation
and Operations Guide; doc/op/op.txt.

There are basically three different scenarios which are treated
in the following:
* Mailing Lists and Large Aliases (1-n Mailing)
* 1-1 Mass Mailing
* High Volume Mail

Depending on your requirements, these may need different options
to optimize sendmail for the particular purpose.  It is also possible
to configure sendmail to achieve good performance in all cases, but
it will not be optimal for any specific purpose.  For example, it
is non-trivival to combine low latency (fast delivery of incoming
mail) with high overall throughput.

Before we explore the different scenarios, a basic discussion about
disk I/O, delivery modes, and queue control is required.


* Disk I/O
-----------------------------------------------

In general mail will be written to disk up before a delivery attempt
is made.  This is required for reliability and should only be changed
in a few specific cases that are mentioned later on.  To achieve
better disk I/O performance the queue directories can be spread
over several disks to distribute the load.  This is some basic tuning
that should be done in all cases where the I/O speed of a single
disk is exceeded, which is true for almost every high-volume
situation except if a special disk subsystem with large (NV)RAM
buffer is used.

Depending on your OS there might be ways to speed up I/O, e.g.,
using softupdates or turning on the noatime mount option.  If this
is done make sure the filesystem is still reliable, i.e., if fsync()
returns without an error, the file has really been committed to
disk.


* Queueing Strategies and DeliveryMode
-----------------------------------------------

There are basically three delivery modes:

background: incoming mail will be immediately delivered by a new process
interactive: incoming mail will be immediately delivered by the same process
queue: incoming mail will be queued and delivered by a queue runner later on

The first offers the lowest latency without the disadvantage of the
second, which keep the connection from the sender open until the
delivery to the next hop succeeded or failed.  However, it does not
allow for a good control over the number of delivery processes other
than limiting the total number of direct children of the daemon
processes (MaxChildren) or by load control options (RefuseLA,
DelayLA).  Moreover, it can't make as good use as 'queue' mode can
for connection caching.

Interactive DeliveryMode should only be used in rare cases, e.g.,
if the delivery time to the next hop is a known quantity or if the
sender is under local control and it does not matter if it has to
wait for delivery.

Queueing up e-mail before delivery is done by a queue runner allows
the best load control but does not achieve as low latency as the
other two modes.  However, this mode is probably also best for
concurrent delivery since the number of queue runners can be specified
on a queue group basis.  Persistent queue runners (-qp) can be used
to minimize the overhead for creating processes because they just
sleep for the specified interval (which shold be short) instead of
exiting after a queue run.


* Queue Groups
-----------------------------------------------

In most situations disk I/O is a bottleneck which can be mitigated
by spreading the load over several disks.  This can easily be achieved
with different queue directories.  sendmail 8.12 introduces queue
groups which are collections of queue directories with similar
properties, i.e., number of processes to run the queues in the
group, maximum number of recipients within an e-mail (envelope),
etc.  Queue groups allow control over the behaviour of different
queues.  Depending on the setup, it is usually possible to have
several queue runners delivering mails concurrently which should
increase throughput.  The number of queue runners can be controlled
per queue group (Runner=) and overall (MaxQueueChildren).


* DNS Lookups
-----------------------------------------------

sendmail performs by default host name canonifications by using
host name lookups.  This process is meant to replace unqualified
host name with qualified host names, and CNAMEs with the non-aliased
name.  However, these lookups can take a while for large address
lists, e.g., mailing lists.  If you can assure by other means that
host names are canonical, you should use

                FEATURE(`nocanonify', `canonify_hosts')

in your .mc file.  For further information on this feature and
additional options see cf/README.  If sendmail is invoked directly
to send e-mail then either the -G option should be used or

        define(`confDIRECT_SUBMISSION_MODIFIERS', `C')

should be added to the .mc file.


* Mailing Lists and Large Aliases (1-n Mailing)
-----------------------------------------------

Before 8.12 sendmail delivers an e-mail sequentially to all its
recipients.  For mailing lists or large aliases the overall delivery
time can be substantial, especially if some of the recipients are
located at hosts that are slow to accept e-mail.  Some mailing list
software therefore "split" up e-mails into smaller pieces with
fewer recipients.  sendmail 8.12 can do this itself, either across
queue groups or within a queue directory.  The latter is controlled
by the 'r=' field of a queue group declaration.

Let's assume a simple example: a mailing lists where most of
the recipients are at three domains: the local one (local.domain)
and two remotes (one.domain, two.domain) and the rest is splittered
over several other domains.  For this case it is useful to specify
three queue groups:

QUEUE_GROUP(`local', `P=/var/spool/mqueue/local, F=f, R=2, I=1m')dnl
QUEUE_GROUP(`one', `P=/var/spool/mqueue/one, F=f, r=50, R=3')dnl
QUEUE_GROUP(`two', `P=/var/spool/mqueue/two, F=f, r=30, R=4')dnl
QUEUE_GROUP(`remote', `P=/var/spool/mqueue/remote, F=f, r=5, R=8, I=2m')dnl
define(`ESMTP_MAILER_QGRP', `remote')dnl
define(`confDELIVERY_MODE', `q')dnl
define(`confMAX_QUEUE_CHILDREN', `50')dnl
define(`confMIN_QUEUE_AGE', `27m')dnl

and specify the queuegroup ruleset as follows:

LOCAL_RULESETS
Squeuegroup
R$* @ local.domain      $# local
R$* @ $* one.domain     $# one
R$* @ $* two.domain     $# two
R$* @ $*                $# remote
R$*                     $# mqueue

Now it is necessary to control the number of queue runners, which
is done by MaxQueueChildren.  Starting the daemon with the option
-q5m assures that the first delivery attempt for each e-mail is
done within 5 minutes, however, there are also individual queue
intervals for the queue groups as specified above.  MinQueueAge
is set to 27 minutes to avoid that entries are run too often.

Notice: if envelope splitting happens due to alias expansion, and
DeliveryMode is not 'i'nteractive, then only one envelope is sent
immediately.  The rest (after splitting) are queued up and queue
runners must come along and take care of them.  Hence it is essential
that the queue interval is very short.


* 1-1 Mass Mailing
-----------------------------------------------

In this case some program generates e-mails which are sent to
individual recipients (or at most very few per e-mail).  A simple
way to achieve high throughput is to set the delivery mode to
'interactive', turn off the SuperSafe option and make sure that the
program that generates the mails can deal with mail losses if the
server loses power.  In no other case should SuperSafe be set to
'false'.  If these conditions are met, sendmail does not need to
commit mails to disk but can buffer them in memory which will greatly
enhance performance, especially compared to normal disk subsystems, e.g.,
non solid-state disks.


* High Volume Mail
-----------------------------------------------

For high volume mail it is necessary to be able to control the load
on the system.  Therefore the 'queue' delivery mode should be used,
and all options related to number of processes and the load should
be set to reasonable values.  It is important not to accept mail
faster than it can be delivered otherwise the system will be
overwhelmed.  Hence RefuseLA should be lower than QueueLA, the number
of daemon children should probably be lower than the number of queue
runnners (MaxChildren vs. MaxQueueChildren).  DelayLA is a new option
in 8.12 which allows delaying connections instead of rejecting them.
This may result in a smoother load distribution depending on how
the mails are submitted to sendmail.


* Miscellaneous
-----------------------------------------------

Other options that are interesting to tweak performance are
(in no particular order):

SuperSafe: if interactive DeliveryMode is used, then this can
be set to the new value "interactive" in 8.12 to save some disk
synchronizations which are not really necessary in that mode.