5.2 SMTP Worker Pool support
The SMTP worker pool feature allows a site to run large lists on a server and also (with appropriate hardware) to set up high-speed, low-volume delivery nodes for command responses and the like.
SMTP worker pools sound quite complex but are very simple once you understand what they actually do. LISTSERV recognizes 27 mail delivery pools, called A through Z plus a default pool, called '-' (hyphen). By default, LISTSERV puts all outgoing mail in the default pool, and the workers process mail for all the pools. You can tell LISTSERV to put certain types of messages in different pools, as follows:
- The "Delivery-Pool= x,y" list header keyword (must be set by the LISTSERV maintainer) lets you define the delivery pool for postings to the list (x) and for administrative messages related to the list (y). The latter includes bounces, error monitoring reports, welcome messages, etc. It does NOT include command responses of any kind (even if there is just one command and it is for the list) or responses to postings (posting rejection, request for cookie, forward to editor, etc), which are considered to be command replies and always go to the default pool. If y is missing, x defines the pool for both postings and administrative mail.
- The DEFAULT_LIST_POOL keyword, which defines site defaults for the above (space separated).
- The POOL=x parameter for DISTRIBUTE.
- The DEFAULT_DIST_POOL keyword, which defines the default value for the above.
If you use any of the above, LISTSERV will start assigning outgoing mail to the various pools, while the default pool continues to receive command replies and spontaneous messages that are not tied to any particular list (system errors, etc). By default, all the workers are set to process mail from any of the pools, so it will not have any effect until you reconfigure the workers to only manage some of the pools:
SMTP_FORWARD_n=blah blah;POOL=xyzt
(use POOL=* for the default behaviour where all pools are processed). Therefore, each individual worker can be configured to manage either a specific pool, a series of pools, or all the pools (the latter is probably not useful). Note that you must always provide at least one worker to process the default pool (POOL=-), since you will always have messages in this pool. Obviously you don't need to provide workers for pools you do not use.
In the current implementation, the pools are one-letter prefixes to the xxx.MAIL filename, rather than separate directories. This is a little bit less efficient, but it makes it a lot simpler for a LISTSERV maintainer to visually check the contents of the queue. Here is a concrete example where we define pool L to be large lists, X to be a high-priority express service for small lists, and M to be the default pool for normal lists:
DEFAULT_LIST_POOL=M
SMTP_FORWARD_1=2*FAST.LSOFT.COM;FAST-BACKUP.LSOFT.COM;POOL=X-
SMTP_FORWARD_2=REGULAR1.LSOFT.COM;BACKUP1.LSOFT.COM;POOL=M
SMTP_FORWARD_3=REGULAR2.LSOFT.COM;BACKUP2.LSOFT.COM;POOL=M
SMTP_FORWARD_4=5*BULK.LSOFT.COM;BACKUP3.LSOFT.COM;POOL=L
A simpler setup, where we create a fast channel for command replies, would be:
DEFAULT_LIST_POOL=M
SMTP_FORWARD_1=2*FAST.LSOFT.COM;FAST-BACKUP.LSOFT.COM;POOL=-
SMTP_FORWARD_2=whatever we had before;POOL=M
This keeps list messages routed as before and routes command replies to a new, high-speed, low-volume SMTP MTA.
Incidentally, assigning a worker to a single pool is faster because it can then look for files called M*.MAIL and so forth. A worker that manages multiple pools needs to scan for *.MAIL and skip the files it does not service. Either way, introducing pools adds some measure of overhead corresponding to the time spent skipping over files which are not serviced. This would only be a problem in practice if you had a bunch of workers configured to skip most of the files.
Worker pools can get quite interesting for specialized applications when combined with another worker feature called “open time”. This feature lets you define that certain servers should only be used during certain times of the day:
SMTP_FORWARD_n=BULKBOX.XYZ.COM(09:00-17:00,19:00-23:00);OTHERBOX.XYZ.COM
This defines a worker that will use BULKBOX during the times specified, and OTHERBOX the rest of the time. A typical use is to bring bulk delivery boxes to the rescue during peak interactive hours, while making sure not to use them when they are processing large postings. But you can also use it like this:
SMTP_FORWARD_n=SOMEBOX.XYZ.COM(23:00-06:00);POOL=W
This defines a worker that is only allowed to do useful work from 11pm to 6am. If this is the only worker that services pool W, you won't be seeing much activity in that pool outside these hours. Not probes, not welcome messages, nothing (but commands can still be sent and the command reply will be delivered). This could be a good way to shut off a very large list, or to cause a large mailing to start off at a very precise time. The workers are accurate to the minute. Likewise, the workers will stop on a dime (after finishing the current message) at the end of the opening interval. Currently there is no weekday support; however if ithis is needed, it can be implemented with a daily reboot.