Copyright © 2005 L-Soft international, Inc.
15 June 2005
| IMPORTANT: LISTSERV 14.4 requires a version 14.4 LAK! You must obtain and install a LISTSERV version 14.4 product LAK prior to upgrading your server, or LISTSERV will not start after the upgrade. |
For the first time in L-Soft's history, we have been working on two new LISTSERV versions in parallel: 14.4 and 15.0. While the goal of the 14.4 project was to release important new functionality as soon as possible, a larger team was assembled to tackle the more ambitious 15.0 release concurrently with the development of 14.4. Of course, the new features introduced in 14.4 will also be included in 15.0.
Spam has become such a central issue in the e-mail industry that it places new demands not only on the functionality and performance of e-mail software, but also on its very architecture. Literally hundreds of creative solutions have been deployed by customers to fight spam. Many of these solutions require the use of a specific brand of mail server, or in some cases of custom "homebrew" modifications to existing mail server software. Because of L-Soft's top-performing but proprietary mail-merge architecture, this was only possible for customers who did not need to send personalized messages. Version 14.4 changes that, with the introduction of Embedded Mail-Merge ("EMM" in short).
In a nutshell, EMM means that LISTSERV performs the mail-merge operation instead of delegating it to the mail server. This makes it possible for any SMTP-compliant outbound mail server to be used in conjunction with LISTSERV, even for mail-merge operations, including passive probing and other features that internally use mail-merge. Of course, the use of EMM is entirely optional, and is disabled by default for compatibility with earlier versions.
The biggest challenge with EMM was of course performance. We have worked very hard on developing an efficient implementation that would scale up to the largest workloads imaginable today. My desktop, an inexpensive AMD 3400+, turned out to be able to generate and forward 500 mail-merge messages per second.1 While this is sufficient for most people, customers with time-critical information need even higher delivery rates, which are possible with another feature called "DISTRIBUTE workers." In short, EMM will provide more than enough scalability for current and future needs.
There are of course other improvements besides EMM, which you will find in the "Highlights" section below. Background list maintenance in particular is a must-have for sites with many lists, or even for sites with large lists and interactive (web interface) users. Version 14.4 will also introduce support for new operating systems, but this will be announced separately once we complete our quality assurance tests.
In keeping with L-Soft's commitment to the flexibility and scalability of our products, LISTSERV 14.4 showcases new features that make using, running, and maintaining LISTSERV smoother and more adaptable to new and existing email installations. These changes include modifications to the Web interface ("wa") that make administering lists and composing messages much easier. LISTSERV 14.4 remains the only e-mail list management application to include virus protection, and new F-Secure version certification is part of this release, as well as the ability to use other anti-virus applications to protect email going through LISTSERV.
USABILITY: Embedded mail-merge
USABILITY: [HPO only] DISTRIBUTE workers
USABILITY: Configuration-Owner= list header keyword restricts who may modify the list configuration (header)
WA: Current version
WA: Web-based LAK input
WA: New list deletion interface
WA: New list posting interface
WA: Other improvements
ANTI-VIRUS [Windows] Foreign anti-virus systems are now partially supported
ANTI-VIRUS: [Windows] F-Secure Anti-Virus 5.50-SR1 for Windows Servers now supported
ANTI-VIRUS: [Linux-x86] F-Secure Anti-Virus 4.63 for Linux Servers now supported
ANTI-VIRUS, ANTI-SPAM: [AVS sites only] Improvement to how spam/virus "floods" are handled
PERFORMANCE IMPROVEMENTS: [HPO] Performance improvements for sites with many lists
OTHER: Miscellaneous changes and fixes
Important OS Support Information
Applying LISTSERV 14.4
Any SMTP server can now be used to deliver messages that use LISTSERV's mail-merge functions. When operating in this mode, the proprietary mail-merge features of LSMTP are not required. Of course, customers who already have LSMTP can ignore this section and continue to use it as before. Support for LSMTP-based mail-merge is not being withdrawn.
Embedded mail-merge is activated in LISTSERV by adding the new EMBEDDED_MAIL_MERGE variable to the site configuration file and restarting LISTSERV.
Examples:
VM: |
EMBEDDED_MAIL_MERGE = 1 |
VMS: |
EMBEDDED_MAIL_MERGE "1" |
unix: |
EMBEDDED_MAIL_MERGE=1 export EMBEDDED_MAIL_MERGE |
Win: |
EMBEDDED_MAIL_MERGE=1 |
Embedded mail-merge is a "brute-force" method. In this mode, each individual merged message is written to LISTSERV's spool as a single-recipient MAIL file. In turn, each individual MAIL file is then delivered to the outbound SMTP_FORWARD host(s).
LISTSERV HPO's advanced mail-merge algorithms enable it to handle between 2 and 5 times more embedded mail-merge volume, as compared to LISTSERV Classic. As always where such general performance estimates are made, individual results can and probably will vary.
Sites with high mail volumes will probably want to spread the load across multiple outbound mail servers. Please refer to the "SMTP worker" feature (SMTP_FORWARD_n configuration variable) in the Site Manager's Operational Manual for more information. In addition, the new DISTRIBUTE "workers" (detailed in the next section) can be used to move mail smoothly in very high volume situations.
LISTSERV Lite does not support this new feature because it does not support mail-merge at all.
Spreading the delivery load across multiple outbound mail servers will, in most cases, allow LISTSERV to deliver millions of messages per day using embedded mail-merge. But, for really high volumes, eventually there will come a time when the server running LISTSERV is the bottleneck. With embedded mail-merge, it is LISTSERV, rather than the mail server, that inserts merge fields in messages and executes conditional blocks. As a result, the processing cost of merging the data is shifted from the mail server to LISTSERV, and more CPU resources are required on the LISTSERV server.
The solution is DISTRIBUTE workers – subordinate LISTSERV servers working in conjunction with the primary LISTSERV server to help it offload the work of generating mail-merge messages and forwarding them to the outbound mail servers. Each such subordinate LISTSERV server is called a "DISTRIBUTE worker." In a nutshell, a DISTRIBUTE worker is a bare-bones LISTSERV installation that does nothing but process embedded mail-merge distributions. You can have an unlimited number of DISTRIBUTE workers, providing enough scalability for the most demanding workloads. Currently, a worker cannot delegate any work to sub-workers, but the primary instance can drive any number of workers, and delegation is not a very resource-intensive process.
The primary LISTSERV instance will quickly divide and delegate mail-merge distributions among the DISTRIBUTE workers. The primary instance can assign itself a share of the work, or delegate everything to the workers. The less work it assigns to itself, the more responsive it will be to other tasks, and the more bounces it will be able to process. L-Soft does not recommend assigning any mail-merge work to the primary instance, except when there is only a single DISTRIBUTE worker.
In most cases, each worker will run on a separate, dedicated server, but this is not a requirement. On a very large server with many CPUs and as many independent disk drives, it could make sense to run multiple instances on the same server.
DISTRIBUTE workers are enabled with two options set in the primary instance's site configuration:
1. The DIST_FORWARD site configuration parameter defines the hostnames of the DISTRIBUTE workers and optionally the weighting used to determine what portion of the load each worker should bear.
The syntax for the DIST_FORWARD parameter setting is:
[[n*]hostname [[n*]hostname [...]]This is a space-separated list of all your DISTRIBUTE workers.
For "hostname", you can specify just the hostname if the userid is LISTSERV. If the userid is not LISTSERV, then you must specify a full RFC822 e-mail address pointing to the USERID= address for that server.
Optionally, you can specify a weight (the "n" parameter), which defaults to 1. The number of recipients sent to a particular server is proportional to its weight. The weight must be greater than zero. If a particular server is specified multiple times, the weights are added.
By default, the primary server also takes one share of recipients. In other words, it has a weight of 1. You can change that using the special hostname "SELF". So, if you want a double share for your primary LISTSERV because it runs on a bigger server, just add 2*SELF to the list. SELF is the only worker allowed to have a weight of zero. In this case, nothing is processed locally (except for jobs that are below the threshold – see below). L-Soft recommends 0*SELF if there are two or more DISTRIBUTE workers.
Examples:
VM: |
DIST_FORWARD = DIST1.EXAMPLE.COM 2*DIST2.EXAMPLE.COM |
VMS: |
DIST_FORWARD "3*SELF 2*DIST.EXAMPLE.COM" |
unix: |
DIST_FORWARD="0*SELF DIST1.EXAMPLE.COM DIST2.EXAMPLE.COM" export DIST_FORWARD |
Win: |
DIST_FORWARD=DIST1.EXAMPLE.COM DIST2.EXAMPLE.COM |
2. The DIST_FORWARD_THRESHOLD setting is optional, and defines the minimum number of total recipients a delivery must have in order to make use of the workers. To take an extreme example, it would be inefficient to forward a message with a single recipient to a worker when it could be processed directly at a lower cost. The default value, 100, is usually suitable when the primary LISTSERV server also takes a share of the work. When using the 0*SELF option, it can be necessary to reduce DIST_FORWARD_THRESHOLD.
Examples:
VM: |
DIST_FORWARD_THRESHOLD = 100 |
VMS: |
DIST_FORWARD_THRESHOLD "100" |
unix: |
DIST_FORWARD_THRESHOLD=100 export DIST_FORWARD_THRESHOLD |
Win: |
DIST_FORWARD_THRESHOLD=100 |
There are no special configuration steps for the DISTRIBUTE worker, other than to enable embedded mail-merge and configure SMTP_FORWARD_n.
To use this feature, the primary LISTSERV server must run LISTSERV HPO. Each DISTRIBUTE worker also requires an individual license known as a "DISTRIBUTE worker license" and identified by a SCOPE=DISTWORKER option in its license key. These licenses are cheaper than regular LISTSERV licenses, but do not provide much functionality beyond DISTRIBUTE operation. Alternatively, it is also possible to run a DISTRIBUTE worker using a regular LISTSERV HPO license if a fully functional LISTSERV installation is desired on the worker server, or if a spare HPO license is available. Using a LISTSERV HPO license for the DISTRIBUTE worker will also provide better performance.
This keyword is not available in LISTSERV Lite and can only be set by the LISTSERV maintainer.
This new optional keyword defines who is authorized to change the list configuration. The default value is "Configuration-Owner= Owner", which allows the list owner(s) to modify the list header. The values for this keyword are a subset of access-level. Valid values are: Owner, Postmaster, net-address (for example, user@host.com), Owner(listname), and (listname).
The purpose of this keyword is to make it easier for LISTSERV administrators to manage non-technical list owners. It allows the LISTSERV maintainer to create a list where the owner can manage subscriptions normally, but cannot make changes to the list configuration.
It is also possible to use this keyword to create a separate mailing list of LISTSERV users who do not have full LISTSERV maintainer privileges, but can manage lists other than those they own explicitly.
For third-party TCPGUI programmers, there is a new X-LOGCK function, CONFOWNER(listname), to allow an application to find out in advance whether a PUT command will succeed. It works just like OWNER(listname) but checks "Configuration-Owner=" if present.
LISTSERV 14.4 requires that the Web interface commonly known as "wa" is version 2.3.31 or later. To determine what version of "wa" you currently have running, invoke "wa" with the parameter ?DEBUG-SHOW-VERSION . For instance:
Unix – http://yourserver/path-to-wa/wa?DEBUG-SHOW-VERSION
Windows, OpenVMS – http://yourserver/path-to-wa/wa.exe?DEBUG-SHOW-VERSION
Site Administrators can now enter their LAKs within the Web interface. Under Server Management, select LAK Input to pull up an interface that automatically displays current license information and contains a text input box for entering a LAK. Text can be manually keyed in or copied and pasted from a text file or e-mail message.
Site administrators now have a Web interface from which to delete LISTSERV lists from the server. From Server Management, select Delete Mailing List.
Messages can be created in plain text, HTML, or multipart. An upload applet will upload HTML content from a local drive to LISTSERV. Attachments can be uploaded as well. From the list home page, select Post to the List.
Various improvements have been made for viewing archived HTML and multi-part messages.
Some newly discovered vulnerabilities were fixed.
This feature is not available on anti-virus stations (AVS).
Version 14.4 adds anti-virus protection for those sites that for policy or other reasons must run anti-virus systems other than F-Secure on their servers. Support for these "foreign" anti-virus systems is not as comprehensive and is not fully integrated with LISTSERV in the same way that FSAV is, but LISTSERV will give the AV system the opportunity to scan messages.
To have LISTSERV messages scanned by an AV system other than FSAV, add the FOREIGN_ANTI_VIRUS parameter to your site configuration.
Win: |
FOREIGN_ANTI_VIRUS=1 |
The anti-virus system must be installed on the LISTSERV server, and must block creation of a .EXE file containing a virus with the error "access is denied". Only anti-virus systems with real-time scanning capabilities are supported.
Use of this feature is at your own risk. Because the foreign AV system is not fully integrated with LISTSERV as is FSAV, LISTSERV cannot check that the AV system is up and running and has up-to-date virus definitions. It can only assume that any file that is not denied access does not contain a virus. If you use this feature, it is recommended that you test it thoroughly and set up an external monitoring system to make sure that the anti-virus system is always active when LISTSERV is.
Since the foreign AV system is not integrated with LISTSERV, LISTSERV does not know which viruses have been detected. Anti-virus reports will identify every virus detected as "unknown virus".
This feature is part of the LISTSERV message scanner, and therefore requires a valid and current maintenance LAK. See http://www.lsoft.com/products/listserv_av.asp for more information.
L-Soft certified F-Secure 5.50 for Windows Servers when LISTSERV 14.3 was released. F-Secure Anti-Virus 5.50-SR1 is a "service release" of FSAV 5.50, released on 4 April 2005. It consolidates a number of fixes into a new standard kit. FSAV 5.50-SR1 is also fully supported by LISTSERV 14.3 and later.
The fixes include (per F-Secure):
These hotfixes solve the critical vulnerability issue described in the security advisory FSC-2005-1. This vulnerability may allow remote hackers to execute code of their choice on affected machines. In addition, they solve problems that may appear after virus definition database updates. The AVP engine may fail to work properly after definition updates under certain conditions.
This fix improves the scanning logic.
This hotfix improves the memory handling.
This hotfix solves issues with compatibility with F-Secure Anti-Virus for Windows Servers and VERITAS BackupExec.
L-Soft recommends updating to FSAV 5.50-SR1 if you have not done so already. You can identify the original 5.50 build by its build number, 10260. 5.50-SR1 has a build number of 11110. The new 5.50-SR1 kit may be downloaded from ftp://ftp.lsoft.com/f-secure/fsavsrv550-SR1.exe.
Before installing or upgrading, you may want to review L-Soft's Installing F-Secure Anti-Virus document, as well as the LISTSERV/F-Secure FAQ.
F-Secure Anti-Virus is now certified up to and including version 4.63 for use with LISTSERV 14.4. Customers can obtain an installation key for FSAV 4.63 from their sales representative. To ensure optimal virus detection, this update is recommended, but not mandatory. Because of known issues with versions of FSAV older than 4.50, L-Soft strongly recommends an update to at least FSAV 4.50. These issues are:
The FSAV for Linux Servers 4.63 kit may be downloaded from ftp://ftp.lsoft.com/f-secure/fsav-srv-4.63.4110-2-lsoft.tgz.
Before installing or upgrading, you may want to review L-Soft's Installing F-Secure Anti-Virus document, as well as the LISTSERV/F-Secure FAQ.
Two new AVS-specific options, AVS_DROP_SPAM and AVS_DROP_VIRUS, are available to deal with mail flooding that may result from a mail-bombing attack on a LISTSERV site protected by an external LISTSERV Anti-Virus Station (AVS).
Both options, which default to 0, can be set to 1 to tell an AVS server to drop messages on the floor if they turn out to be spam or infected with a virus. The normal behavior is to return the message to the primary LISTSERV instance, which can then log it and decide what to do with it. In the case where the primary instance is configured to drop the message, it may be more efficient to drop it at the AVS.
Examples:
VM: |
AVS_DROP_SPAM = 1 AVS_DROP_VIRUS = 1 |
VMS: |
AVS_DROP_SPAM "1" AVS_DROP_VIRUS "1" |
unix: |
AVS_DROP_SPAM=1 export AVS_DROP_SPAM AVS_DROP_VIRUS=1 export AVS_DROP_VIRUS |
Win: |
AVS_DROP_SPAM=1 AVS_DROP_VIRUS=1 |
Available in LISTSERV HPO only
The TUNE_MANY_LISTS site configuration parameter was released in LISTSERV 14.3 to provide significant improvements for sites with many lists and large amounts of physical memory. LISTSERV 14.4 adds additional improvements for sites with many lists. Some of these are always active in LISTSERV HPO but are further improved by setting TUNE_MANY_LISTS to 1. Others are only activated when TUNE_MANY_LISTS is set to 1.
Examples:
VM: |
TUNE_MANY_LISTS = 1 |
VMS: |
TUNE_MANY_LISTS "1" |
unix: |
TUNE_MANY_LISTS=1 export TUNE_MANY_LISTS |
Win: |
TUNE_MANY_LISTS=1 |
The main performance improvements for LISTSERV HPO 14.4 are:
General hardware tuning tips for sites with many lists:
The following miscellaneous fixes and changes are included in LISTSERV 14.4.
L-Soft formally supports FreeBSD (Intel) and Linux (S/390) since LISTSERV 14. A comprehensive list of operating systems (and versions) under which LISTSERV is supported can be found at: http://www.lsoft.com/products/listserv_os.asp
LISTSERV version 14.1 (formerly 1.8e-2002a) was the last version for several operating systems that have become obsolescent over the life of this product cycle. In addition, the following operating systems were no longer supported starting with version 14.3:
Sites running these operating systems should migrate to a different operating system. Please contact your sales representative for further information.
Sites running the Windows 95 shareware should note that their licenses will not activate the product under Windows XP. Please contact your sales representative for alternatives if you are planning to upgrade to Windows XP (optionally you may migrate to the LISTSERV Lite Free Edition). Sites running the Windows 95 Lite Free Edition can simply upgrade to the Windows NT/2000/XP LISTSERV Lite Free Edition. You may also elect to continue running LISTSERV under Windows 95/98/Me, but there will be no further new versions or fixes for that platform.
|
IMPORTANT: Install your LISTSERV 14.4 product LAK before upgrading! A valid product LAK (License Activation Key) with "REL=14.4" must be installed before upgrading, or LISTSERV will not start after the upgrade. If you have not received a LISTSERV 14.4 product LAK, please contact your sales representative or SALES@LSOFT.COM before upgrading! To find out if you can upgrade to LISTSERV 14.4 with your current license key, please issue a SHOW LICENSE command to LISTSERV and examine the response. It will be similar to this: License type: Permanent Expiration date: None - perpetual license Maintenance until: 31 May 2005, serial number MNT-XYZ-1 Capacity: Unlimited Version: 14.4 Serial number: XYZ-1 Build date: 15 May 2005 Your license key will be valid for the 14.4 upgrade if your current LAK is for version 14.4 or higher. The initial 14.4 release will not require a current maintenance LAK to activate it (although the maintenance LAK will of course be required if you use LISTSERV features that require current maintenance). Later fixes or "level-set" releases WILL require a current maintenance LAK to activate. The installation kits on the L-Soft FTP site can be used to install a new copy of LISTSERV or upgrade an existing installation. |
To download LISTSERV 14.4, simply go to L-Soft's Web site (or to FTP.LSOFT.COM) and download an evaluation copy of LISTSERV or LISTSERV Lite, then follow the installation instructions for your operating system. The kits can be found at:
http://www.lsoft.com/download/listserv.asp
http://www.lsoft.com/download/listservlite.asp
1. This document does not include upgrade instructions. Please see the installation guide specific to your OS platform for upgrade instructions.
VMS: http://www.lsoft.com/manuals/1.8e/vmsinst.html
Unix: http://www.lsoft.com/manuals/1.8e/unixinst.html
Windows: http://www.lsoft.com/manuals/1.8e/ntinst.html
VM sites currently at the 1.8e or 14.3 level should download ftp://ftp.lsoft.com/listserv/vm/upd144.hex and install it per the "Fixes and Upgrades" section of ftp://ftp.lsoft.com/listserv/vm/00-read.me. VM sites currently at the 1.8d or earlier level must first upgrade to 1.8e (LISTSERV 14) before applying this level set fix.
2. The Unix kit now only requires a single download. You do not need to also download a common.tar.Z file as in previous versions, as those files are now included in the single OS-specific installation file.
3. LISTSERV version 14.4 is available only for operating systems currently supported by L-Soft. When browsing FTP.LSOFT.COM, you may find installation kits for other operating systems, such as Ultrix or SunOS 4.x, but these kits will be based on older versions and/or code bases. L-Soft no longer has development systems for unsupported operating systems and is not in a position to compile LISTSERV 14.4 for these systems.