Description of the changes for release 1.8c of LISTSERV(R) ---------------------------------------------------------- Copyright 1996,1997 L-Soft international, Inc. January 20th, 1997 ************************************************************************* ************************** List owner's notes *************************** ************************************************************************* The release notes for version 1.8c of LISTSERV(R) have been split into two documents: list owner's notes and LISTSERV maintainer's notes. The present list owner's notes describe all the changes that list owners need to be aware of, although in some cases list owners may be impacted by changes described in the LISTSERV maintainer's section. For instance, the availability of a new systemwide option may prompt the LISTSERV maintainer to make a change affecting all the lists on the LISTSERV host. Thus, list owners are advised to read the maintainer's notes as well. ************** * Highlights * ************** - New database functions (available for both VM and non-VM versions) with the same degree of functionality as the original VM database functions, and a more user-friendly syntax. - (non-VM) Web archive interface allowing users to browse list archives interactively. This includes a WWW interface to the new database functions with online help. - "Hands off" bounce processing allowing for accurate, fully automated removal of undeliverable addresses without the list owner having to see a single bounce again. This is accomplished through the delivery of individually addressed list FAQs tagged with a special marker that uniquely identifies the recipient. - Support for "notary" delivery error format (RFC1891 et seq), the oncoming Internet standard for delivery errors. - The INDEX subscription option is now available for non-VM systems and a new GETPOST command (also available under VM) has been introduced to facilitate the retrieval of individual postings. - (non-VM) Enhanced, hierarchical file server functions with sub-catalog support. The LISTSERV administrator can now create catalogs for individual lists, delegating file management operations to individual list owners. - Improved spam detector and spamming counter-measures. - Support for "super-lists", allowing the hierarchical organization of related lists without any cross-posting or duplication. - New SUBJECTHDR subscription option and "Subject-Tag=" list header keyword to insert the list name (or any other word) in the subject of all messages coming from the list, on a per-subscriber basis. - Comprehensive, 40-page user's guide and shorter "quick start" guide now available online. - Per-subscriber daily message quota. *********************************** * Documentation: New user's guide * *********************************** A new, 40-page user's guide is now available online. A plain text version is included with the 1.8c update, replacing the old "INFO GENINTRO" and "INFO PRESENTATION" documents. A shorter, "quick start" guide is also available as "INFO QUICKSTART". You can also FTP a copy of these new manuals from FTP.LSOFT.COM, CD DOCUMENTS. The files are available in a variety of word processing formats - check the FTP server for more information. ************************************* * Usability: New database functions * ************************************* Version 1.8c introduces a new set of database functions, available on all supported operating systems. These new database functions were designed to be more functional and easier to use than the old (VM) implementation, and include a WWW interface (coupled to the web archive interface). Under VM, the old and new database functions coexist and share the same data files, with slightly different syntax and capabilities. The new functions will be faster in most cases and should be used whenever possible. For more information about the new database functions, please see the User's guide. The following checklist gives a brief overview of the differences between old and new database functions: - Searches are initiated using the new SEARCH command, rather than by submitting a DATABASE SEARCH job. The syntax of the SEARCH command is virtually identical to that of the SEARCH clause in database jobs. This command automatically produces an index and, where applicable, context information showing the search words as they appear in the posting. Context information is not available for keyword searches (eg SEARCH xxx WHERE SENDER CONTAINS JACK), or for negative searches (SEARCH NOT XYZ IN xxx - all the postings containing XYZ have actually been excluded, so you will never find XYZ in them). - There is a new operator, NEAR, which is the default for the "locate" clause. In other words, SEARCH JOE SMITH IN XYZ-L looks for JOE and SMITH close to each other. You can still use AND explicitly. Note that 'a NEAR b NEAR c' is defined as '(a NEAR b) AND (b NEAR c)', so this new operator is not fully commutative. - While (on VM) you can use the new SEARCH command to search non-list databases, such as BITEARN or PEERS, you cannot currently retrieve edited results with the GETPOST command for these databases. You should continue to use the old database functions for non-list databases. - The new SEARCH command will only display up to 100 hits. To get the next 100 hits, use a SINCE clause or an item range qualifier: SEARCH xxx IN XYZ.9182- will display all hits with **************************************************************** * Usability: New GETPOST command and INDEX subscription option * **************************************************************** The INDEX subscription option is now available on non-VM systems. In addition, a new command, GETPOST, has been added to facilitate the retrieval of individual messages from the archives. The syntax of the GETPOST command is: GETPOST listname itemrange1 > See the User's guide for a more detailed description of the GETPOST command. Note to VM list owners: depending on the configuration of your host server, you may notice a slight change in the format of the messages returned to subscribers who use the INDEX subscription option. This is because LISTSERV now uses the GETPOST command in conjunction with INDEX subscriptions, as this is more efficient than the database jobs used in the previous version. ******************************************** * Usability: "Hands off" bounce processing * ******************************************** Delivery error notices (commonly called "bounces") are probably the single largest obstacle to the effective operation of mailing lists by non-technical list owners. Any public list with more than a handful of subscribers is bound to be plagued with a number of daily bounces, many of which were written by and for technical experts and make absolutely no sense to the Internet novice. It may not even be clear what message and above all what susbcriber the bounce is referring to. More often than not, novice list owners end up discarding all these bounces unread (usually by sending them to a separate account whose mail is never read), because they simply do not know what to do to correct the problem. Unfortunately, this is not a good situation at all. Here is what happens for each invalid address in a mailing list: 1. The server hosting the mailing list, not knowing that the address is invalid, attempts to deliver the message. This first delivery costs just as much as a delivery to any other address. 2. The target host must then create a delivery error notice and send it back to the server hosting the mailing list. This is usually an expensive operation that, when multiplied by hundreds of daily occurrences, can slow down the target server. While this may be "someone else's problem", on the Internet a "good neighbour" is expected not to cause other sites to waste undue resources. Just like in real life, people do not speak highly of "bad neighbours", and this is a factor to consider if the lists are run for PR purposes. 3. The server hosting the list undergoes the resource cost of receiving this delivery error, and routing it to LISTSERV for processing. 4. LISTSERV determines that this is a delivery error in a format that it is unable to decode, and forwards it to the list owner for manual processing. 5. The server hosting the list delivers the message to the list owner. All told, the resource cost at the host site for a bad address is about 4-5 times higher than for a working address. This means that if a list should accumulate 20% of bad addresses through lack of bounce processing, the cost of running the list will be roughly doubled, and the capacity of the server will be halved. Since bad addresses do not go away on their own, the number of bad addresses just keeps increasing with time, and what initially looked like a minor and perfectly acceptable resource surcharge quickly turns into a serious capacity problem. The solution to this problem is automatic bounce processing, where the computer processes the bounces without human intervention. LISTSERV has had automatic bounce processing since 1992, however this process is not 100% accurate, owing mainly to the lack of a standard for the format of delivery error messages (there is now an IETF standards-track document at the "Proposed Standard" stage, but to date only a handful of products have implemented it). To improve the accuracy and effectiveness of automatic bounce processing, LISTSERV can now optionally take a more active role in the detection of invalid addresses. In addition to reacting to incoming bounces, some of which are unfortunately not intelligible to computer programs, LISTSERV can also be directed to "probe" the mailing list at regular intervals by sending an individually addressed FAQ (or similar document) to the subscribers. This FAQ contains a special marker which uniquely identifies the subscriber, making it possible for LISTSERV to determine that mail for this particular subscriber is bouncing, even when the delivery error message itself is otherwise totally unintelligible. Upon receipt of a bounced probe message, LISTSERV will schedule the delivery of additional probes to the failing address, at an appropriate and controlled rate, to monitor the address over the period of time requested in the "Auto-Delete=" keyword for the list. Thus, bounced probes are acted upon with exactly the same degree of leniency as ordinary delivery errors. By sending additional probes as the need arises, LISTSERV can effectively monitor bouncing addresses even on low-volume lists where there may not be more than a few messages per week. To enable active monitoring, the list owner must take the following steps: 1. Obtain permission from the LISTSERV administrator, if you were not already using the "Renewal=" feature. Probes incur the same system overhead as a renewal message, and are in fact implemented as an option to the "Renewal=" keyword. Since this overhead can be quite significant on some operating systems or configurations, you should obtain permission from the LISTSERV administrator before proceeding, especially if yours is a large list or if you know that the mail host is overloaded. The LISTSERV administrator may also be able to advise you on the appropriate frequency (typically monthly) for the list probes. 2. Retrieve and customize the PROBE1 administrative message, replacing its contents with a FAQ, mini-newsletter, or other appropriate message. This step is optional; the default PROBE1 template is fully functional, if uninteresting. The new &DAYSEQ(n) feature (described in the updated list owner's guide) can be used to create "rotating" collections of FAQs that are both more informative and less "annoying" to the subscribers than a monthly FAQ that is always the same. 3. Add a "Renewal=" keyword to the list header, specifying the "Probe" option. For instance, * Renewal= Yes,Monthly,Probe directs LISTSERV to probe the subscribers on a monthly basis. Please refer to the list owner's manual for more information on the "Renewal=" keyword. These steps do not affect the delivery of bounces to the list owners. That is, you can activate list probing without suppressing the delivery of unparseable bounces to your account. If on the other hand you do want to completely delegate bounce processing to LISTSERV, you should perform the following step: 4. Change the "Auto-Delete=" keyword in the list header to read: * Auto-Delete= Yes,Full-Auto (as opposed to "Yes,Semi-Auto" or "Yes,Manual"). When used in conjunction with active probing, this will direct LISTSERV to suppress the delivery of all bounces to the list owners. It is not possible to suppress bounce delivery without enabling active probing, because the accuracy of passive bounce processing is too low. While this varies widely from one system to another, or even from one list to another on a given system, passive bounce processing typically catches between 50% and 80% of bad addresses. Active probing, on the other hand, should have an accuracy in excess of 99%. Supported mail servers ---------------------- Because LISTSERV receives its incoming mail (including bounces and in particular the special probe bounces) from the mail server servicing the SMTP port of the machine on which LISTSERV is running, active probing may not work with all mail servers, or may require you to upgrade to a certain revision level, as follows: - VM: active probing is supported with all versions of LMail(TM). - VMS: active probing is supported with version 1.1a of LSMTP(TM), and with all versions of MX that include support for the LISTSERV interface. PMDF(R) users should create a dedicated domain for LISTSERV (such as LISTSERV.XYZ.COM) and add a rewrite rule to redirect all traffic for that host to the LSV channel. This also simplifies the creation of new lists (with this setup, it is no longer necessary to define PMDF aliases for the lists). - unix: the current versions of sendmail are not compatible with the list probe, however a patch is available from FTP.LSOFT.COM in the LISTSERV/unix/Contrib directory to implement the necessary changes to sendmail. - Windows: active probing is supported with version 1.1a of LSMTP and with all versions of the SMTP listener. In mixed environments where one mail product processes incoming mail while another is responsible for outgoing mail, the incoming mail server is the one that determines whether active probing is supported. Users of version 1.0a of LSMTP will need to upgrade to version 1.1a before enabling active probing, regardless of the operating system they are using. L-Soft has decided to release 1.1a as a free upgrade, however you must still obtain a (free) 1.1a license key through normal sales channels before installing the new version. Performance considerations -------------------------- Probing has the same impact on the host server as renewal, and the same considerations apply. In particular: - If the server (or network) is already overloaded, you should refrain from using probing until suitable resources are available. - You should never activate probing or renewal on a large list without first checking with the LISTSERV administrator. - You should avoid using "Renewal=" specifications which force LISTSERV to process all renewals on a particular day. If you order LISTSERV to probe 30,000 people on a specific day, it will comply, but service may be impacted. Using less specific syntax options (such as "3-Monthly" as opposed to a series of explicit dates) makes it possible for LISTSERV to spread the probing activity over a period of time. Thus, instead of probing 30,000 people every 30 days, it can instead probe 1,000 people every day, with much reduced impact on the host system. - The first time you activate probing or renewal for a large one-way list, most of the subscribers are likely to be overdue for probing, especially if the list was imported from a database via a bulk ADD. In this case it is best to change the "Renewal=" keyword during a weekend or other period of reduced activity. *********************************************************************** * (non-VM) Usability: Enhanced file server functions and sub-catalogs * *********************************************************************** The file server functions have been enhanced in version 1.8c to facilitate the management of file catalogs (see maintainer's release notes), allow users to manipulate files using the file naming syntaxes they are used to (DOS/Windows, unix, VMS or VM), and allow for the creation of "sub-catalogs" whose management can be delegated to individual list owners. Support for popular file naming conventions ------------------------------------------- To make the file server functions more intuitive to non-technical users, LISTSERV now accepts file specifications in a variety of popular formats. This means users can use the file naming conventions of the operating system they are used to, and do not need to learn a new way to refer to files. For instance, the file NEXT.MEETING in the WG3 sub-catalog can be accessed as: - /wg3/next.meeting - \wg3\next.meeting - WG3:NEXT.MEETING - WG3:[000000]NEXT.MEETING - WG3:<000000>NEXT.MEETING - NEXT MEETING WG3 Note that this file can also be accessed as NEXT.MEETING (without specifying the WG3 directory) unless there are several files by that name in the LISTSERV file system. LISTSERV will reply using the same syntax convention, as the users would expect it. Overview - delegating file management authority ----------------------------------------------- The sub-catalog enhancement allows the LISTSERV administrator to delegate file management authority in a controlled and secure manner. Multiple list owners can be given the authority to maintain their own sub-catalog in a predefined directory. With the LISTSERV-ISP add on (under development), a quota can be imposed on the directory in question. The procedure works as follows: 1. The LISTSERV administrator creates the sub-catalog and identifies the directory where the files will be stored, and the person(s) who will be in charge of managing it ("catalog owners"). 2. The catalog owners use the GET and PUT commands to update their catalog and register new files in their directory. Each file has its own GET and PUT file access codes, allowing the catalog owners to further delegate the management of individual files to third parties ("file owners"). 3. The file owners manage the files in question using the GET and PUT commands. Authorized users can then retrieve the files using the GET command. Creating a sub-catalog ---------------------- To create a sub-catalog, the LISTSERV administrator edits the file called SITE.CATALOG (or site.catalog under unix) in LISTSERV's main directory (the directory where SYSTEM.CATALOG/system.catalog is located - refer to the list owner's guide for more information). A sub-catalog is defined as follows: TEST.CATALOG /home/lists/xyz/test.catalog ALL JOE@XYZ.COM (1) (2) (3) (4) (5) Notes: (1) The name must end in '.CATALOG', but otherwise it can be anything. In particular, there does not need to be a list by that name. (2) This is the directory where ALL the files defined in the sub-catalog will be stored. DO NOT USE LISTSERV'S MAIN DIRECTORY FOR THIS PURPOSE! The catalog owner will be given FULL ACCESS to all the files in this directory, so make sure to create a new, empty directory. If the sub-catalog is being set up for a list owner, it may be a good idea to put the list archives and the sub-catalog in the same directory. (3) A file name must be provided for the sub-catalog file itself. This name, however, does not need to match (1). (4) This file access code controls the authority to INDEX the sub-catalog. This will also be the default GET access code for all the files registered in the sub-catalog. (5) This file access code defines the catalog owner(s) and default file owner(s) for all the files in the sub-catalog. Note that there is no need to reboot LISTSERV after updating the SITE.CATALOG file. Also, bear in mind that you are responsible for the OS-level security of the directory you create for the catalog. The file access codes in SITE.CATALOG only affect operations that go through LISTSERV; it is your responsibility to make sure that other users of the computer are given the appropriate access level to any directory you create for LISTSERV's purposes. Updating the sub-catalog ------------------------ Once the sub-catalog is created, the catalog owner(s) can register new files using the following procedure (in this example, it will be assumed that the sub-catalog is called TEST.CATALOG): 1. Send a GET TEST.CATALOG command to LISTSERV (or, if the catalog is brand new, start from an empty file). 2. Register new file(s) in the catalog (see below). 3. Use the PUT TEST.CATALOG PW=XXXX command to store the updated catalog. Alternatively, if the catalog owner has an account on the LISTSERV host system and write access to the directory associated with the sub-catalog, the file can be edited directly. Note however that, in that case, the LISTSERV-ISP quota system will be inoperative as it has no control over disk accesses which do not go through LISTSERV itself. The format of sub-catalogs is similar to that of SITE.CATALOG: MY.FILE my.file ALL JOE@XYZ.COM (1) (2) (3) (4) Notes: (1) This defines the name of the file as seen by LISTSERV users. That is, the command to retrieve the file will be 'GET /test/my.file' (or 'GET TEST:MY.FILE', 'GET MY FILE TEST', and in most cases just 'GET MY.FILE'). (2) This defines the name of the actual disk file where the contents of MY.FILE will be stored. Normally, you should specify the same as (1), or just an equal sign (LISTSERV will then substitute the name you provided for (1)). However, in some cases you may want to make a particular file available under multiple names. This can be done by registering multiple files (ie multiple values for (1)), and using the same (2) value every time. (3) This file access code determines who can order the file through a GET command. See the list owner's guide for more information. (4) This file access code determines who can update the file with the PUT command. See the list owner's guide for more information. Note: (2) defaults to the value of (1), and (3) and (4) default to the GET and PUT access codes of the sub-catalog itself, respectively. So, in most cases a sub-catalog entry will be as simple as: MY.FILE Additionally, comment lines (starting with an asterisk) or blank lines can be interspersed with file definitions. These comments will be echoed when the sub-catalog is indexed (see below), in sequence with the file definitions. For instance, your catalog could read: * * Files for the XYZ sub-project * XYZ.AGENDA XYZ.BUDGET XYZ.PROPOSAL-1 XYZ.PROPOSAL-2 Indexing the sub-catalog ------------------------ If TEST.CATALOG is defined as: TEST.CATALOG /home/lists/xyz/test.catalog xxx JOE@XYZ.COM Any user who matches the 'xxx' file access code is authorized to issue an INDEX TEST command to get a formatted version of the catalog. For compatibility with older versions of LISTSERV, GET TEST.FILELIST will produce the same results. If there is a mailing list called TEST, a list of the archive files will also be appended automatically. Sub-directories --------------- Sub-directories may be created within a sub-catalog, allowing files to be partitioned hierarchically up to an unlimited depth. See the list owner's guide for more information. ************************************************* * (non-VM) Usability: New web archive interface * ************************************************* Version 1.8c introduces a WWW archive interface that allows users to browse and search list archives (the LISTSERV maintainer controls which lists are or are not visible through this interface). Postings can be organized by date, by topic or by author, and a search function with online help is provided. URLs in the text of the postings are displayed as 'live' hyperlinks. LISTSERV's WWW interface has the following advantages over "hypermail" style web archiving: - The information on the web is always up to date. New postings are shown as soon as they are received. - The postings can be organized in the manner that best suits the reader: by date, by author, by topic, with or without table of contents, with or without showing the author, etc. - Only one copy of the information is kept, and in particular there is no need to create an individual HTML file for each posting. This design allows the interface to scale up gracefully to lists with hundreds of thousands of archived postings, which would otherwise require hundreds of thousands of individual HTML files, wasting disk space (each file takes up at least one disk block) and stressing the file system past reasonable limits. - The search forms can be used to create search requests matching all postings in the last X days. The resulting URL can then be bookmarked and reloaded on a regular basis. - List owners can customize the main page for their lists without any intervention by the LISTSERV maintainer, by updating one of the mail templates for their list. The LISTSERV maintainer can customize common pages and header/trailer HTML statements by updating system mail templates. To take advantage of this new interface, you must first ensure that the "Notebook=" options for your list are compatible with the WWW interface. In most cases, you will not have to do anything, but certain options are incompatible with the use of the WWW interface and may need to be changed: - The archive frequency must be WEEKLY, MONTHLY or YEARLY. SEPARATE and SINGLE notebooks are not supported. L-Soft generally recommends converting lists with SINGLE notebooks to YEARLY unless there is a compelling reason to have all the messages in exactly one file. - For optimal performance, the archive frequency may need to be adjusted to produce an "adequate" number of topics and messages in each archival period. The definition of "adequate" depends on your users, the kind of equipment they have, and how they connect to the Internet. As a rule, home users will prefer a larger number of smaller archives whereas office users with large screens and T1 or better connectivity will tolerate a larger table of contents. - The archives must be public as there is no userid/password control in the web archive interface. - On most systems, the directory in which your list archives are kept must be specified in absolute rather than symbolic form, or the WWW interface may not be able to access it. Symbolic form is when the directory name is a single letter, for instance "Notebook= Yes,L, Monthly,Public" ("L" is the directory specification). In most cases, your list header will read "Notebook=Yes,E:\LISTS\XYZ-L,Monthly,Public" and you will not have to worry about this. You must then ask the LISTSERV maintainer to enable your list for the WWW interface. This may require the installation of a web server and of the WWW interface code itself. You can then modify the WWW_INDEX mail template to customize the main archive page for your list. See the list owner's guide for more information on customizing mail templates. ************************************* * Usability: Improved spam detector * ************************************* The spam detector has been improved in version 1.8c, as follows: - The spam detector is now able to catch the first few copies of a new spam (in most cases). Previously, dozens of copies (out of the thousands sent by the spammer) would be let through before enough data was collected to identify the posting as a spam. - The number of spam alert messages sent to the LISTSERV maintainer has been reduced to one per offender. - The spam database is trimmed more aggressively, saving storage and processing time for sites with lots of mailing lists. - The algorithms have been generally refined and improved, making the spam detector react more promptly and more accurately. New detection techniques have been introduced. There are two major changes that list owners need to be aware of: spam quarantine and "anonymous" spam alerts. Spam quarantine --------------- One of the most arduous problems the spam detector has to face is the accurate detection of the first few copies of the spam. When the first copy reaches the first LISTSERV server worldwide, it is just a posting like any other. It will take repeated occurrences of this same posting for LISTSERV to realize that it is in fact a spam. However, it is desirable to block this very first copy as well, and this can only be accomplished by introducing a delay in the processing of "suspicious" messages. This "quarantine" gives the spam detector some time to gather the necessary evidence to determine if the message is a spam or not. The default value is 10 minutes, and can be changed by adding: * Loopcheck= Spam-Delay(xxx) in the list header (the value is in minutes). The LISTSERV maintainer can also change the system default by adding a SPAM_DELAY variable to the LISTSERV configuration with the desired value (also in minutes). A value of zero disables this new feature. The default value of 10 minutes is adequate in most cases; it can be lowered on fast, large, active servers and may need to be increased on servers with chronical backlogs. Currently, LISTSERV determines whether a message is suspicious or not based on the sender's posting history. This however may be changed in future versions to further improve the efficiency of the spam detector. "Anonymous" spam alerts ----------------------- On occasion, you may receive a spam alert from LISTSERV where the offender's e-mail address is replaced with the word "anonymous". These alerts are generated by new detection algorithms where, for various reasons, it may sometimes be desirable to hide the identity of the potential offender, usually because there is a fair chance that the posting is in fact legitimate for the particular lists to which it was posted (for instance because these lists were configured to tolerate a high degree of cross-posting). In this case, information about the text of the message may be released and ultimately lead to a spamming alert that will block further copies of this same message, while the identity of the poster remains hidden. **************************************** * Usability: Spamming counter-measures * **************************************** The spamming industry has changed radically since the inception of the LISTSERV spam detector in 1Q95. While basement spam companies remain mostly unchanged, a number of larger companies have emerged, with enough financing to purchase their own high-end servers and T1 or better Internet connectivity. Instead of targeting mailing lists and usenet groups, these companies use their own hardware and leased lines to send mass-mailings directly to the users' mailboxes. The LISTSERV spam detector cannot filter these messages because they do not go through LISTSERV at all. The e-mail addresses, however, are often obtained from mailing lists. A number of changes have been introduced in version 1.8c to make it more difficult for spammers to collect information about existing lists or e-mail addresses. Other changes may need to be introduced by the list owner, possibly after a discussion among the members of the list; these are described in this section as well. LIST GLOBAL now requires a search string ---------------------------------------- The most popular source of list information among spammers, at least where LISTSERV is concerned, was the LIST GLOBAL command, which returned a conveniently formatted list of all the public LISTSERV lists. With over 10,000 public lists (DEC96), this file is now over 1M in size and exceeds the message size limits of most Internet providers. It is too large to be printed and read sequentially, and most requestors do not realize that the file they are ordering is going to be that large. By April 1996, the postmaster at LISTSERV.NET was receiving several complaints a day about the "bad surprise" of having had one's mailbox crammed with "this useless junk" and losing important mail due to a "mailbox full" error. The LIST GLOBAL command was then changed (initially only on LISTSERV.NET) to require a search string of at least three characters, and since then only about a couple dozen complaints have been received from users who wanted the full listing. About half of them later turned out to be spammers. By and large, the users reacted positively to this change, which has been incorporated in version 1.8c. Note that the list of lists can also be searched interactively through the CataList(SM) search engine, available (among others) at: http://www.lsoft.com/lists/listref.html The CataList actually offers more information than LIST GLOBAL, being based on the full LISTS database (of which LIST GLOBAL is an excerpt). The CataList engine has built-in restrictions to prevent abusive use. LISTS command now restricted to local users ------------------------------------------- Another popular method for collecting information about available lists was to send a LISTS commands to individual LISTSERV (and non-LISTSERV) sites. While this required more programming work for the spammer, it also gave better results as the LISTS command would also list local lists ("Confidential= Service"), which outnumber public lists 3:1. The LISTS command has been changed in version 1.8c to only return information to local users, ie users whose hostname matches one of the patterns in the LOCAL configuration variable. Typically, if the server is running on a machine called LISTSERV.XYZ.EDU, the LOCAL variable will have been defined as *.XYZ.EDU. This allows any user in the XYZ.EDU domain to send a LISTS command to the server, while preventing spammers from getting any useful information. To clarify: - This change has no impact on local users, who can still access the same information as before. It may be necessary for the LISTSERV maintainer to review the setting of the LOCAL configuration variable to make sure that all local domains are properly identified. - This change prevents non-local users from obtaining information about local ("Confidential= Service") lists. This change will completely protect local lists from spam (both direct and indirect) as there will be no way for the spammers to get information about them. - "Confidential= Yes" lists remain completely hidden as before. - Non-local users must now use the LIST GLOBAL command or the CataList to search for public lists of interest. The LIST GLOBAL command was also changed so that you can no longer ask for a list of all the lists hosted on a particular server, as this would defeat the purpose of the LISTS command change. "Review=" now defaults to "Private" ----------------------------------- The default setting for the "Review=" command was changed from "Public" to "Private", to make it more difficult for spammers to retrieve subscriber e-mail addresses. The list owner may of course override this change by explicitly adding a "Review=" keyword to the list header. L-Soft recommends reviewing the setting of the "Review=" keyword to determine whether it adequately meets your spam-protection needs. If the list is confidential (or at least local, ie "Confidential= Service"), "Review= Private" ought to be sufficient to prevent the addresses from being "stolen" by spammers, as it is very difficult to find out about the existence of the list. For a public list, "Review= Owner" might be more appropriate. Compilation copyright --------------------- The LIST GLOBAL command now inserts a copyright statement at the top of the returned listing, as follows: ------------------------------------------------------------------------- Copyright 1996 L-Soft international, Inc. L-Soft international, Inc. owns the copyright to this compilation of Internet mailing lists (the "Compilation") and hereby grants you the right to copy the enclosed information for the sole purpose of identifying, locating and subscribing to mailing lists of interest. Any other usages of the Compilation, including, without limitation, solicitation, tele-marketing, "spamming", "mail-bombing" and "spoofing" are strictly prohibited. ------------------------------------------------------------------------- A "compilation copyright" is a copyright that applies exclusively to the compilation (directory, list, etc) on which the copyright is being asserted, as opposed to the works listed in the compilation. For instance, you could decide to make a list of all the records published by pop singers born in Alaska from 1963 to 1964. You would then be allowed to assert a copyright on this compilation, so that nobody could use it without your permission. This does not, of course, mean that you own the rights to any of the records in question, or that they may not appear in other people's compilations of pop records. It just means that people cannot copy your compilation without your permission. Similarly, L-Soft does not claim any ownership rights to any of the lists present in LIST GLOBAL. The copyright applies solely to the information that the server is returning when you send the LIST GLOBAL command. Without a copyright statement, anyone was able to use the LIST GLOBAL data for any purpose, including spamming, mail-bombing, subscription spoofing, and so forth. Many of these activities, while harmful to the Internet community, are arguably not unlawful under many jurisdictions. At any rate, the use of the LIST GLOBAL data in conjunction with these activities would not, in itself, have broken any laws. Consequently, people would be allowed to sell the LIST GLOBAL data in a variety of formats adapted to, say, popular spamming programs, and this would have constituted a perfectly legal (and thus unstoppable) activity. The copyright statement makes it possible to challenge the use of the LIST GLOBAL data independently from the legality of activities such as spamming or subscription spoofing. Conclusion ---------- The spamming counter-measures introduced in version 1.8c will provide the following benefits to LISTSERV users: - Local ("Confidential= Service") lists will become totally immune to both direct and traditional spamming, as it will be impossible for spammers to find out about their existence through programmable methods. Note that local lists whose existence became known to spammers prior to the installation of 1.8c may remain exposed. However, all new lists will be protected. - Public lists (especially those created after the installation of 1.8c) will become much more resistant to spamming attacks, as it will be very difficult for the spammers to gain knowledge of their existence. The increased "Review=" protection will also make it harder for them to collect membership data. The dynamics of the spamming industry are simple: reach as many people as possible, at a minimal cost. Direct mailings were not attempted until the LISTSERV spam filter and the usenet "cancelbots" significantly reduced the number of people who could be reached with the much cheaper list/news spamming methods. The counter-measures introduced in version 1.8c will significantly increase the spammers' costs for collecting membership lists or other information about LISTSERV lists (and especially local lists), making other targets comparatively attractive. Spamming is only cost effective when the cost of acquisition of a particular e-mail address is significantly less than with traditional (mundane) mailing list brokering. Hiring people to surf the web in search of local lists, then pose as an interested party and try to talk the list owner into being allowed to join the list is a slow, unreliable and expensive process - there are over 30,000 local LISTSERV lists with an average of only 75 subscribers each. Running a simple program on a standard full usenet feed, on the other hand, is and will remain a very cost effective way to acquire massive amounts of e-mail addresses in a very short time. It is unlikely that spammers will hire armies of "list investigators" to engage in arguably fraudulous misrepresentations to collect a couple million e-mail addresses when they can get better results legally with a $2000 PC reading usenet news 24h a day. *************************************************************** * Usability: MIME digests and other MIME-related enhancements * *************************************************************** Version 1.8c introduces support for MIME digests, that is, list digests formatted according to the specifications of the Internet MIME standards (RFC1521/1522 et seq). These digests coexist with the traditional (RFC1153) LISTSERV digests. Individual subscribers can select which type of digest they wish to receive, and the list owner can set a default for the list. In addition, when the list owner did not indicate any preference through the "Default-Option=" keyword, LISTSERV automatically activates MIME digests for SUBSCRIBE requests received from users with MIME-capable mail readers. This effectively makes MIME digests the default option while ensuring that users who do not have a MIME-capable mail reader are not sent a digest that they cannot decode in a convenient way. The advantages of MIME digests over traditional ones are: - Full support for all MIME enhancements: national/8-bit characters, file transmission, multimedia attachments, etc. - Advanced mail client support (in some mail products) for the digest function. Typically, opening the digest will display a table of contents showing all the available messages in the digest. Clicking on one of the messages will then open it in a separate window. More advanced functions (threading, sorting, etc) may be available in some mail clients. The disadvantages are: - The MIME digest format is harder to read when using a mail client which has no special support for MIME digests. In this case, it is usually preferable to switch back to the traditional digest format (see below). - Some users actually dislike the way their mail client provides specific support for mail digests. Some clients, for instance, immediately burst MIME digests into a collection of messages, which are entered individually into the user's in-box. While this may seem to be a good idea at first, it means that discarding a digest may actually require hundreds of messages to be discarded individually. Users who would rather continue to receive traditional digests can send a SET * NOMIME command to LISTSERV to indicate their preference for traditional digests for all the lists they are currently subscribed to. This command does not have any effect on regular (MAIL) or INDEX subscriptions and thus can be safely wildcarded. Similarly, users who subscribed prior to the installation of 1.8c but would rather receive MIME digests can do so with a SET * MIME command. Again, this only affects digest subscriptions and will not cause non-digest subscriptions to switch to digested mode. List owners can add "Default-Options= MIME" to the list header to select MIME digests as the default digest option for all new subscribers. As usual, this keyword has no effect on existing subscribers (use SET listname MIME FOR *@* to change existing subscriptions). Please note that the MIME option only indicates a preference in the event that the users should switch to digest mode on their own. Use "Default-Options= MIME,DIGEST" to add all new subscribers with MIME digests (as opposed to MAIL or INDEX). Conversely, "Default-Options= NOMIME" indicates that MIME digests should only be activated when explicitly requested by the user (through the SET listname MIME command). While currently its only effect is to indicate a preference for MIME vs traditional digests, the MIME subscription option may, in a future version, control new MIME-related functionality. Finally, MIME fields are now always written to the list archives, even when "Notebook-Header= Short" is in effect. **************************************** * Usability: Support for "super-lists" * **************************************** In order to facilitate the organization and management of hierarchical mailing list structures (such as lists mirroring the internal organization of a business unit), version 1.8c introduces support for "super-lists" ("super" as in the opposite of a "sub-list"). A super-list is a container list that automatically includes all the subscribers in a pre-defined set of sub-lists, recursively. Super-lists can be created hierarchically to any depth. The LISTSERV administrator creates a super-list in the same manner as any other list, and inserts a "Sub-lists=" keyword in the list header to identify the list as a super-list and provide a list of the sub-lists whose membership information should be inherited automatically. All the sub-lists in question must be on the same machine as the super-list. For security reasons, only the LISTSERV administrator is allowed to add or modify the "Sub-lists=" keyword. In most respects, a super-list behaves exactly like an ordinary list. It has its own list owner, its own list archives, its own set of list header keywords, and so forth. It is even possible to subscribe to the super-list directly, assuming the "Subscription=" keyword allows it. The only difference between a super-list and a regular list is what happens when you post to it. With the super-list, the membership of all the sub-lists is added (recursively) to the list of people who have subscribed to the super-list itself, and duplicates are suppressed. The posting is then processed normally, archived based on the "Notebook=" keyword in the super-list, and so forth. As noted above, users can subscribe to the super-list directly, and this is in fact an important aspect of the operation of super-lists. If you are subscribed to the super-list itself, the subscription options used to deliver "super-messages" to you are taken from your subscription to the super-list, just like with any other list. For instance, if you are subscribed to the super-list with the NOMAIL option, you will not receive any super-message, even if you are also subscribed to one of the sub-lists without the NOMAIL option. By subscribing directly to the super-list and setting the NOMAIL option, you are indicating that you do not wish to receive any of the messages posted to the super-list and explicitly overriding any subscription options in the sub-lists. When you are subscribed to multiple sub-lists, but not to the super-list itself, LISTSERV uses the following rules to determine which subscription options should be used for super-messages: 1. NOMAIL subscriptions are ignored. You will get the super-message if you have an active (non-NOMAIL) subscription to at least one sub-list. The rationale for this rule is that posting to the super-list should be equivalent to posting to each and every sub-list, minus the duplicates. When posting to all the sub-lists, a single non-NOMAIL subscription would be sufficient for the user to receive the message, and consequently the message should be delivered when posted to the super-list. The only way not to receive postings from the super-list is to subscribe to the super-list directly and set yourself to NOMAIL. 2. The DIGEST and INDEX options are ignored and internally converted to MAIL. The rationale is as follows: - In most cases, super-lists will be used for administrative messages or for announcements, rather than for active, ongoing discussions, which would typically be carried out on the smaller sub-lists. Thus, it seems improper to inherit the DIGEST or INDEX options from a high-volume discussion sub-list and apply it to a low-volume announcement super-list. - This rule can be applied consistently regardless of the way the super-list is configured. If the DIGEST or INDEX options were to be retained when possible, there would still be cases where they would have to be changed to NOMAIL because the super-list does not have list archives or does not allow digests. This lack of consistency would make it more difficult for non-technical users to work with super-lists. - Similarly, super-lists will normally be deployed to avoid duplicate messages, ie in situations where users are subscribed to multiple sub-lists, possibly with different options for each list. Inheriting the DIGEST and INDEX options would require additional rules to control the behaviour of the super-list in the presence of conflicting subscription options (sub-list A set to DIGEST and sub-list B set to INDEX or MAIL). To receive super-messages in DIGEST or INDEX format, the users must subscribe to the super-list directly and set the appropriate option. Topics, if defined, are evaluated on a per-list basis. That is, for every sub-list (and for the super-list as well, if appropriate), LISTSERV determines whether the topic of the message is one that the user wanted to see. If not, it acts as if the user were not subscribed to this particular list. This mechanism works very well if all the sub-lists have the same set of topics (or at least a well-defined set of common topics). Collections of sub-lists with widely different sets of topics should be avoided as there is no reasonable way to make super-list topics work in this context. Please refer to the 1.8c list owner's guide for more information about super-lists. ************************************************* * Usability: New SUBJECTHDR subscription option * ************************************************* The new SUBJECTHDR subscription option directs LISTSERV to add the name of the list in the "Subject:" field of every posting coming from the list: Subject: [XYZ-L] Question about ink mixing With some mail programs, this can make it much easier to sort messages coming from the list, for instance by using special filters to save all the messages from a particular list to a designated folder. Replies are handled automatically to avoid inserting multiple copies of the list name. Like all other subscription options, SUBJECTHDR can be turned on or off by individual subscribers, or by the list owner on behalf of any (or all) subscribers. The list owner can also define a default value using the "Default-Options=" keyword. By default, LISTSERV will insert the name of the list in the subject field. In some cases, this may not be the most appropriate solution, especially if the list name is very long. The list owner can change the text of the insert using the "Subject-Tag=" keyword: * Subject-Tag= FE This is particularly useful for groups of related mailing lists. ********************************************** * Usability: Support for parallel moderation * ********************************************** In some cases, it may be useful to have a team of moderators working "in parallel" on the same messages, for instance when it is critical that the messages be approved or rejected as soon as possible. In this case, it is counter-productive to have LISTSERV assign each message to a moderator in the usual round-robin fashion; instead, all the moderators need to be sent a copy of the message, and each of them should be able to approve any message, without however causing duplicate copies to be posted to the list. This is now possible, using the following keyword syntax: * Send= Editor,Hold * Moderator= All,moderator1,moderator2,... If the moderator list starts with the word "All", approval requests for incoming messages are sent to all the listed moderators. Any moderator can then approve the message using the normal OK confirmation procedure. If several moderators attempt to approve the same message, the first one will succeed and subsequent requests will fail, stating that the message was not found (because it was deleted after being approved and posted). It is also possible to use "Moderator= All" without "Send= ...,Hold", allowing the moderators to edit the messages before reposting them. In this case, however, a separate synchronization mechanism needs to be used to avoid duplicates, as two moderators are unlikely to make exactly the same edits to the message. Even if LISTSERV were able to identify the two submissions as being the same message, it would not know which one to choose over the other. ************************************************** * Usability: Support for anonymous subscriptions * ************************************************** The SUBSCRIBE command has been enhanced to support the following syntax: SUBSCRIBE listname ANONYMOUS (as usual, the command is not case sensitive) This indicates that the subscriber wishes to join the list anonymously, ie without specifying a name. The CONCEAL subscription option is automatically set, granting the subscriber the maximal level of protection available. An existing subscription can be made anonymous by sending a SUBSCRIBE listname ANONYMOUS command. This will both delete the name from the list and set the CONCEAL option. An anonymous subscription can be made non anonymous by sending a SET listname NOCONCEAL command and a SUBSCRIBE command with the desired name. In addition, when a list operates with "Default-Options= CONCEAL", LISTSERV no longer requires or expects a name to be specified on the SUBSCRIBE command. If a name is provided, it is accepted and entered in the list file, as before. If, however, no name was supplied either on the SUBSCRIBE command or in the mail headers, LISTSERV accepts the subscription (instead of replying with a description of the expected syntax) and makes it anonymous. ***************************************************** * Usability: Per-subscriber daily message threshold * ***************************************************** The syntax of the "Daily-Threshold=" keyword has been extended to allow the specification of a second, per-subscriber daily message threshold. For instance, * Daily-Threshold= 100,10 indicates that subscribers are allowed to post up to 10 messages per day each. The 11th and following messages are returned to the sender, who will have to resend them on the following day in order to have them posted. The list owner and editors are exempt from this quota. The default per-subscriber message quota is infinite. The implementation of the global daily threshold (100 messages in the present example) is unchanged from the previous version: the 101th and following messages are queued instead of being returned, and will be automatically reprocessed when the list owner sends a FREE command. The decision to return messages exceeding the per-user threshold (instead of queuing them) was based on the design objectives for this new function, which were to provide a simple mechanism to educate outspoken subscribers and make them understand the need to keep their contributions to a reasonable volume. Queuing the messages would have encouraged a "It will go through when it goes through" attitude and ultimately resulted in a situation where the postings from hyperactive subscribers are distributed several days after they are written, confusing the other subscribers and raising questions about unexplained mail delays and problems with the host system. Returning the message while requiring a retransmission, on the other hand, provides instant feedback to the hyperactive user and requires a conscious choice as to which postings should be resent the next day. ***************************************** * Usability: Miscellaneous enhancements * ***************************************** - By default, the bottom banner is no longer appended to every single message when composing a digest. Instead, a single copy is included in the digest preamble. Please note that some digestification programs and mail readers may not show the digest preamble at all, and that frequent readers may skip the preamble out of habit. In general, there is no guarantee that the bottom banner will be read at all unless it is included on each and every message in the digest. To revert to the previous behaviour (bottom banner on each and every message), add the special "Bottom_banner" option to your "Digest=" keyword, as in: * Digest= Yes,Monthly,Bottom_banner The top banner is always included on each and every message as it is normally used to insert a one-line copyright statement. - LISTSERV now decodes MIME messages sent to the LISTSERV address (but not those sent to the list addresses) before processing them. In most cases, this will allow correct processing of LISTSERV requests which used to be rejected by 1.8b, for instance because they were base64 encoded. In some cases, however, it can lead to LISTSERV rejecting MIME messages with proprietary text-like formats when the requests previously happened to work. LISTSERV requests should be sent in plain text format whenever possible. - "OK"-style confirmation messages can now be tailored, on a list per list basis, using the new CONFIRM1 mail template. - LISTSERV now keeps track of subscription dates. This information is displayed by the QUERY command, when it is available. Since users who subscribed before the installation of 1.8c do not have any known subscription date, this information is not always present. - When parsing list headers, LISTSERV now recognizes e-mail addresses and processes them using a slightly different parser, which preserves the case of the left-hand side of the address and, for BITNET sites, removes any superfluous '.BITNET' suffix on the right-hand side. Thus, it is no longer necessary to use double quotes to register a lower-case address in the list header. - The mail template processor now supports HTML-like variable closure, in addition to the traditional LISTSERV closure (both methods are supported concurrently; there is no need to select one over the other). Here are examples with traditional and HTML closure: Traditional: For more information, please send mail to &EMAIL or call &PHONE. HTML: For more information, please send mail to &EMAIL; or call &PHONE;. Previously, HTML writers who used HTML closure conventions would not get the expected results. This change makes it easier for webmasters to get the desired results the first time. - New mail template variables: &LITE, &ISODATE, &DAYSEQ(n). &LITE has the value 1 when running the new LISTSERV Lite product, and 0 otherwise. This variable can be used to write generic templates that account for the differences between the two products. &ISODATE returns today's date in ISO format, ie yyyy-mm-dd. &DAYSEQ(n) is used to create FAQ templates with rotating topics and is described in the 1.8c list owner's guide. - The "Renewal=" keyword now supports "xx-Daily" as a valid interval specification. While the use of intervals of less than a week is and remains inadvisable, FAQ templates with rotating topics may require the selection of a very precise renewal interval (for congruence purposes), which was not possibly with "xx-Weekly" granularity. Please refer to the list owner's guide for a discussion of rotating FAQ support. - The QUERY ... WITH command now supports topic selection. Examples: QUERY XYZ-L FOR *@* WITH TOPICS: ADMIN FORUM -> Shows all members subscribed to both ADMIN and FORUM topics. QUERY XYZ-L FOR *@* WITH TOPICS: -ADMIN -> Shows all members not subscribed to the ADMIN topic. QUERY XYZ-L FOR *@* WITH TOPICS: ADMIN -TEST -> Shows all members subscribed to the ADMIN topic, but not to the TEST topic. - The "List-Address=" keyword now allows the 'username' part (ie the name of the list) to be redefined. Previously, only the host name could be changed. It is important to note that the only effect of the "List-Address=" keyword is to change the way the list identifies itself in list postings, command replies, etc. It does not instruct the mail system to create forwarding entries to support the new name, nor does it establish the specified name as an alias for the list (use "List-ID=" for this purpose). In general, you should not use this keyword without first consulting with the LISTSERV maintainer. - LISTSERV now requires confirmation (through the "OK" mechanism) when commands are sent through a WWW browser, even if they apply to a list whose security level or "Subscription=" keyword does not require confirmation. This change was made both to avoid abuse and because many occasional WWW users do not know their e-mail address or enter it incorrectly, whereas people who use e-mail regularly do of course have a working e-mail address in their configuration. ------------------------------------------------------------------------- CataList is a service mark of L-Soft international, Inc. LISTSERV is a registered trademark licensed to L-Soft international, Inc. LSMTP is a trademark of L-Soft international, Inc. LMAIL and L-SOFT are trademarks of L-Soft international. PMDF is a registered trademark of Innosoft International, Inc. Unix is a registered trademark of X/Open Company Limited. DEC and VMS are trademarks of Digital Equipment Corporation. Windows is a trademark of Microsoft corporation. All other trademarks, both marked and not marked, are the property of their respective owners. -------------------------------------------------------------------------