Q: How can I track my subscriber activity in LISTSERV?

By Jacob Haller
Senior Support Engineer, L-Soft

List owners often want details on how the use of their mailing lists has changed over time. Have more people subscribed to the list this year than last year? When was the last time someone posted to the list? Did that controversial post last month result in people leaving the list? Did adding a link on my main web page cause an increase in new subscribers? Did new spam blocking result in mass auto-deletions over the past six months?

The changelog feature in LISTSERV is designed to answer exactly these sorts of questions. Changelogs keep track of all activities on a LISTSERV list, including postings, subscribes, unsubscribes, address changes and auto-deletions. They allow the list owner to track such activities in great detail without having to do manual bookkeeping and without having to ask the site administrator for the main LISTSERV logs.

Adding a changelog to your LISTSERV list can be as simple as adding a single line to your list configuration:

• Change-log= Yes

Adding the keyword as above will cause LISTSERV to create a listname.CHANGELOG file in the main LISTSERV directory (\LISTSERV\MAIN on Windows, ~listserv/home on Unix). From the time that you enable the changelogs, LISTSERV will begin recording all list activities. Changelogs are not retroactive; they collect new data but cannot provide information from before the feature was enabled.

The easiest way to access the reporting data contained in the changelogs is through the LISTSERV web interface. Under the "List Management" menu, select "List Reports > List Activity Reports". Then select the list for reporting from the "Select List" drop-down. There are two types of reports: History Reports and Statistics Reports. The history report shows the full history for the selected time period. For example, we could request a history of all auto-deletions for the month of February, and LISTSERV would return a line for each Auto-Delete, showing the address that was deleted, the date and time of the deletion and the reason for the deletion:

The statistics report returns summary information. For example, we might request a monthly interval report of all auto-deletions for 2017. In that case, LISTSERV would return one line per month, showing the number of deletions for each month:

These reports can also be automatically downloaded in CSV format for easy import into a spreadsheet program. Under the "Report Format" drop-down, you should find the options Web Site, History in CSV Format, or Stats in CSV Format. Running one of the CSV reports will prompt the browser to download the CSV file and either save it or open it in the spreadsheet program of our choice.

If you have experience writing scripts, then it is also relatively easy to write our own program to parse the changelog files directly. The *.changelog files are in plain text and have an easy-to-parse format:

The file is space-delimited. The first field is a timestamp; the second shows the action type; the third shows the subscriber email address for the action; and the remainder of the line depends on the action type. For POST events, the line shows the subject line of the message. For SUBSCRIBE/ADD commands, it shows the name field. For AUTODEL events, it contains the status message from the non-delivery notification. Using a fairly simple script, we could easily build our own list activity reports by working directly with the changelog files.

If LISTSERV is configured for DBMS access, it is also possible to write the changelog data directly to a database table. This is done by setting the CHANGELOG_DBMS, CHANGELOG_DBMS_CONNECTION and CHANGELOG_DBMS_TABLE site configuration parameters.

Additional information about changelogs can be found in the LISTSERV Site Manager's Manual, and information about storing changelog information in a DBMS can also be found there.

Rotating the Changelogs

Since the default is for all changelog entries to be stored in a single file, customers may be concerned about the difficulty in getting rid of old entries, particularly if the server that LISTSERV is installed on is tight on disk space. You can therefore configure your mailing list to automatically start a new changelog by specifying an interval. For example, to rotate the changelog monthly, you would use:

• Change-log= Yes,Monthly

The result of this is that, at the end of the month, LISTSERV renames listname.changelog file to listname.changelog-YYYYMM (where 'YYYY' is the year and 'MM' the month). You then end up with a separate changelog file for each (in this case) month of data. Changelogs may be rotated yearly, monthly or weekly.

Note, however, that the List Activity Report section of the web interface only looks at the current changelog file, so if a mailing list is configured to rotate its changelogs yearly, the list activity report won't be able to find any information about activity before January 1 of the present year. Information about earlier activity will have to be extracted directly from the rotated changelog files.

GDPR Concerns

The EU's General Data Protection Regulation requires, among other things, that a company must be able to provide on demand a report in a common machine-readable format (such as XML) which lists every instance of a customer's personal data held by that company. That includes (but is not limited to) data stored in changelogs.

In response, L-Soft has developed a PowerShell script, which, using either the LCMD.EXE or LCMDX.EXE command interfaces that ship with the Windows version of LISTSERV, can pull the relevant data using standard LISTSERV commands and produce an XML report containing the results. While the script itself is Windows-specific, by using the LCMDX.EXE option (which communicates directly with the server's TCPGUI port), it is possible to generate reports from any Unix-based LISTSERV site as well, as long as the site has the LISTSERV web interface enabled.

The script also works under PowerShell Core 6.1, making it possible to run the script from Linux and MacOS workstations by using the Unix version of 'lcmdx'.

You can get this script, and instructions on how to use it, at:
http://download.lsoft.com/downloads/gdprscan/gdprscan.zip

Subscribe to LISTSERV at Work.