Q: How can I run a fully automated A/B-split campaign with LISTSERV Maestro?

By Robert Graf-Waczenski
Senior Applications Programmer, L-Soft

Sending an A/B-split job to your current subscribers and keeping the campaign running so that new subscribers also receive it can seem like a challenging task. This tip shows how easy it is to use A/B-split testing with sampling and auto-repeat functionalities to ensure that new subscribers automatically get the most popular version of your content.

Creating the A/B-Split Job

To create an A/B-split job, select the option shown above from the menu. The following screen opens:

Choose the "A/B-Split Job with Sampling" option and enter a meaningful title for the campaign. In our example, we are announcing a new recurring event on Fridays not only to our existing subscriber base, but we also want future subscribers to automatically get the same message.

This results in the following summary screen:

As you see, "Define Recipients" is highlighted as the next step in the workflow. Completing this step is as straightforward as completing the recipient definition of a standard job. Simply open your subscriber warehouse, select the list to which you want to send your campaign and continue to the next step.

Defining the Sampling Variant Jobs

With the recipient defintion step completed, you can now proceed to define the variant jobs.

Click on "Define Variant Jobs". The following screen opens:

This screen shows that the two sampling variants are not yet defined completely. In our example, two variant jobs are enough because our content authors provided us with two layout variants to test for our announcement message.

Click on any of the links above to open the workflow screen of the associated sampling variant:

As you see, the workflow for the definition of a sampling variant of an A/B-split job is shorter than the workflow of a standard mail job, which makes defining the sampling variants even easier. The workflow for a sampling variant job ends with the "Send Test Emails" step. Once this step is completed, you are routed back to this screen:

Once you have completed the definition of all sampling variants, the screen looks like this:

Scheduling and Authorizing the Sampling Delivery

When you are satisfied with the sampling variant jobs, the "Schedule Delivery" step comes next in the A/B-split workflow:

Click on the "Schedule Delivery" step to open the following screen:

Since we decided to send all sampling variants at the same time, this means that we only have to enter a single date and time here. Click "OK" to complete the step. You are now ready to authorize the delivery of the sampling variants:

Note that authorizing the sampling delivery means that all variant jobs are authorized in one step, which in our example means that LISTSERV Maestro initiates delivery of both sampling variants the moment the "Authorize Sampling Delivery Now" button is clicked.

The Main Job

We have now authorized the delivery of the sampling variants. Depending on your decision for the sampling schedule, this may in fact mean that messages are already being delivered to the random sampling subset of your subscribers. But even if you scheduled the sampling delivery for the future, you can continue defining the settings for the main job and authorizing it, which is what we are describing in this section.

The job summary now looks like this:

In our example, selecting the winner by its open rate was the only choice available and has, therefore, been selected automatically by LISTSERV Maestro. If tracked links or share icon clicks are available in your sampling variants, then you can decide to use any of those metrics for LISTSERV Maestro to select the winning variant.

Scheduling the delivery of the main job is the step that naturally requires the most attention. The screen is separated into two parts:

This first part of the settings deals with the initial delivery schedule, separated into the configurable event collection time and the choice when to perform the delivery of the main job. See the on-screen help for more information. In our example, we will use "Send Immediately", which means that the best performing variant will be sent to the rest of your subscribers as soon as the 2-day event collection period for the sampling variants has ended.

The second part of the settings is initially not shown. Click the link at the bottom to show the advanced scheduling options:

With options configured like this, our main job will not only be delivered to the subscribers that are on our list now, but LISTSERV Maestro will repeat the delivery every week. With "Prevent Duplicate Messages" selected, this means that subscribers that already exist will receive the message with the initial delivery only, while new subscribers (who have not yet received the message in previous deliveries) will receive the message during the next delivery.

This yields the behavior that we intended, so we are ready to authorize the main job delivery:

Click on "Authorize Main Job Delivery" to open the authorization confirmation screen:

Clicking the "Authorize Main Job Delivery Now" button is the final step in the A/B-split job workflow.

Note that A/B-split jobs in LISTSERV Maestro can be configured in many different ways to accommodate the considerable variety of possible A/B testing use cases. You can, for example, very easily have more than two sampling variants, and it's also possible to use delivery triggers to initiate the sampling delivery and the final main job delivery from a process external to LISTSERV Maestro.

You can also configure the sampling variants to have individual delivery schedules. This can be useful if you want to test different delivery days or times of the day.

The additional configuration options are described in more detail in the online help, which is available by clicking the question mark icon in the top-right corner of each of the screens and selecting "Help About Current Page".

Next Steps