|
Tech Tip (LISTSERV Maestro) – Issue 3 – 2007
Q: How do I perform A/B-split testing with LISTSERV Maestro?
Answer by Holly Taylor
Technical Writer, L-Soft
LISTSERV Maestro 3.1 offers two types of A/B-split testing jobs – a normal A/B-split job and an A/B-split job with sampling. This tech tip will discuss the difference between the two types and how to best use them to help you achieve a higher success rate with your messages.
An A/B-Split Job lets you define a list of recipients with several different message content and tracking definitions in the form of variant jobs, which are simply sub-jobs of the A/B-split job. During delivery, the list of recipients will be evenly split between all of the variant jobs, and each variant job will then be delivered to its recipient share. Then, the tracking results can later be used to assess the "success" of each variant job in comparison to the other variants.
An A/B-Split Job with Sampling is a special version of a normal A/B-split job. It contains two types of job categories – Sampling Variants and a Main Variant – and, together, they are the sub-jobs of the A/B-split job. The bulk of the recipients is reserved for the main variant, while only a smaller part of the recipients is evenly divided among the sampling variants. The sampling variants are authorized and delivered first, while the main variant is held back. The tracking results of the sampling variants are then used to assess the "success" of each variant (just like a normal A/B-split job). With these results, you can copy the content of the most successful sampling variant into the main variant (maybe with some additional changes).
Defining A/B-Split Jobs
To define an A/B-split job, click on the Mail Job menu and select New. The Start New Job screen opens. In the Initial Job Settings section, select the All new job option, and then click the Job Type drop-down arrow and select A/B-Split Job.
Once you've finished entering the rest of the job's information, click [OK]. The A/B-Split Job Details screen opens. This screen lets you access the details of an A/B-split job so that you can view and edit the various job parts and variant jobs, if applicable.
Each section of this screen contains information pertinent to the job along with its status. As each section is successfully completed, a green shield is shown. To enter or change job information, simply click the Edit link associated with that section. The following sections need to be completed before delivery can be authorized:
- A/B-Split Job – This section lets you edit the job title, job category, job-ID prefix, delivery notification, "authorization due by" date, and auto-archive settings.
- Recipients – This section lets you define the recipients of the A/B-split job.
- Job List – This section lets you define the variant jobs that are part of the A/B-split job. Each variant job is displayed in a separate row, with job ID, job title, and with four icons for the four aspects that are unique to each variant job: Content, Tracking, Sender, and Test. These icons reflect the states of the corresponding workflow steps.
To edit a variant job, click on its ID. The Job Details screen opens. From here, simply define the message content, the sender, and tracking, and then test the delivery. You can change the Job Title for a variant job by selecting Job Info from the Mail Job menu. Once you're done, click on the job ID at the top of the screen to return to the A/B-Split Job Details screen.
Tip: From the Job Details screen, you can also copy the details from a different variant job. To do so, click the Mail Job menu and select Copy Settings from Other Variant.
Click the Add Variant Job link at the bottom of the list to add another variant job to the A/B-split job. In addition, a Delete link may appear next to each variant job, allowing you to delete the corresponding variant job (the link does not appear if only the minimum number of variant jobs is present in the A/B-split job).
- Delivery – This section lets you schedule the delivery of the A/B-split variant jobs to take place immediately after authorization or at a future date and time relative to a selected time zone. The delivery settings apply to all variant jobs equally, as does the authorization. This means that the A/B-split job can only be authorized when all variants have been completed (all four states of all variant jobs are complete and marked with the green checkmark), and once the A/B-split job is authorized, all variants are authorized (and will be delivered according to the delivery settings defined here).
Once the above information is completed (green icons and checkmarks will let you know when everything is completed), and you are ready to authorize delivery, then click the Mail Job menu and select Authorize Delivery.
The Authorize Delivery screen opens, presenting you with a final overview of the job. Verify that everything is correct, and then click the [Authorize Delivery Now] button.
Note: Working with and defining the A/B-split job details is very similar to working with a standard mail job. For more information, see the User's Manual for LISTSERV Maestro [1].
Defining A/B-Split Jobs with Sampling
To define an A/B-Split Job with Sampling, click on the Mail Job menu and select New. The Start New Job screen opens. In the Initial Job Settings section, select the All new job option, and then click the Job Type drop-down arrow and select A/B-Split Job with Sampling.
Once you've finished entering the rest of the job's information, click [OK]. The A/B-Split Job Details screen opens. This screen lets you access the details of an A/B-split job so that you can view and edit the various job parts and variant jobs.
The sections of this screen are the same as for normal A/B split jobs, with the following exceptions:
- A/B-Split Job – Editing this information also lets you define how many recipients will be used for the sampling job and how many will be used for the main job.
- Job List – In addition to the variant jobs that are part of the normal A/B-split job, a separate tab is available for defining the main variant of the job.
Note: The main variant can never be deleted, and the Main Variant Job tab does not contain a link to add more variants because only one main variant is permitted.
- Delivery – There are two separate delivery settings that need to be defined: Sampling Delivery, which applies to all sampling variants, and Main Delivery, which applies only to the main variant. Because of this, the delivery settings are located on both tabs: the Sampling Variant Jobs tab and the Main Variant Job tab. Similarly, the authorization of the sampling variants and the main variant happen after each other. The sampling variants have to be authorized first, and, similar to a normal A/B-split job, it is only possible if all sampling variants are complete (with green checkmarks). Once the variant delivery is authorized, all sampling variant jobs will be delivered according to the Sampling Delivery settings. Authorization of the main variant is in turn only possible after the delivery of all sampling variants has been completed (either successfully or with failure). Also, the main variant job must of course be completed first (with green checkmarks).
Once authorized, it will then be delivered according to the Main Delivery settings.
Once the above information is completed (green icons and checkmarks will let you know when everything is completed), and you are ready to authorize delivery, then click the Mail Job menu and select Authorize Sampling Variant Delivery.
The Authorize Delivery screen opens, presenting you with a final overview of the job. Verify that everything is correct, and then click the [Authorize Delivery Now] button. The sampling variant jobs are authorized and delivered first, while the main variant job is held back. You can then use the tracking results of the sampling variants to evaluate the "success" of each variant (just like a normal A/B-split job). With these results, you can copy the content of the most successful sampling variant job into the main variant. After this is done, you can authorize the delivery of the main variant.
Note: Working with and defining the A/B-split job details is very similar to working with a standard mail job. For more information, see the User's Manual for LISTSERV Maestro [2].
Evaluating the Success of the A/B-Split Testing Job
For reporting purposes, each variant job is treated as a separate job. Once the jobs have been delivered, you can use the report tab on the individual variants' Completed Job Details pages to generate quick reports for each of the variants or use the reporting tool to combine variant jobs into a single report. For more information on working with reports, see Section 12.5 and Section 13 of the User's Manual for LISTSERV Maestro [3].
References
[1] [2] [3] LISTSERV Maestro User Guide
L-Soft Sweden AB
http://www.lsoft.com/resources/manuals.asp
|
