Overview
Experiment campaigns (formerly called Multivariate Test Campaigns, or MVT Campaigns) allow users to send message variations to a subset of recipients for the purposes of testing engagement. This practice is traditionally known as A/B testing. With MessageGears Experiment Campaigns, users can test and evaluate up to 26 campaign variations with MessageGears Accelerator.
There are two types of Experiment available in Accelerator: Subject Line and Full Template
- Subject Line allows you to quickly and easily enter different message titles in your tests without any template changes. Subject lines can be personalized using FreeMarker.
- Full Template allows you to change the message appearance and uses the subject saved with each Template.
Prerequisites
There are a few considerations to keep in mind when before starting a new Experiment Campaign:
- The best experiments use a small number of variations and subtle changes within each. This approach is especially important when testing full templates.
- Although you can test the subject line or template variations, you cannot change the associated context data.
Accelerator allows you to set a specific test size among your recipients. You may elect to set either:
- A proportion of the total recipients (Percent), or
- A defined number of total recipients (Absolute)
For Subject Line variants, the Winner Selection content is locked in at the time of the initial testing launch.
For Full Template variants, the Winner Selection content is copied at the time of winner selection; any changes you make to the templates/shared content prior to the winner launch will be respected in a winner selection for Template variant Experiments.
Once an Experiment test is sent, you can then refer to the test details in your job notification and use the resulting engagement data to determine the response winner. Then, send messages to the remaining recipients using the winning content.
Table of Contents
Setting up an Experiment Campaign
Adding Settings to Experiement Campaign
Setting up an Experiment Campaign
To begin, you will either need to create a new Campaign or copy a previous Campaign. After you have created your new Campaign, navigate to Campaigns > Experiments. If you need to create a new Campaign, you can find guides on how to create them by clicking on the following link.
Once you have done this, you will have access to the Setting Page for your Campaign, this will allow you to set what the baselines will be for this Campaign, including Audiences, triggers, and templates.
Adding Settings to Experiment Campaign
After you have created and named your new Experiment Campaign, the next step will be to add some required elements to it. This section can also be used to update or add to a previously existing Experiment Campaign by clicking on the Settings section on the top-right side of the screen.
The following is a list of the required sections of the Experiment Campaign that you will need to fill out to proceed, they are Name, Template, Subject Line, and Audience.
Jacquard (Phrasee)
Jacquard is a way to optimize language within a Message and test variants. These variants are generated in Jacquard and can incorporate Campaign settings/recipient tokens to make them more relevant and personalized. You can find more information on the uses of Jacquard by clicking here.
Additional Options
It is not required but it is recommended that you also set up a Notification Email Address from the selection box at the bottom of this main section. This will allow you to receive Error Notifications from the sent Experiment Campaign. This will also allow you to select a progress update notification for your Campaign as an additional option. Additionally, it is highly recommended that the Context Data be entered from the View Advanced Options section as listed below.
Additional options can be found by expanding on the View Seedlist Options and View Advanced Options at the bottom of the page.
View Seedlist Options
- Campaign Identifier: An identifier name that is automatically generated for the Campaign separate from the main name. This can be edited but it is not necessary.
- Select a Seedlist: Seedlists are email addresses automatically added to a recipient list for the purpose of testing and monitoring the campaign’s content rendering and deliverability across different email providers and content filters.
View Advanced Options
- URL Append: Adds additional tracking information to URLs for internal analytics
- Context Data: Data that isn't specific to individual recipients
- Pre-Campaign Trigger: The database trigger that is executed when the Campaign is about to launch
- Post-Campaign Trigger: The database trigger that is executed when the Campaign is has finished
- Audience Recording: This recording dictates where the extracted customer list will be inserted.
Be sure to Save your information before proceeding further.
Running a Test
After the selections of settings are filled out, you can start a new test. (from the settings page or the overview page). This will allow you to take those settings and run a variety of tests against the Subject Line, the Full Template, or the Jacquard Subject Line. You can then select up to 26 different variants to test against each other. You can begin this process by clicking the Start New Test button in the top-right corner.
Adding Variants
Now is when you will start to add the variants for your Experiment Campaign. Select the Subject line for each Variant that has been created and give it a different title.
Launching Experiment Campaign
You will be given access to the selection screen that will allow you to select the distribution numbers. You can either schedule this to run immediately or on a future date. This is where you may elect to set either a proportion of total recipients (Percent) or a defined number of total recipients (Absolute).
You now have the choice to set the Experiment Campaign to launch Immediately or on a Scheduled Date. If you opt to set a Scheduled Date, this will allow you to set a Recipient Extract Ahead timeframe for your Experiment Campaign. What this means is that you can select an exact time for the report of the completed test to be sent to you. However, it is very important to remember that it is dependent on you to estimate the necessary amount of time to set up leading up to the scheduled extraction time, so be sure to take this into account so that the system will have enough time to process your request.
Below you will find examples of both the Scheduled and Immediate Launch Screens:
Scheduled Launch Screen
Immediate Launch Screen
You will now need to select either Manual Launch or Auto Launch. Should you choose to Auto Launch Winner during this, you will need to select the "Winner" on this page as well. You have the options of Click Rate, Adjusted Open Rate, and Open Rate (vs) to select, and each one will give you specific options for how many hours after the campaign is completed to run. If you need a refresher, the following list will show you what your selections will be testing for.
Click Rate: The number of customers who clicked a link in your email (after opening it).
Adjusted Open Rate: This is an estimate of email opens performed by a verified customer (non-automated).
Open Rate: shows the number of email recipients who opened the email.
Once you have clicked on the Save & Continue button, you will be brought to the final test screen and have the option to Launch or to go Back to Scheduling.
Post-Launch Actions
After Launch, you will be brought back to the main Experiement test page where you can view a saved test, scheduled test, and active tests.
Active Test
If you open an active test, it will show you the current (winner selection ) results and metrics while it is still compiling the results. You can view this by clicking on the ID Number on the Active Test section on the lower portion of the main screen.
The test page that you are presented with will show you the current metrics of the event as it is processing.
Metrics that are included in this are compiled by the Number of messages in the job, shown on the right side, and will then give users the email percentages of the following factors:
- have been delivered
- open rate
- adjusted open rate
- unique open rate
- click rate
- unique click rate
- bounced
- had content errors
- message(s) suppressed
- unsubscribe rate
- complaint rate
- are waiting to be sent / retry mode
Comments
Please sign in to leave a comment.