Marketing Brand and Messaging

Run message tests to find which messages get the most response

How do you find out which messages get the most response on each channel?

In a message test, the Results Analyst agent compares two versions of one message, such as two email subject lines. The version that gets more responses wins. The Test Designer agent writes the testing rules and sets up each test. The Results Analyst confirms each winner. You approve each change to your core messages.

Published Updated

What you get
Message testing program · Saltmarrow Studio
Testing program running
2
tests running at all times, one on email and one on ads
6
test ideas ranked by how much each could raise responses
Testing rules One test a month on each channel, one change in each test
Certainty before you pick a winner 95% sure that a result is not due to chance
Email subject line test Version B won, with an open rate of 31% compared with 24% for version A
Ad headline test Run for two more weeks to reach 1,000 views for each version
Core messages Winning subject line added
Next email test The wording of the offer

A new test starts on each channel as each test ends.

See it run

Message tests for Saltmarrow Studio, in progress

Newsletter list
2,600 contacts · Two newsletters a month, plus ads in Google Ads
First tests with a confirmed winner 1 / 2
Schedule
Every month
A new test on email and a new test on ads
Run log
  1. You
    “Test the subject lines of the newsletter and the headlines of the ads.”
    Your instruction
  2. Testing rules agreed for 2 channels Test Designer
    You chose email and paid ads as the first channels, because both have steady volume. The Test Designer agent planned one test a month on each channel. In each test, two versions differ in one element only. The sample size is the number of people who receive or see each version. The Test Designer calculated a minimum sample size of 400 sends for each email version. The minimum for each ad version is 1,000 views. The confidence level is how sure you are that a result is not due to chance. You chose a confidence level of 95% for every test. The Test Designer wrote the testing rules in Google Sheets.
  3. 1 of 2 first tests has a winner Results Analyst
    The Test Designer agent set up a subject line test on the newsletter. The same agent set up a headline test on your ads. Each version of the subject line had 1,300 sends. You chose the open rate as the measure for subject line tests. The open rate is the share of readers who open the email. Version B had an open rate of 31%. Version A had an open rate of 24%. The Results Analyst agent confirmed that the gap between the two open rates is statistically significant. Statistically significant means that the gap is not due to chance. Each ad headline had 640 views, below the minimum of 1,000. You chose to run the ad test for two more weeks.
    •
  4. You
    Approve the core message update You
    The Results Analyst agent added the winning subject line to your core messages. You chose to replace the old subject line at once. The Results Analyst logged the old subject line in a change log as a backup. You chose a written note and the change log to tell your team about each change. Approve the update to your core messages.
  5. You
    Review the ranked list of test ideas You
    You chose three sources of test ideas: words your customers use, your own ideas and versions that lost an earlier test. The Results Analyst agent ranks the test ideas by expected effect. Expected effect is how much a test could raise the response. So far, the list has six ideas. The top idea is a test of the wording of the offer on email. Review the ranked list of test ideas.
  6. Core messages used Upstream
    Your core messages come from Define Core Messages.
  7. Channel messaging guide used Upstream
    Your channel messaging guide comes from Tailor Messages for Each Channel.
How it works

You choose how sure a result must be to count as a winner

You choose the confidence level for each test. The usual level is 95%. You can accept 90% for faster tests. The Test Designer agent prevents a winner from being picked too early. You also decide what the Results Analyst agent does when a test does not reach its minimum sample size. For example, the Results Analyst extends the test once. If the sample is still too small, the Results Analyst ends the test with no result.

What it needs
Your core messages, with the main messages you use now
An email platform, ad platform, or social media tool with enough volume for testing
At least 200 sends or views for each version, so results are reliable
The channel messaging guide, as the source of the versions to test
What you get

A testing process with one test always running. Written testing rules, the first statistically significant results added to the core messages, and a ranked list of at least five test ideas.

Run sequence

Start to finish

You start from your core messages and your channel messaging guide. Then you, the Test Designer agent and the Results Analyst agent set up your message testing program in four parts.

  1. 1
    Testing Rules

    Write the rules for testing

    Test Designer plans how often tests run
    Test Designer calculates a minimum sample size
    You agree on the testing rules
    Test Designer + You
  2. 2
    First Test Results

    Run the first message tests

    Test Designer starts tests on busy channels
    Two versions differ in one element only
    Results Analyst confirms statistical significance
    Test Designer + Results Analyst
  3. 3
    Updated Core Messages

    Add the winners to core messages

    Results Analyst adds each winning version
    Results Analyst tells your team about changes
    You approve the core message updates
    Results Analyst + You
  4. 4
    Ranked Test List

    Rank the next test ideas

    Results Analyst ranks ideas by expected effect
    Results Analyst keeps one test ready
    You review the test list
    Results Analyst + You
Before it starts

What Works asks you

A few questions in plain words. The answers set up the run for your business.

Which of your channels have enough volume for statistically significant results?
YouEmail and paid ads, with steady volume
What confidence level (how sure you are the result is not chance) must a test reach before you pick a winner: the usual 95%, or lower for speed?
You95% for every test
If time is short, should you run one test a month in total, or keep one test per channel?
YouOne per channel from the first day
If a test stays below its minimum sample size, should it run longer, join the next cycle, or end with no result?
YouExtend once, then end with no result
Should a confirmed winner replace the old message at once, or run next to it until it works in real use?
YouReplace, and log the old one as a fallback
Where should new test ideas come from: your own ideas, customer language, or losing versions worth a second test?
YouCustomer language, own ideas, second tests
Who runs it

The agents behind this workflow

The Test Designer agent writes your testing rules and sets up each test. The Results Analyst agent confirms each winner and ranks your next test ideas.

Playbook specialist

Messaging Strategist

Owns this area

Tests your messages with buyers, adapts them for each channel and customer segment, and turns the tested messages into sales enablement content.

On this workflow

Test Designer

Sets the testing rules and sets up each valid test that changes one element

What it can do
Set a testing schedule and the one-change ruleCalculate a minimum sample sizeSet up a test on its channelPull message versions from the core messagesPrevent a winner from being picked too early

Results Analyst

Confirms statistical significance, names winners, and updates the core messages and the test list

What it can do
Confirm statistical significanceName a winning version from the resultsUpdate the core messages accuratelyRank test ideas by expected effectKeep one test always ready
Jeanie, your right hand Works Crew

Jeanie leads a core crew of seven that runs your company day to day. They direct the specialist and workflow agents in each playbook, including the ones on this workflow.

Connect your tools

Bring what you already run

Works connects to the apps you already use.

  • Google Ads This workflow uses Google Ads.
  • Google Sheets The Test Designer agent connects to Google Sheets.
  • Mailchimp This workflow uses Mailchimp.
Other apps you can use
Your email marketing app

or any email marketing app you use

Your ad platform
  • Meta Ads Manager
  • LinkedIn Ads
  • Microsoft Advertising
  • TikTok Ads

or any ad platform you use

Your analytics tool

or any analytics tool you use

Your spreadsheet app
  • Microsoft Excel
  • Airtable
  • Smartsheet
  • Zoho Sheet

or any spreadsheet app you use

These are starting points, not a limit. Connect the tools you already use.

Connect your work

The workflows before your message testing program

Linked cards are live pages. The others are part of the Marketing playbook in Works.

Works · AI business OS

One workflow inside your whole business

In Marketing, Run Message Tests is part of the Brand and Messaging work.

Prebuilt playbooks, plus your own
StrategyMarketingSalesCustomer SuccessPartnershipsProducts & ServicesOperationsFinancePeopleLegal & Compliance + Build your own
The Marketing playbook
74
Workflows
20
Specialist agents
147
Workflow agents
226
Templates
You're here
Run Message Tests
One of 74 Marketing workflows · run by 2 workflow agents
Templates

What comes with it

Built for this workflow
Message Test Tracker XLSX
A sheet where you track each message test. It opens in Works after you sign up.
Test Hypothesis Canvas DOCX
A document where you write the result you expect from each test and the reason. It opens in Works after you sign up.
Also included here
Positioning Validation Tracker XLSX
A sheet where you track the tests of your positioning. It opens in Works after you sign up.
Channel Message Matrix XLSX
A sheet where you match each message to each channel. It opens in Works after you sign up.
Ad Copy and Creative Matrix XLSX
A sheet where you plan the text and the images of each ad. It opens in Works after you sign up.
A/B Test Results Model XLSX
A sheet where you calculate the results of each test of two versions. It opens in Works after you sign up.
Experiment Backlog XLSX
A sheet where you list the experiments you plan to run. It opens in Works after you sign up.
Experiment Hypothesis Canvas DOCX
A document where you write the result you expect from each experiment and the reason. It opens in Works after you sign up.

Each template opens inside Works after you sign up.

Questions

What people ask

What is a message test?

In a message test, the Results Analyst agent compares two versions of one message. For example, half of your email list gets subject line A. The other half gets subject line B. The winner is the version that gets more responses. The Results Analyst confirms a winner only when the gap in responses is not due to chance.

How many sends or views does each version need?

Each version should get at least 200 sends or views, so the result is reliable. The Test Designer agent calculates the minimum sample size for each channel. Busy channels reach the minimum faster. You decide what the Results Analyst agent does when a test stays below its minimum. For example, the Results Analyst extends the test once.

Which measure do you use to pick the winner?

You choose the measure for each channel. The open rate is the share of readers who open an email. The click rate is the share of people who click a link. The conversion rate is the share of people who take the action you want. You can use the open rate for subject line tests only.

How do you use AI to run message tests?

The Test Designer agent writes the testing rules. The same agent sets up each test on its channel. The Results Analyst agent confirms each winner. The Results Analyst adds each winning version to your core messages. The Results Analyst also ranks your next test ideas. You agree on the rules and approve each change.

Find the messages that get the most response

Sign up to run your first message test.

Sign Up