My Headline Experiment Results + A Warning About Title Tests

I tested the new title A/B feature and learned about a potential issue

Hi Substacker,

Just a very quick note today, since I am skiing this week (yes, we have snow in Australia, though not much).

In Australia, our ‘mountains’ are more like hills. This is about as craggy as they get! But skiing on a blue-sky day is glorious: jackets unzipped, silvery gum trees and lots of sunscreen needed. (Photo: author and son 2014)

I’m writing to let you know the results of my experiment testing the title of my last post.

If you missed the experiment, find it here.

I tested three titles and sub-titles and predicted the ‘vanilla’ title would get a higher open rate. It did 🙁, but not by much¹.

The ‘vanilla’ title - Grow Your Audience with Data-Driven Confidence - was automatically added to the post and sent to the remainder of the list.

Below you can see the full results. Substack sends the results of the title test to you after the test is complete.

The results of the title test on my previous post. The open rates were not very different between the 3 titles I chose.

Warnings!

Reader Jacob warned of potential deliverability issues caused by title tests in certain rare circumstances. This will only affect you if you ever resend old posts; for example, if you resend (previously) paid-only posts to free subscribers at a later date.

If you do resend previously sent posts, then title tests could cause delivery problems. Jacob explains why here. If you don’t resend posts, you should be fine. Thanks,

Jacob Bartlett

.

And Larissa told me her title test reset itself when she edited a scheduled post, resulting in the test sending posts with identical headings and removing the headline she liked best 😣Frustrating! Thanks for the heads up,

Larissa Zimberoff

.

🍒🍒

Okay, that’s it from me for this week. Hopefully, this has given you a full picture of how the new title test feature works.

Have you tried it yet? How did it go for you? Tell me in the comments,

Karen

1. Statisticians would probably tell you the three results in my test (28.5%, 26.5% and 25.2%) are not significantly different… that is, the small differences in the open rates are simply a result of natural variability (‘noise’) in the measurements, and if I repeated the test tomorrow one of the other headlines could just as easily come out on top.