Grow Your Audience with Data-Driven Confidence

Introducing Substack's new headline feature

Hi Substacker, happy Thursday¹!

Every time Substack launches a new feature, I sigh. Another one? Really?

And then I write a post telling you to avoid the hype, to take it slow, to not let the new feature distract you from the most important thing - writing and pressing ‘send’ each week.

Every new feature is a new distraction, a new shiny button; a new excuse to take your eyes off the things you need to do that will actually get you closer to your goals on Substack.

Avoid the shiny button, I say, don’t get distracted.

But today I’m flipping that advice on its head. Today I want you to press the shiny button on Substack’s latest feature.

What is this week’s shiny button?

This week, Substack announced that all publications with at least 200 subscribers now have access to a headline testing tool.

Substack is calling it ‘Title Test’, but marketers will know it as A/B Headline Testing or Split Testing.

What is the point of testing a headline after you’ve sent a post, you may ask? Well, the cool thing about this feature is it will different headlines by sending different versions of your post to a portion of your subscribers, before automatically deciding which works best, then sending the best headline to everyone else.

So most of your readers get the version with the best-performing headline.

Which is actually pretty awesome.

I pressed the shiny button today (and I liked it!)

How it works

• You draft a post as usual, with a title and subtitle

• During the publishing process you can choose to activate ‘Title Test’

• You add one or more alternative title(s) and click ‘Start Experiment Now’

• Substack sends your post to some of your subscribers, with the different headlines you’ve chosen split equally across the test group, and monitors open rates

• Substack chooses the headline with the highest open rate, adds it to your post and sends it to the rest of your subscribers.

Find the title test feature in the publish dashboard (access by clicking ‘Continue’ in a draft post). Only publications with more than 200 subscribers can use this feature.

Does it work?

Substack says the title test feature performed well in beta testing and was well-received by creators who got to try it. They shared an example with three headlines. One headline got a 20% open rate while the other two got just 15%.

Michelle Teheaux of Untrickled reported even better results on Notes, with her winning headline getting an open rate of 27% compared to 14% and 17% for the alternatives.

Michelle Teheaux of Untrickled shared this screenshot on Notes, showing the results of a recent title test (Source)

These are good results. If you can boost your open rates by even 5 percentage points, you’ll find it easier to build relationships with your subscribers. And better relationships mean more shares, fewer unsubscribes, more recommendations and better growth.

Will I use this feature going forward? Yes.

Did I use this feature for this post? Yes. Read on to see what I tested…

What are the downsides of A/B testing on Substack?

Although I’m uncharacteristically excited about this new feature and think it will truly add value to my work and yours, there are a couple of downsides.

The main downside is that it adds yet another task to the process of sending a post.

Honestly, we already have SO MUCH TO DO every time we want to press ‘Send’, things like choosing images, adding tags, writing an SEO description… err what a drag.

And now we’ve also got to come up with more headlines?

It’s too much! I’ve been publishing here for four years and I’m stressed about adding another task to the pre-publish to-do list - I can’t imagine how overwhelming this must feel to a newbie.

The second downside became glaringly obvious when I tried to draft an alternative headline for the post you’re reading now. I’ve written this whole post around the theme of ‘shiny button syndrome’, also known as shiny object syndrome.

It has the working title ‘Press The Shiny Button’ and has an image of a shiny button for the social preview.

What if the best-performing headline isn’t one about shiny buttons? Will the post make sense? Will my theme fall flat when the post is opened? Arp! The solution here is that I should only include alternative headlines with the same theme, but that might reduce my chances of hitting on a winner.

The third downside is related to the duration of the title test. Substack allows you to choose how long to run your title test. The default period is one hour. The maximum duration is six hours.

The setup favours publications where most subscribers live in a single time zone.

My other publication has readers in 104 countries, so people read my posts across every possible international time zone. In addition, since that publication is for work, and sent to company email inboxes, many of my subscribers will wait 4 or 5 days before opening each weekly issue.

This means any title experiment I do for that publication will only sample readers who are in one or two time zones and who open emails quickly, not necessarily my most engaged readers. I’ll need to tread carefully.

Choosing an experiment duration for title testing is tricky when your audience is spread across every timezone in the world

FAQs

Q: Why can’t publications with fewer than 200 subscribers use this feature?

A: Because the dataset is small when it’s used on small publications, making the test results unreliable and unhelpful.

Q: How do I find this feature?

A: Find it in the publish dashboard for a post. To access the publish dashboard, click ‘Continue’ while in a draft post on desktop. It is not accessible in the app.

Q: Does Substack create alternative titles for us to use?

A: No, you need to come up with your own alternative titles for the test.

Q: How many titles can I test at one time?

A: You can test up to 4 titles, but I recommend sticking to 2 or 3 unless you have more than 10K subscribers, as the dataset might be too small for good results.

Q: What about subtitles?

A: You can include subtitles in the test if you wish.

Q: What proportion of my list should I send the test to?

A: You can choose between 5% and 50% of your list for the test size. I recommend you use the proportion recommended by Substack in the test settings, as it will be optimized to get the most reliable data for your size of audience. For example, Substack recommends I send a test to 20% of my 6.2K audience and 30% of my 4.2K audience.

Test size: Substack recommends I use 20% of my 6.2K audience and 30% of my 4.2K audience for title tests

Q: What about test duration?

A: You can choose a duration of 30 minutes to 6 hours. The best duration depends on your audience and their timezone spread. If most of your readers are in similar time zones and you are confident they open your emails and posts soon after you publish, choose 1 hour. If your audience has a wider time zone spread, choose a longer duration.

Remember, subscribers who aren’t in the test will receive your post after the test has concluded. So if you choose a 3-hour duration for the test, most of your subscribers will receive your post 3 hours later than usual.

Tip: to find your audience location, go to Publication dashboard > Stats [tab] > Audience [tab] > [scroll to] Audience location

Q: What’s the difference between ‘headline’ and ‘title’?

A: Nothing, I’m using both terms interchangeably (just keeping you on your toes!)

You’re part of my experiment!

Guess what, I just experimented on you (😏 thanks). I used the ‘title test’ feature for this post.

Here are the three titles and subtitles I tested:

• Press The Shiny Button

  • Why you should (actually) try the newest Substack feature

• Grow Your Audience with Data-Driven Confidence

  • Introducing Substack's new headline feature

• Is Headline Testing Worth It?

  • Substack just launched headline A/B testing

Here’s what I predict will happen:

1. The second headline/subhead combo will perform best (but I hate it coz it’s so vanilla).

2. I’ll be cross because I like my shiny button image and theme and it failed in the test

3. Also, I’m almost certainly wrong in my predictions.

If you are in the test group…

You received this post 3 hours earlier than everyone else. The experiment is scheduled to begin at 8:19 am New York time (10:19 pm Sydney time), Thursday.

You would have seen a headline that might be different from the final one. To check the final headline, view this post on the web after 11:19 am New York time.

If you weren’t in the test group…

You received the email at around 11:19 am New York time (1:19 am Sydney time) and your email had the title chosen by Substack for its headline. Really hope it’s not the boring vanilla one.

Final thoughts

I usually tell you to ignore the FOMO and shiny object syndrome that accompanies every new Substack feature launch. It’s easy to get distracted from the important things when all the Substack gurus are losing their minds over whatever-it-is-this-month.

This is not one of those times.

This new feature is worth your attention.

This new feature will help you get better headlines. Good headlines will help you have more impact and grow your Substack publication.

Better headlines improve open rates and that allows you to build better relationships with more of your subscribers. Better relationships mean more fans, more shares, more recommendations and easier growth.

So go on, brace yourself for yet another new pre-publishing task and try the new title test feature on Substack. And let me know how it goes.

Cheers!

Karen

P.S. I wrote about headlines, their importance, and some tricks for doing better with them last month. You can read that by clicking the preview box below.

1. Sorry Kiwis, Aussies and Islanders! Happy Friday to us.