Substack Reader Surveys, Part 2: Viewing and Using Results
A follow-up post
Hey there,
Last week I wrote about Substack’s new feature, which is Reader Surveys.
Reader surveys are a native Substack feature that writers can use to find out more about their readers, or seek their reader’s opinions on their work.
With reader surveys you can…
ask many questions;
ask questions in different formats, such as multiple choice or short answer;
keep sharing the link to the survey over and over in as many posts as you wish;
access the answers from your publication dashboard.
Click the preview box below to read last week’s post, which introduced surveys and how to create them.
This week I explored what it’s like to view and work with the responses to surveys.
Unfortunately, I wasn’t thrilled with what I found.
Reminder: Use reader surveys by dropping a ‘Link to Survey’ button into a post or welcome email. You can create your own survey or use the default survey.
Post editor > Buttons [dropdown list] > Link to Survey > Survey to insert [choose from list] > Button text [edit if desired]
How to access reader survey results
To access the responses collected by a reader survey, navigate to the Stats section of your publication dashboard, choose the Reader surveys tab, find the survey from the list and click ‘View Results’.
Publication dashboard > Stats > Reader surveys [tab] > Survey name [in list] > View Results [button]
Technical glitches
Around 80 percent of my attempts to access my survey results have been plagued with technical glitches. Sometimes the results don’t load. I haven’t been able to access page two of multi-page responses. The three-dot menu on the results page does not respond when I click it.
This is most likely to be a browser/javascript problem at my end, and not a system-wide problem, so you should be okay. I hope.
What to do with survey results
Use surveys to:
learn more about your subscribers;
collect positive feedback and testimonials to share for promotional purposes;
seek feedback on the format, frequency and features of your publication;
discover which topics or past issues your readers have enjoyed the most (and the least);
collect ideas for new posts.
You can view the results of reader surveys on-screen, as unique answers, percentage answers and raw numbers of answers for each option. When viewed on-screen, the results are anonymized. That is, you don’t see any information about who provided each response. You also don’t see any date or time information.
You can export the answers as a csv file by clicking the three-dot menu next to the ‘Share’ button on the results page or on the ‘Edit Survey’ page.
Publication dashboard > Stats > Reader surveys [tab] > Survey name [in list] > View Results [button] > Three dot menu [dropdown list] > Export results
Exported data includes the personal information of each respondent, linked to their answers. Each respondent’s email address, their name as per their Substack profile, and their profile URL are recorded in the exported file with their responses.
Wait what?! Reader surveys are set up to look and feel ‘anonymous’. However, publication owners can see the identity of responders in exported results, making the implied anonymity a lie. Not good.
Exported data doesn’t include time-stamps, which is a major flaw, since reader surveys could run for months or years, especially those used in welcome emails. The lack of time-stamps prevents publication owners from seeing how responses have changed over time.
Newer responses are NOT added to the bottom of the csv file when subsequent export files are created, but appear (weirdly) in the middle of the list, with no apparent order to the responses.
Reader surveys as marketing tools
Reader survey results make great ‘testimonials’ and provide social proof around paid subscriptions. By that I mean that when readers feel like ‘everyone’ loves a newsletter, or ‘hundreds’ of people are paying for it, they feel happier to be on board too.
To use reader survey results as testimonials or social proof, make visuals with them. Pie charts and infographics work well for reader survey results. The images below are some I created for my paid newsletter after running surveys using Google forms.
How to edit a reader survey
You can edit the questions in an existing survey from the Reader Surveys tab of the Stats section of your publication dashboard. Find the survey you want to edit in the list and click the ‘Edit Survey’ button next to it.
Publication dashboard > Stats > Reader surveys [tab] > Survey name [in list] > Edit Survey [button]
From the Edit survey page, you can also take the following actions, using the three-dot menu:
change the name of the survey;
export the results; and
delete the survey.
FAQs
Can you ‘lock’ a survey or close it after a certain date?
No, you can’t lock a survey or close a survey to prevent readers from adding responses after a certain date. However, you can delete a survey.
What happens when you delete a survey?
When you delete a survey, all questions and responses are lost, and any links to the survey in your welcome emails or posts will no longer work.
Can you export survey results?
Yes, click the three-dot menu on the ‘View results’ page or the ‘Edit survey’ page to export results to a csv file (a csv file is a simple spreadsheet).
Publication dashboard > Stats > Reader surveys [tab] > Survey name [in list] > View Results [button] > Three-dot menu [dropdown list] > Export results
Are responses time-stamped so you can see how responses change over time?
No, responses are not time-stamped, there are no times or dates in the on-screen results nor are there any in the data file created when you use the Export data function.
The responses in the exported results are not organized chronologically.
Are responses anonymous?
No, exported results contain the email address, name and personal profile URL of each responder, linked to their answers.
How many questions can you have in a survey?
There doesn’t appear to be a limit to the number of questions you can include in a survey. However, it’s a good idea to limit surveys to a small number of questions, out of respect for your readers’ time.
Can non-subscribers respond to surveys?
No, only subscribers can respond to surveys. When a non-subscriber tries to enter a reader survey they are prompted to subscribe and cannot provide responses until they have subscribed.
Can you make a survey that is only for paid subscribers?
There is no survey setting that could be used to prevent free subscribers from responding to a survey. However, you could prevent free subscribers from responding to a survey by only sharing it in emails and posts for paying subscribers.
Can readers see the results?
No. Unlike with Substack polls, readers can’t see responses. For example, they can’t see which option in a multiple-choice question was most popular. If you want to share the results you need to create your own infographics, charts or descriptions (or use a poll instead).
What is the survey experience like for a subscriber?
If you missed it last week, I created a test survey for you to try. Click the button below to experience the survey, discover what the different question formats are like and find out where you are redirected after you submit your responses.
Final thoughts
I’ve gotta say I’m disappointed with the creator experience for Substack’s reader surveys feature.
The user experience is great, and as such a reader survey should be able to collect good data from subscribers. However, the data could be difficult to turn into information for creators with large audiences and for long-running surveys, such as those in welcome emails.
Without date stamps on the responses, and with exported results not presented in the order the responses were received, a Substack creator with a long-running reader survey has no means of seeing how responses change over time.
There is no way to close a survey without deleting it, thereby breaking links in previously published posts.
The appearance of anonymity for survey responders is false, because publication owners can see the email address, Substack profile name and Substack profile URL of each responder, along with their survey answers.
Disappointing.
Okay, that’s it for this week. Just a heads up: I will be cross-posting this to Medium in a few days time, so big apologies if you follow me there and get this a second time.
Have a fabulous week!
Do us all an additional favor on top of this piece: send it to their customer service people as a bug report because it sounds like this feature is not ready for prime time. Substack has a bad habit of releasing features with a lot of bugs or poorly designed and that seems to be getting worse.
Thank you for testing, summarising and reporting this to us, Karen. Clear and concise as always, and much appreciated.
I wanted your opinion and thoughts before getting detailed reader feedback. I'm going to wait a while. Based on your article, the survey isn't quite a poll but isn't fully geared & powered as a readership 'back-office' feedback survey yet.
I wanted to get a simple baseline read and repeat at specific times to see the impact of changes I make to my website/style.
For me, the lack of anonymity outweighs all other issues.