A social media experiment needs five written lines before it starts: a hypothesis, one variable, a success line, a sample rule and a decision. Without them, you are posting two things and picking the one you liked. This page gives you the brief to copy and a way to choose what is worth testing.
TLDR: Write the five lines first. Test one variable at a time, on things that can move results by more than normal noise. Decide the rule for "win", "lose" and "inconclusive" before you look at any numbers.
The one-page experiment brief
Copy this into a doc and fill it in before scheduling the first post.
| Line | What to write | Example |
|---|---|---|
| Hypothesis | If we change X, Y will improve because Z | If we open with the result, saves will rise because people see the payoff first |
| Variable | One thing that differs | Opening line: result first vs question first |
| Held constant | Everything else | Same format, topic, time slot, hashtags |
| Success line | The number and the margin | Median saves per post up at least 20% |
| Sample rule | How many posts, how long | 10 posts per variant over 5 weeks |
| Decision | What happens at the end | Win: adopt. Lose: drop. Within 10%: inconclusive, run once more or stop |
The decision line is the one teams skip, and it is the one that stops arguments later.
Test things that can win
Some levers barely move results. In Mydrop's data, hashtag count shifted median engagement by at most 5% either way, and weekday made almost no difference, with every day within 2% of the median. Testing those means running a month to learn nothing.
The bigger differences showed up elsewhere. Carousels ran about 10% above an account's typical post, a link in the caption cost about 7%, and posting between midnight and 6 am ran 17% below. Format, link placement and opening line are the better test candidates. The full tables are in the Mydrop social media benchmarks.
Account for noise before you pick a winner
Individual posts vary wildly. Across the same data, the middle half of Instagram posts sat between 1.1% and 6.7% engagement. One post at 5% against one at 3% proves nothing.
Three ways to tame noise:
- Repeat the test across at least 8 to 10 posts per variant.
- Compare against your own median, not a fixed number.
- Alternate variants across days so a news event does not hit only one side.
Keep a log so the tests add up
A test that ends as a Slack message is lost in a month. Record every brief and result in one table: hypothesis, dates, result, decision. After a quarter, you have a library of what works for your accounts. For a team version, see a reusable experiment library.
Run the schedule from one calendar
Variants only compare fairly if they go out on a controlled schedule. In Mydrop's content calendar you can label each post with its variant, queue them in alternating slots, and review results in analytics by post. Bulk creation helps when the same idea needs variants across several accounts.
Know when to stop testing
Once a variable has produced a clear winner twice, adopt it and move to the next lever. Do not retest it every quarter. For solo accounts that need a shorter list, see the best A/B tests for solo managers.
Three mistakes that ruin a clean test
- Changing two things at once. A new format and a new caption style in the same variant leaves you guessing which one worked.
- Peeking early. Stopping after three posts because one variant looks ahead is how teams adopt noise.
- Moving the goalposts. If the success line was 20% and the result is 12%, the answer is inconclusive, not "good enough".
Write the brief first and the arguments mostly disappear.
Run your first test this week
Pick one lever from the list above, fill the brief, and schedule the first four posts today. Put the review date in your calendar now. When you want the calendar, variants and results in one place, start free on Mydrop.
Google review
Trustpilot review
Product Hunt comment
