The Community Manager

Introducing A/B Testing For Communities

July 16, 2012
Chris Dowsett

If you’ve been anywhere near a marketing department or advertising team in the last 10 years, you might sigh when I say how brilliantly powerful A/B testing can be. But the truth is, A/B testing can be a huge asset to social communities—particularly when there’s a community site or forum where members come together.

A/B testing is often misunderstood and misused. So, if you’ll bear with me, I’ll start with the basics.

A/B Basics

The whole idea behind A/B testing is that you have baseline variables (things that don’t change) and a single variable that has two variations.

Email is a good example: A company might send an email campaign to 1,000 people with the words “Buy this now, limited time offer” as the headline and a different email campaign to 1,000 people with the words “Special offer ends tomorrow”. Everything else is the same, including the demographics, email contents and graphics. The only thing that changes is the headline. This basic A/B test allows a company to see which headline works better and gets a better response because all other variables are kept the same.

Physical science and those X Y graphs you were taught in grade 4 all draw from the same idea: one variable is tested against an independent variable. By keeping everything else constant, it gives you the opportunity to see what influences someone’s behavior.

Communities can also benefit from the humble and widely-used A/B test.

A/B Testing in Emails

Email is an easy way to try out an A/B test, especially if you use a email service like AWeber or another that allows you to split your subscribers/contacts to conduct an A/B test.

The key here is to keep everything the same except for two variations you’d like to test – some examples could be a header graphic vs no graphic, two different headlines or maybe two different calls to action at the bottom of the email. You could email your members with an offer and phrase the call to action in two different ways. Or maybe you’d like to increase your community newsletter open rate, in which case you might test two different headlines.

A/B Testing on Your Site

Another option is to run some sort of A/B on your community site. For example, you could install a piece of code that randomly displays 1 of 2 banner ads to different people. Each banner ad might would have a different call to action.

Alternatively, you could test the location of the ad (maybe left sidebar vs. right sidebar) while keeping the ad the same. As websites become more social and interactive, lots of suppliers exist that can help you run an A/B test on your community forum – see Optimizely for one example of a company offering an A/B testing solution.

A/B Testing For More Engagement

A/B testing can also help make your community more engaging. In the same way you could test an ad on the sidebar, you could also test a contest or give-away promotion – testing the graphic that encourages the most participation.

Or you could run a feature – say, an ‘Ask the Expert’ in a small business community – for a short period of time, testing out different topic areas. One quick example might be an ‘Ask the Expert About Small Business Loans‘ in July and a separate test in August where you run a ‘Ask the Expert About Small Business Marketing‘ session — changing the topic and monitoring the engagement metrics. In this example, maybe the audience of your small business community is focused on financials and not really interested in marketing. Or vice versa.

A lot of companies will use A/B testing for email marketing campaigns, advertising or website testing — but as long as you stick to the rule that you have a baseline and only tweak one variable, you can be creative and run A/B testing on all sorts of things.

It Just Gets Better and Better

What’s great about A/B testing in a community setting is that it will help you continually improve the site or forum for your members.

Let’s say, for example, that you find out that people are more responsive to a feature series at the top of the right sidebar. Great – next you could test what kind of graphic or insert works the best in this spot and what kind of headline convinces people to engage more. Once you start A/B testing, you’ll notice that you can continue to get more and more specific on improving your community.

The biggest mistake people make is that they’ll change more than 1 variable – such as location and color – which makes it difficult to understand what caused the difference in clicks. But getting the A/B test right can be a simple—but powerful—way to get quick answers on what works and what factors encourage more engagement.

Next up: Ideas and Inexpensive Ways to Survey Community Members

Chris Dowsett

Chris Dowsett

Chris Dowsett (or Dows for short) is your friendly data analyst, statistician and numbers person. Originally from Australia, Chris works for Intuit as a social media analytics manager. He’s also studying a Doctorate focused on improving data use in business decision making.

5 Comments

  1. DavidSpinks

    Hey Chris,
     
    I think there are a lot of interesting opportunities to perform A/B tests in communities.  It seems that everything you covered in this post is email marketing, content marketing or product focused.  Would love to hear you talk a bit more specifically about communities, and the kinds of tests you can run to improve things like user to user interaction, or metrics like how community engagement can result in increased user activity.  
     
    Say for example, within a forum, what kinds of A/B tests would you run to improve the community?

  2. itweetlive

    We actually utilize A/B testing and clustering tools to provide response suggestions based on real-time Twitter analytics in our search engine. A/B testing also allows us to measure engagement rates for responses so users can refine their messaging. The idea is to create communities for clients by helping them do better, smarter outreach to potential customers, fans or community members.
     moreyaaltman ITWEETLIVE.COM

  3. ChrisDowsett

     @DavidSpinks Hi David – thanks for the comment and feedback. Sounds like a great idea for a follow-up post. 

  4. itweetlive

    @ChrisDowsett
    hi Chris talk with us before your follow up post i@itweetliv.com  as mentioned we made great studies on it. we group similar  statues updates and than send ( NOT  automatic )  different responses and measure their  Engagement levels by clicks mentions back re tweets and more.
    Check this search http://itweetlive.com/PublicSearches/242  ( scroll down to the stats) , it is a stage one conversation, notice that there some responses are generating much higher engagement levels.  A stage two of the conversation  is to creat a mention search…  hear is a simple AB test on a response to a RT . http://itweetlive.com/PublicSearches/11760

  5. louisvuitton

Trackbacks/Pingbacks

  1. Top Five Community Management Blog Posts: July 2012 | Dave Cayem - [...] 4. Introducing A/B Testing For Communities [...]
  2. 3 Methods to A/B Test User Interaction in your Community « The Community Manager - [...] my last article, I introduced the concept of A/B testing for communities and provided an overview of testing basics. David…

Submit a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.