In my last article, I introduced the concept of A/B testing for communities and provided an overview of testing basics. David Spinks responded with a great question: how can A/B testing be used specifically to improve user interactions?

Although A/B testing started out in the direct mail industry, the fundamental goal is to understand the user’s experience, interaction and behavior. The direct mail ad executives wanted to understand which headline or image made people pick up their brochures. Nowadays, marketers also use A/B testing to help increase engagement on their websites, like downloading their trial software or filling out their landing page form.

In both instances, the focus is on the triggers and cues that prompt people to complete an action. For communities, it’s helpful to understand how to get people more engaged. This is becoming increasingly important as people have more options, more communities and more networks…and less time.

So let’s take a look at three using real-life examples to show how A/B testing can help you better understand your user’s experience at different points in your community.

It’s important to note that each forum is unique so some of these might not apply—but hopefully they give you an idea about how to A/B test in your community.

1: Registering and Sign-Up

There are a lot of different sign up options. Some forums allow you to sign up with just an email address. Others ask for more details like birthday or location. This means that there are many different possibilities for the registration process.

I worked with a technology company that had started a support forum. Part of the initial set up included testing the registration to see whether different questions impacted completion rates. We tested two versions of the form over two weeks each, using the same marketing tools to get comparable traffic to the site.

The first form was shorter so members could sign up quickly. The second version was slightly longer by a couple of questions but members’ profiles were then more complete when they entered.

The results showed that potential members to the support forum wanted quick access in order to post a question. They weren’t necessarily concerned about profile completion rates. If it was a different type of forum, say a small business networking forum, results might have been different. However, in this example, the expected interaction was the ability to post a quick question because the audience is looking for specific information—rather than spending extensive periods of time promoting their own profiles and networking.

2: Featured Categories vs Featured Discussions

A lot of forum tools, like NING, allow you to feature specific discussions from your community members. Or you can also feature categories of discussions, such as a technology forum featuring a “Questions for the Tech Team category, rather than a specific discussion started by a member.

A colleague of mine was tasked with increasing the engagement of their forum members. The default setting featured one category on the forum page, clearly distinguished from the full list of discussion categories. So they featured the “Tips and Recipe Ideas category, where members could post their own recipe ideas.

My colleague then ran a simple test to see if engagement changed if the forum featured the “Tips and Recipe Ideas category at the top or if it featured specific discussions. The team dedicated a period of time to try each of these options, running the same level of promotion for both trials. For the discussion trial, the team featured three recent member posts from different categories to try to reduce bias and they swapped out the featured discussions every two days.

What was interesting in the data they collected was that featured discussions received replies more quickly than the “Tips and Recipes Idea category (discovered thanks to the time stamp data on replies) but each discussion only received slightly higher engagement.

The team is doing more testing over the next few months but early indications from this data might be that featuring discussions certainly helps get a quicker response but the amount of replies to each post generally tops out at a certain number. Or perhaps featuring discussions longer than two days may change the data. This sort of feedback provides an opportunity to do more A/B testing to gain additional insights into the community.

3: Introducing A New Category

My third example is one that I’m right in the middle of testing. We started a forum for software users to talk and provide tips for other members. So far, we’ve tested things like how often we should send a community-wide email and how much content should go on the front page.

Now we’re looking to test a new category. We found posts in other categories that gave sporadic feedback on products so we thought we’d test to see if there is interest in a specific category for product feedback.

The community manager was cautious about changing the community too much, which makes sense as communities are precious things and sensitive to change, but we saw enough posts in other categories that together we decided to test this new section on product feedback.

Most A/B testing offers two variables in direct contrast to each other – audience A gets version 1 and audience B gets version 2. This example is a little different because we’re looking at engagement data, changing one aspect like the category list and then re-examining data. This makes it a longer-term test because newer categories might cause inflated interest. So we need to look at the data over some months to get a truer representation on the impact of this new category. We also need to be careful to try to limit any outside biases like extra promotion or other changes.

Unfortunately, A/B testing in the real world isn’t always neat and tidy but as long as there’s a concerted effort to limit other changes, you can start to gain insights from testing version A and version B.

———

As mentioned earlier, these are just a three examples of A/B testing that I’ve seen in action. Every community is different and the opportunity for A/B testing changes with each site. Hopefully these examples give you ideas about how you can implement A/B testing on your community to increase user interaction.

To me, increasing interaction begins with understanding what improves the user experience, like featuring a discussion instead of a category or creating the right types of categories. A/B testing can help provide some of that understanding and ideas for community improvement.

I’d be really interested to hear about what other A/B testing you might have done or would like to do as a community manager. Please leave a reply below or shoot me a note on Twitter.

Photo creds: Mojo Media Labs and Global Reporting