How do you measure the quality of collaboration?This is in reference to a community of practice–docs/researchers doing research to establish best practices in their field.

– Jenn Kramer (Question 1 of 2)

Thanks for your questions Jenn. I’ve broken them up and will try to address them across two different articles.

There’s a lot of excitement around crowd-sourcing, co-creation and the idea of the mini social networks around collaboration—so the need to measure the quality of collaboration is important.

Examples of this include some exciting work by GE to build mini social networks around the aircraft engine where GE engineers and airline engineers collaborate online to drive efficiency. Or, in your case, researchers collaborating to develop best practices in their field. And then we’ve seen the launch of services like Celly, which build mini social networks to help with the classroom and other places, all promoting collaboration.

Measuring collaboration is a big task and involves looking across a range of metrics to tell a holistic story. There doesn’t seem to be one metric that can measure collaboration on its own. So it’s worth exploring a few different ideas.

In your situation, I’d consider framing the measures and values in terms of the benefits your software or communities can provide compared to alternatives like conferences that can be expensive, or traditional communication like email that can eat up a lot of time.

Collaboration Starts With Engagement

To me, collaboration starts with engagement so look at some foundation metrics like the percentage of members who post a comment every month.

Participating New Members

Also, look at the growth of “participating new members” which is the amount of new members that join and go on to participate. If you’re looking at a channel like Facebook, look at the proportion of engaged users.

Average Number of Conversation Per Member

If you have a community forum and have access to a detailed level of data, you can also look to the average number of conversations that a member is engaging in. The theory being that the more conversations they engage in, the higher the level of involvement.

Speed and Time To Response

I’d also look at a metric around time. That could be the time it takes, on average, for a question to be answered by the community. In the GE example above, time is a critical component and one of the benefits of mini social networks being created around the machine. The engineers can work on the problem in real time (or close to real time) so that the airline can get the engine working as quickly as possible. This reduces the amount of lost revenue from an inefficient engine as well as the resources saved by speeding the project. Time and speed is becoming a differentiation for business innovation and is worth including if you can.

Size Of Conversations

I’d also look to the size of the conversations happening. Track the number of replies to a topic and the average thread size, as this could indicate which areas are engaging community members and sparking interest. This might separate the quality discussions from the others.

Bring In Other Data Like Member Surveys

I think community data needs to be mixed with feedback from other sources too—like surveys. Community data will only tell you so much and a survey, used correctly, can give you some great direct feedback. You could ask a short 5 – 10 question survey around topics like whether the resources are helpful, what topics the community would like to introduce, satisfaction with the community and anything the community members would improve. You could run a regular quarterly survey to keep a finger on the pulse and get great feedback.

New Ideas Submitted

You could also get creative and look at metrics around the number of new ideas posted to ‘idea questions’ that you’ve posted like “If you could improve a feature of this software, what would it be?. My only caveat here is that you’d need to link metrics to a content strategy, like asking an ‘ideas’ question once a month, and to track the response over time.

Tied to that is crowd-sourcing and you could track the number of new ideas posted in a crowd-sourcing tool alongside the engagement of the community through voting and add-on comments under these ideas. These metrics help measure both collaboration on ideas but also the overall excitement of the community. If new ideas, voting or add-on comments start to drop off —it might highlight a community that’s disengaging.

Representation Metrics For Niche Communities

A more complex metric area is to look at the value of the ideas and projects coming out of the collaboration. If the community is focused around a particular field and researchers are coming together, you could look at the penetration of the community among the researchers.

For example, you could investigate what number of the leading 100 universities in a particular field are represented by researchers. Or some geographic mapping based on ZIP code could help identify the representation of researchers from across the US, for example. This is a little out of the box but could also be some really interesting data. As the community grows and the quality of collaboration increases, more researchers will hopefully be drawn to the community and participate in the discussions.

Pour Data Into A Framework For Measurement

There are many potential metrics here, so I thought it would help to look at a simple process you can follow to help you decide on a framework for your measurement:

1. White-board some notes on what collaboration means to the field or community topic. For some, it might mean connecting researchers throughout the world around one problem so that they can share data sets. Or you might bring together the leading researchers on workplace safety and encourage them to share ideas on what can be done to improve employee safety.

2. Match the ideas on collaboration to some potential metrics. For example, if your goal in the collaboration is to encourage researchers to share new ideas, then you can set up goals around the amount of new ideas posted per month and the number of add-on discussions to the shared ideas for engagement.

3. Decide on a set of metrics that are trackable over time as well as being realistic given your amount of resources.

4. Get some initial data (this is just a starting point as a way to see if there’s progress) into a spreadsheet or tracking tool as soon as possible and set a regular time frame (e.g. weekly) to collect the data.

Hopefully these ideas help you set a framework and come up with a way to track the collaboration around your communities. Collaboration is a hard thing to tie down, but tracking a few of these metrics should provide some feedback on how the community is going and whether collaboration is increasing.

Good luck and if you have any follow up questions, shoot me a note on this article or over at Twitter.