Hellooo Nurse on Flickr I've talked about this topic in a couple of places, but I don't think I've actually posted what we ended up kind of coming up with as pretty decent measurements of the health of a community. But, perhaps I should first define what I mean by community health:

The health of a community is the gauge of where various qualitative and quantitative metrics lie in relation to the goals you set.

Heh. Right. So, what this really means is that you need to first set your goals (with an emphasis on qualitative), and next define your metrics. Of course, after all of that is set out, you can figure out how to achieve that (which is just as convoluted).

Setting Your Goals

Setting your goals is a very personal thing and not so simple. It can also lead to an emphasis on quantitative metrics (i.e. I want 500,000 new members by Christmas). In order to steer goal setting towards a more qualitative, more community conducive perspective, we like to help our clients frame their overall goal in the following statement:

"I want to create a culture of ________________"

Where the blank space is filled in with a word or a phrase to describe the kind of community you want to foster. It could be a culture of gaming or a culture of passion. It could be a culture of generosity or a culture of happiness. After you define the type of culture you want to see unfold around your product, site or service, you can start to define what exactly that means. Let's take a culture of generosity, for instance. What does that mean? How do you want people to act within that community to promote that? What actions should be rewarded? Avoided? What should you build in to encourage this?

Creating a culture of generosity could unfold like this:

  • sharing of ideas openly
  • less competitive, more collaborative
  • encouragement between members for sharing
  • rewards for sharing and collaborating (points? gifts? private kudos?)
  • mentorship program to help new people become well educated on the community
  • a higher emphasis on caregiving

Of course, the site features can also help encourage this and you can start to brainstorm the ways that people can be generous with one another by using these features. As you build the list of the kind of behaviors to encourage, you will also start to understand the kind of metrics you will need to use to figure out whether or not you are successful in creating a culture of whatever it is you have chosen.

Defining Your Metrics

My good friend (and brilliant consultant) Brian Oberkirch recently sent me and some others a quick email asking how we measure the success of our community programs (I think he wanted some more ideas for his FOWA presentation). I didn't want to give him an, "It depends" answer, so I threw out a quick list of the metrics we HAVE used, including:

  • the rate of attrition, especially with new members (I think it is really telling when you drive traffic that doesn't stick around - you will have to really examine whether you are offering something of need)
  • the average length of time it takes for a newbie to become a regular contributor
  • number of referrals (strength of positive word of mouth)
  • multiple community crossover - if your members are part of many communities, how do they interact with your site? Flickr photos? Twittering? Etc.?
  • the number giving as well as the receiving actions - eg. readers receive, posters are giving (advice, knowledge, etc.). PopSugar has a neat reward system built in for this with their gifting for contributions in the community
  • community participation in gardening, policing and keeping the community a nicer place (eg. people who click on the 'report this as spam', people who edit the wiki for better layout, etc.)
  • number of apps built off of your API (if you have one) - a good 'number' measure as the number of apps usually correlate with your social capital

This is in no way a comprehensive list of metrics. These are only some of what we have used recently and what I can recall about using in the past and the list evolves with every new project. I've actually put up a wiki page on Spread Love so that people can add to it as they see fit.

Of course I do alot of research and am interested in what others are doing on this matter. When looking around for more information, I found a pretty decent list of measurements to consider put out by Forum One with a few crossovers:

  • Member engagement: activity and "investment" in community
  • Member Loyalty & Satisfaction
  • Membership Growth and Attrition
  • Member referrals
  • Quality of content and exchange: For instance, resolution time, days thread was active, ratio of validated responses. Support communities are leading the way on best practices and reporting.
  • Tracking the brand through the “Community ecosystem”: Tracking brands and community members as they travel through the larger community ecosystem that spans sites, technologies and devices.
  • Impact of the community on revenue: Particular attention is being paid to the value of members, both to the host communities’ revenue, and the organization’s sales or fundraising.*
  • Mobile interactions with the community: including views and posts from mobiles.

I couldn't find out how they measure investment, loyalty and satisfaction, which I gathered from looking around, they use a combination of webstats and intuition, but these are all good places to start at when thinking about how to measure how well you are reaching your goals.

[*I would add indirect revenue like evangelism, too, as one person being an evangelist for a company is worth multiple purchases]

But What About the Qualitative?

Most of these metrics are still very quantitative, though. What about the qualitative? The anecdotal? Although the numbers are directly measurable to track along the way, the truly interesting results (as well as the biggest rewards) come from the stories...the human interest pieces. They come from gathering awesome emails you receive from really happy community members or running into someone at a party that, when they find out you work for Company A, they squeal in delight and tell you how much you rock. Adversely, you may find a forum thread somewhere where someone rants about how awful their experience with you was. Maybe there is a lack of discussion altogether, which is the anti-anecdotal. Maybe you aren't even interesting enough to be talked about. :|

I don't know if any companies do this, but I'd encourage you set up internal cheers and jeers (or call them whatever means something to you) wiki pages. On the cheers wiki page, paste in the happy emails, blog posts, etc. and personal stories of positive encounter with customers. On the jeers wiki page, paste in the negative feedback, the angry emails and/or the lack of reaction at all. Don't compare yourself to anyone else but your own goals. And don't get down if the jeers page is full and the cheers page is empty. Think about the jeers as things to work on and customers to win over...as opportunities. Think about the cheers page as ideas for ways to create more of that...more opportunity.

When you are getting enough feedback, have weekly anecdotal meetings where everyone shares these stories and brainstorms the opportunities. You will know when you are successful on this end when you have to limit these meetings to people's 5 favourite cheers and 5 favourite jeers each so that the meeting doesn't last a whole day.

You can, then, attach a quantitative metric to the opportunities: how many jeers can you turn into cheers? Set a goal there, too. Start simple and add progressively more challenge. Creating more happy customers is always a good metric.

Really Important Stuff to Remember About Metrics

Metrics are rarely useful for the community members themselves. Most of what matters is experience. People may revel in the fact that there are over 25 million people signed up at the same service that they are using, but what matters is that their closest friends are using it and that they are getting value out of it. The 25 million people just have the experience in common.

Who are metrics for then? Business people. Venture capitalists. Journalists. Outsiders. People who want a number to tell the whole story (mostly because they are not part of the community itself and it's really hard to explain the impact of a great community to an outsider). In the end, it comes down to Social Capital and as I've discussed, Social Capital is incredibly elusive. It is measurable, but only relative to the source (how do you measure happiness? everyone has a different experience of it), which probably makes it the loveliest, most perfectly decentralized system in the universe, which is where we are headed, but so many people can't grasp that yet.

But hell, we need to communicate outside of the experienced boundaries of our communities, so we have metrics. We use metrics to entice more people to come and experience our communities beyond the numbers. We use metrics to try and communicate the pride we have in the amazing things happening in our communities. We sometimes have to compare to give others a reference point. But in the end, we know in our hearts of hearts the real measure is in the experience of it. The proof, as they say, is in the pudding (why do they say that, though?).

When I was in Rotterdam, I had the pleasure of hanging out with Biz Stone, co-founder of Twitter. Over and over again, people tried to get him to tell them the 'number'. "How many people are using Twitter? Must be millions! Tens of millions!" But he wouldn't budge. He would answer calmly, "We don't reveal our numbers."

I really respected that. Twitter probably does have tens of millions...egad, could even be pushing 100 million for all I know...but by Biz not revealing that number, he made people think harder about their experience of Twitter. He made people think harder about the stories of Twitter. He made people really consider the impact of twitter. A number would shadow that really important stuff. I'm glad he doesn't tell and I hope him and the rest of the Twitter team never do.

So, there you go. This is how we currently view measurements and metrics and all of that stuff that makes the bidness world tick.

10 Comments