FreshNetworks Blog

One of my favourite podcasts is Listen to Lucy from the FT’s Lucy Kellaway and this week she has a great piece addressing how brands are responding on Twitter. Specifically how Starbucks responds to some Tweets about the brand. The piece is, like all her podcasts, humourous but with a serious message. And in this case I think its a message many brand would benefit from taking on board – how to respond to people on Twitter, or indeed how not to.

The case she discusses is of UK satirist, Armando Iannucci and a Tweet he made about Starbucks and the hygiene of their stores. She remarks on how the Starbucks UK MD is responding to this and similar Tweets about the brand and the regularity at which he is doing this.

Whilst I think that there is a real benefit of engaging with customers online in this way and it is important for brands to put in place a clear and thorough process for reacting and responding to mentions of their brand online. But, as we spend much of our time telling clients, the key is not to feel that every mention needs to be responded to. In fact in most cases mentions of your brand online do not merit a response.

Kellaway makes this point succinctly. The Starbuck UK MD, she says, should have other things to worry about than one message about the hygiene of his stores. Indeed, as she says, he should probably be more worried about the fact that he only finds out about this from a Tweet and not from his own staff.

There is a real danger with social media. Because it is easy to find mentions of your brand online there is a temptation to think that you need to respond to them. Kellaway’s point, and one that brands should take into account when planning their social media strategies, is that overall business strategy should not be driven by what is said on Twitter. In fact you should not build a process of reacting and responding that treat messages in social media in a different way from through other mediums.

The best approaches to customer service are not to have a special social media route to get your problems dealt with, but to feed social media into your existing channels. If you have a customer care team, it is they who should deal with mentions in social media (where they need to be dealt with). Social media mentions should not be elevated to a special level that received particular attention over and above how you deal with your other customers and their issues, comments and suggestions. You should integrate social media into your business not treat it as a special case.

You can Listen to Lucy on here

Starbucks is the most popular brand on Facebook when ranked by the number of people who ‘Like’ a brand (‘Fans’ as they used to be called). Over 7.5 million people like the coffee chain on Facebook, almost 2 million more than like the second most popular brand, Coca-Cola.

This data comes from Famecount which ranks brands (and people) based on the number of people who follow, like or friend them in social networks. It shows that food and drink brands are in each of the top five places, with fashion brands making up most of the remaining places in the top ten. Consumers are interested in what these brands are doing, or at least want to flag their interest in the brand or product on their own Facebook profile.

The top ten brands on Facebook (Global)

*Note: figures updated where relevant to be correct as of June 10 2010

Do the number of Facebook Likes matter?

Data like this is great for understanding user behaviour in Facebook. Showing us for which brands, and for which type of brands, users are more likely to click to say that they ‘Like’ it. However, for the brand, does the number of people who like you on Facebook matter? Not always.

The number of people who like you on Facebook is not the most important measure on Facebook. A more powerful measure is the number who engage with the brand. Liking a brand is an easy step and people do it for many reasons. At one end of the engagement spectrum because they want to hear from and exchange ideas with the brand. At the other end of the spectrum because they just want this ‘Like’ recorded as a badge on their Facebook profile. They may have no intention (or indeed desire) to engage at all with the brand.

And it is this engagement number that is of more use for brands. They want people who talk to them, like their posts and images, share their content and are active advocates of the brand. This means more than just ‘Liking’ the brand but doing something with it and engaging more deeply with it in Facebook. For any brand it is typically better for it to have 1 million fans, of which 5% engage with you on a regular basis, than to have 2 million fans with less than 1% engaging.

This number also shows the value of your presence in social media. It can be relatively easy for brands to build large numbers of ‘Likes’. It is less easy to get them to actually do something and to engage with you. But it is when they do that brands get real value.

So Facebook ‘Likes’ are important for brands, but actual engagement is even more important.

Over the last few months we’ve been conducting an in-depth review of the leading social media monitoring tools in conjunction with our sister company, FreshMinds Research.

We’ve compared how Alterian, Brandwatch, Biz360, Nielsen Buzzmetrics, Radian6, Scoutlabs and Sysomos performed when monitoring conversations about global coffee brand Starbucks. We compared over 19,000 online conversations and have written about the following topics:

• The basics of social media monitoring
• How to set up the tools (search strings, comparison challenges and coverage)
• The location of social media conversations
• Social media monitoring and duplication
• Data latency and the tool with the most up-to-date results
• The problem with automated sentiment

If you’ve enjoyed our posts so far you can find a more detailed analysis of all these topics and more in our final report – “Turning Conversations into Insights: a Comparison of Social Media Monitoring Tools”.

Download the final report from social media agency FreshNetworks

Sentiment analysis is a complex beast. Even for humans. Consider this statement: “The hotel room is on the ground floor right by the reception”. Is that neutral, or is it positive or negative? Well the answer is probably that it is different things to different people. If you want a high room with a view away from the noise or reception the review is negative. If have mobility issues and need a room with easy access it is positive. And for many people it would just be information and so neutral. Sentiment analysis is difficult even in human analysts in ambiguous or more complex situations. For social media monitoring tools it is also complicated and not always as simple or as clear-cut as we might like or expect.

As part of our review of social media monitoring tools we compared their automated sentiment analysis with the findings of a human analyst, looking at seven of the leading social media monitoring tools – Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos. And the outcome suggests that automated sentiment analysis cannot be trusted to accurately reflect and report on the sentiment of conversations online.

Understanding where automated sentiment analysis fails

On aggregate, automated sentiment analysis looks good with accuracy levels of between 70% and 80% which compares very favourably with the levels of accuracy we would expect from a human analyst. However this masks what is really going on here. In our test case on the Starbucks brand, approximately 80% of all comments we found were neutral in nature. They were mere statements of fact or information, not expressing either positivity or negativity. This volume is common to many brands and terms we have analysed we would typically expect that the majority of discussions online are neutral. These discussions are typically of less interest to a brand that wants to make a decision or perform an action on the basis of what is being said online. For brands the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails.

No tool consistently distinguishes between positive and negative conversations

When you remove the neutral statements, automated tools typically analyse sentiment incorrectly. In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative. In one case the accuracy was as low as 7% and the best tool was still only 48% accurate when compared to a human. For any brand looking to use social media monitoring to help them interact with and respond to positive or negative comments this is disastrous. More often than not, a positive comment will be classified as negative or vice-versa. In fact no tool managed to get all the positive statements correctly classified. And no tool got all the negative statements right either.

Why this failing matters to brands

This real failing of automated sentiment analysis can cause real problems for brands, especially if they are basing any internal workflow or processes on the basis of your social media monitoring. For example, imagine that you send all your negative conversations to your Customer Care team to respond to where relevant. If two-thirds (or maybe more) of the ‘negative’ conversations sent over are actually positive then this process starts to break down. Perhaps more importantly, a lot of the negative conversations will never make it to the Customer Care team in the first place (having been incorrectly classified as positive). Unhappy customers don’t get routed to the right people and don’t get their problems dealt with. The complete opposite of why many of our clients want to use social media monitoring in the first place.

So what can we do

As with any test, our experiment with the Starbucks brand won’t necessarily reflect findings for every brand and term monitored online. Our test was for a relatively short time period and we only put a randomised, but relatively representative, sample of conversations through human analysis. However, even with these limitations, we were surprised by the very high level of inaccuracy shown by the social media monitoring tools investigated. For businesses looking to make decisions or perform actions on the basis of a conversation being positive or negative this is potentially quite dangerous.

Of course there is much that can be done here and over time the tools can be trained to learn and to improve how they assess conversations about a given brand. But the overall message remains: automated sentiment analysis fails in its role of helping brands to make real decisions and to react to conversations about it online.

Read the other posts from our social media monitoring review 2010.

This is the fifth post in our Social Media Monitoring – 2010 review series. In it we’ll be looking at the issue of duplication, one reason for some of the seemingly large differences between the seven leading social media monitoring tools under investigation – Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos.

We saw from the first test in our review that the different tools produced markedly different volumes for the search terms we were using – all associated with Starbucks. The smallest number of conversations was found by Biz360 and the largest by Radian6 – over 11x the difference.

So which tools are reflecting better the conversations and discussions about Starbucks? Is bigger necessarily better? Are the tools with the largest number of conversations the best? We don’t think so.

The difference in volumes is striking. If you were using Radian6 you would get the impression that eleven times as many conversations were going on about Starbucks and related terms than if you were using Biz360. There are many reasons for this and bigger is not in this case necessarily better.

Think about the following: retweets, spam, signatures, adverts. Should these be counted in your study or not? Different tools treat them in different ways and so, as we saw with the issue of location, the actual number of conversations is not always as it seems.

Firstly, there’s the source of the conversation – who did is start with? Is there more than one conversation around the same topic or is someone copying it? Is this the same tweet that’s been retweeted or is it a new conversation? These distinctions are important. If a Tweet contains certain keywords it is often retweeted automatically many many times by ‘bots’ which search Twitter for these terms and automatically reposts them.

How do you deal with spam and adverts – taking blog posts titles or key terms from them and posting them on other sites. Should these be included or not in your counts of conversations? Are they real conversations if they are automatically taken from your site and used on others? Are they important to understand if they are being used in spam sites. Or indeed sites or an unsavoury nature (you know the ones I mean!)?

How many times should a comment in a forum thread be counted? If a comment is repeated in different places or on different pages should it be counted as a new conversation? Indeed if one person posts their comment on multiple sites to try to drive traffic or showcase their point of view should all of these instances be counted as a new conversation?

You need a social media monitoring tool that deals with these and other situations. The tools that identify the most conversations are often not the most useful or accurate. They may include a range of conversations that are irrelevant, spam or double-counted. Whereas any organisation looking to understand what people are saying about your brand online wants a more accurate portrayal of what is being said.

Bigger is not necessarily better and duplication is a serious issue that needs to be addressed in any social media monitoring.

Next…

More detail on these tests, and the results,  can be found in our final report which will be available to download on Friday 16th April. We’re also holding a free social media monitoring breakfast seminar on 15th April in London, where we’ll be presenting the findings of our report, as well as giving practical tips and advice about social media monitoring and the best way to analyse results. This event is now fully booked but you can follow the results live as they are announced on Twitter from 08.30 (London time) on Thursday by following #smm10.