FreshNetworks Blog

Keeping track of discussions surrounding your brand or competitors is crucial for successful social media monitoring and listening. One challenge is the sheer range and volume of conversations that take place online, and determining what to do with them.

Sentiment analysis is a difficult task to automate as irony and sarcasm can generate false results, affecting accuracy. Being able to identify what action a posting needs, if any, is also difficult, as spam or bot messages might drown out genuine users.

We spoke to Simon Campbell about the exciting approach SoDash has taken to social media monitoring. SoDash uses artificial intelligence which means the tool can be “trained” into determining the sentiment and category of a social media posting. This advanced approach to social media monitoring can potentially result in greater effectiveness at gathering intelligence from online conversations, and reacting to them appropriately.

What do you feel is the most accurate definition of a social media management tool?
I think the key is how you define “management”. There are a lot of social media monitoring and reporting tools, but I think the real value comes from engagement, which is something the traditional offerings in the market do quite poorly. I think the most accurate definition for the perfect social media management tool would be: A tool that helps you “monitor”, “filter”, “engage with” and “report on” social media in the most efficient way by automatically identifying opportunities and reducing workload through improved workflow.

Why do you think they are valuable to brands or businesses (ie, time savers etc)?

Tools by definition are there to make life easier and the good ones will cleverly filter all of the noise out there in social media, deliver just the relevant messages and provide a much improved work flow so that social media can be managed with minimal resource and maximum efficiency. Businesses need to be able to simply monitor and engage with their customers and prospects within social media as this represents how their brand is viewed and can relate directly to the bottom line.

What do you think is the most accurate way of tracking social media activity without using a tool?

It is a fairly laborious task without using any tools at all as it involved creating individual searches in things like Twitter and manually monitoring them, and setting up Google Alerts for numerous phrases and again, manually checking them all in the different places. There are some free tools that go some of the way to help monitor (such as TweetDeck) but they still rely on someone sitting in front of it all the time and it does not do anything especially clever except for pulling in the information to one place.

Explain how SoDash works and why it is an effective tool for social media management.

SoDash is a social media dashboard for brands and organisations to monitor and interact with the market. The reason it is unique owes to its artificial intelligence algorithms that learn what is important to your business through tagging. Once trained, it will automatically tag messages that are sales leads, positive or negative comments about your brand or competitors, deliver market information, ghost write and send responses and much more. Whilst some tools out there are good for monitoring social media, SoDash enables you to take control of social media and make it work for you with minimal resources.

What platforms does SoDash cover?

SoDash currently covers Twitter and Facebook with full monitoring of blogs, forums, YouTube, LinkedIn and others coming in September. We can currently also link to any specific source if requested. It is important to understand about engagement in the different platforms. Twitter is by far the most engaging, as it is an open platform. Facebook is great if you have a page with lots of fans that you need to manage but you cannot access and engage with private profiles.

How are you different from other social media management tools on the market?

SoDash is unique because it has in-built artificial intelligence which enables it to be trained to filter, recognise and tag messages based upon the criteria that is important to your business. Due to the artificial intelligence algorithms, it is also much more accurate when looking at things like sentiment analysis as again it is trained in relation to all aspects of the messages, including the structure, punctuation and person messaging, not just positive or negative words as with other tools. Essentially, other tools on the market have been developed to focus on monitoring whereas SoDash is built for engagement with monitoring as a given.

Who do you see as your main competitors?

Companies that use SoDash might also look at Radian6 or CoTweet. Both were built initially with monitoring and reporting in mind and, as with other tools on the market, they do not incorporate artificial intelligence so are reliant on manual filtering and responses. We have come across agencies who might continue to use something like Radian6 alongside SoDash although SoDash will soon be able to offer the full breadth of monitoring and reporting to cover all angles. Another of the features that customers are highlighting as a strong aspect of SoDash in comparison to other tools is the ease of use.

What sort of future developments can we expect to see from SoDash?

With the core functionality in place, the SoDash roadmap now focuses upon bringing on more channels/platforms and the automation of more specific reporting, especially to cover internal factors such as response times to messages (all of which can be provided now if requested). There are also some really cool advances that no one else has on the radar right now, but you will have to wait to see those!

As a social media agency,  FreshNetworks aims to bring you the best posts in social media, online communities, marketing and customer engagement online. In case you missed them, find below our top five posts in June.

1. Social media monitoring review 2010 – download the final report

Over the first few months of 2010 we conducted an in-depth review of the leading social media monitoring tools in conjunction with our sister company, FreshMinds Research. We compared how Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos performed when monitoring conversations about global coffee brand Starbucks, analysing over 19,000 online conversations.

Many thousands of you have already read our posts about the review and downloaded the final whitepaper. If you haven’t yet, you can find a more detailed analysis of all these tools and more in our final report – Turning Conversations into Insights: a Comparison of Social Media Monitoring Tools.

2. Why a museum is the UK’s top brand on Twitter

The Famecount dataset is, like much data, not perfect but it does highlight some surprises that we can all learn from. The brand it has as the top Twitter brand in the UK is one such surprise. Rather than the big FMCG, fashion and media firms they include in their brands ranking, the top UK brand on Twitter for them is a museum, @Tate.

There are some structural reasons why the Tate will attract followers. Twitter is great for events and experiences and a museum has lots of these. But the success and popularity of the Tate is about much more than this. It’s thanks to the way they use Twitter. In this post we look at the three simple characteristics of the way the Tate uses Twitter that all brands can learn from, and that contribute to their success.

3. The most beautiful tweet ever written (as judged by @stephenfry)

In June, Stephen Fry declared the most beautiful Tweet ever written at the Hay Festival. The winning tweet, from Marc MacKenzie, is a concise but informative tweet and perhaps is a great example of how people are using this new medium. But what makes this tweet the most beautiful ever written?

The beauty in Twitter, and in the tweets people send, is that they convey emotion, opinion, information and expression in a relatively short period, and they, broadly speaking, do so in public. Unlike other conversational forms, Twitter, even when you direct a tweet at a specific person, has a broader audience and often an audience you don’t know. And of course you only have 14o characters with which to express yourself. Marc MacKenzie’s tweet is a good example of this new medium – the audience is unclear and the tweet manages to convey information, opinion, belief and also humour. All in 140 characters.

4. The top ten brands on Facebook

Starbucks is the most popular brand on Facebook when ranked by the number of people who ‘Like’ a brand (’Fans’ as they used to be called). Over 7.5 million people like the coffee chain on Facebook, almost 2 million more than like the second most popular brand, Coca-Cola.

This data comes from Famecount which ranks brands (and people) based on the number of people who follow, like or friend them in social networks. It shows that food and drink brands are in each of the top five places, with fashion brands making up most of the remaining places in the top ten. Consumers are interested in what these brands are doing, or at least want to flag their interest in the brand or product on their own Facebook profile.

5. The problem with automated sentiment analysis

As part of our review of social media monitoring tools we compared their automated sentiment analysis with the findings of a human analyst, looking at seven of the leading social media monitoring tools – Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos. And the outcome suggests that automated sentiment analysis cannot be trusted to accurately reflect and report on the sentiment of conversations online.

In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative. In one case the accuracy was as low as 7% and the best tool was still only 48% accurate when compared to a human. For any brand looking to use social media monitoring to help them interact with and respond to positive or negative comments this is disastrous. More often than not, a positive comment will be classified as negative or vice-versa. In fact no tool managed to get all the positive statements correctly classified. And no tool got all the negative statements right either. Automated sentiment does not work, and for businesses relying on it can cause problems.

Sentiment analysis is a complex beast. Even for humans. Consider this statement: “The hotel room is on the ground floor right by the reception”. Is that neutral, or is it positive or negative? Well the answer is probably that it is different things to different people. If you want a high room with a view away from the noise or reception the review is negative. If have mobility issues and need a room with easy access it is positive. And for many people it would just be information and so neutral. Sentiment analysis is difficult even in human analysts in ambiguous or more complex situations. For social media monitoring tools it is also complicated and not always as simple or as clear-cut as we might like or expect.

As part of our review of social media monitoring tools we compared their automated sentiment analysis with the findings of a human analyst, looking at seven of the leading social media monitoring tools – Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos. And the outcome suggests that automated sentiment analysis cannot be trusted to accurately reflect and report on the sentiment of conversations online.

Understanding where automated sentiment analysis fails

On aggregate, automated sentiment analysis looks good with accuracy levels of between 70% and 80% which compares very favourably with the levels of accuracy we would expect from a human analyst. However this masks what is really going on here. In our test case on the Starbucks brand, approximately 80% of all comments we found were neutral in nature. They were mere statements of fact or information, not expressing either positivity or negativity. This volume is common to many brands and terms we have analysed we would typically expect that the majority of discussions online are neutral. These discussions are typically of less interest to a brand that wants to make a decision or perform an action on the basis of what is being said online. For brands the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails.

No tool consistently distinguishes between positive and negative conversations

When you remove the neutral statements, automated tools typically analyse sentiment incorrectly. In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative. In one case the accuracy was as low as 7% and the best tool was still only 48% accurate when compared to a human. For any brand looking to use social media monitoring to help them interact with and respond to positive or negative comments this is disastrous. More often than not, a positive comment will be classified as negative or vice-versa. In fact no tool managed to get all the positive statements correctly classified. And no tool got all the negative statements right either.

Why this failing matters to brands

This real failing of automated sentiment analysis can cause real problems for brands, especially if they are basing any internal workflow or processes on the basis of your social media monitoring. For example, imagine that you send all your negative conversations to your Customer Care team to respond to where relevant. If two-thirds (or maybe more) of the ‘negative’ conversations sent over are actually positive then this process starts to break down. Perhaps more importantly, a lot of the negative conversations will never make it to the Customer Care team in the first place (having been incorrectly classified as positive). Unhappy customers don’t get routed to the right people and don’t get their problems dealt with. The complete opposite of why many of our clients want to use social media monitoring in the first place.

So what can we do

As with any test, our experiment with the Starbucks brand won’t necessarily reflect findings for every brand and term monitored online. Our test was for a relatively short time period and we only put a randomised, but relatively representative, sample of conversations through human analysis. However, even with these limitations, we were surprised by the very high level of inaccuracy shown by the social media monitoring tools investigated. For businesses looking to make decisions or perform actions on the basis of a conversation being positive or negative this is potentially quite dangerous.

Of course there is much that can be done here and over time the tools can be trained to learn and to improve how they assess conversations about a given brand. But the overall message remains: automated sentiment analysis fails in its role of helping brands to make real decisions and to react to conversations about it online.

Read the other posts from our social media monitoring review 2010.