Social media monitoring review 2010: Test 1 results
9th April 2010, 09:19 am by James Turner
This is our second post from the Social Media Monitoring – 2010 review series.
In it we’ll be giving you an insight into how we have set up the comaprison of tools (which proved rather a challenge) and the volume of online conversations that each social media monitoring tool was able to uncover.
Setting up the search string
We decided to use Starbucks as a test brand for our social media monitoring because it’s a global brand that is frequently discussed online. Also, the word ‘Starbucks’ doesn’t have any other meaning or use other than being a brand/company name.
As well as tracking the word ‘Starbucks’, we also tracked the phrase ‘Flat White’, a new addition to the Starbucks coffee range which launched in December 2009 . We also tracked their new ready brew coffee, ‘Via’, which was released in the autumn of last year in the US and in March 2010 in the UK. We wanted to see what impact this new development was having on online conversations about the brand . Finally, because Starbucks is associated with its ‘Reward Card’ and the phrase ‘Fair Trade’ we tracked these subjects too. To keep things fair we created a similar search string for each tool.
It is important to note that some tools are capable of more sophisticated search strings than others. So we were testing to the lowest-common-denominator in this sense.
Comparison challenges
Although the tools are very different, we wanted to try and evaluate them all as fairly as possible. Thus the tools were used ‘out-of-the-box’, as they come, for the fairest comparison. Again, there are limitations with this approach. Some of the more sophisiticated options offered in some tools are only relevant to more experienced users. And some providers (e.g. Neilsen) are set up to provide a much greater level of analyst support than, for example more technology focussed firms like Radian6.
Our sense for the market is that most firms are still learning the art of social media monitoring and that tools are often managed day-to-day by people with only limited training in how to use them in anger. This drove our approach to the research.
Coverage
As the tools all have different coverage, whether it’s for different media or markets, we set up the same filters for each tool to create a comparable ‘universe’ of conversations for Starbucks. Our test was carried out using only the English language and for the same time period on each tool.
Sentiment analysis
One of the areas we wanted to test was the sentiment analysis accuracy of each tool. In order to compare the automated sentiment (ie, sentiment that is coded automatically positive or negative by the tool) with our own analysis we had to extract the conversations and manually code them. Some tools don’t allow you to extract certain conversations, others do. Where we weren’t able to extract sentiment for some reason, we’ve marked the tool:
Number of conversations
The seven tools gave very different results when looking at overall conversations - the smallest number of conversations was found by Biz360 and the largest by Radian6 – over 11x the difference! But remember, more conversations is not necessarily better – there is often duplication.
*You can usually make arrangements with your account manager if you need more data.
Conversation types
When you compare the conversations by media type, again each of the tools shows quite a different result:
*Scoutlabs doesn’t allow you to extract Twitter conversations with sentiment. The tool does allow you to browse the latest twitter conversations though.
At this top-level, it’s clear the tools are each doing something quite different…
Next…
More detail on these tests, and the results, can be found in our final report which will be available to download on Friday 16th April. We’re also holding a free social media monitoring breakfast seminar on 15th April in London, where we’ll be presenting the findings of our report, as well as giving practical tips and advice about social media monitoring and the best way to analyse results. You can register for the event by clicking on the button below:
Read the other posts from our social media monitoring review 2010.
For social media agency support get in touch or follow us on Twitter.
Mark Evans:
We’re looking forward to seeing the complete report next week.
cheers, Mark
Mark Evans
9 April 2010, 12:46 pmDirector of Communications
Sysomos Inc.
@sysomos
Matt:
“Also, the word ‘Starbucks’ doesn’t have any other meaning or use other than being a brand/company name.”
Not quite: http://www.google.com/search?hl=en&q=starbucks+bsg&aq=f&aqi=&aql=&oq=&gs_rfai=
Also, it would be helpful to also show the individual media breakdowns as a count of raw messages for each type, instead of just as a share. Radian6 probably has the most messages from Twitter, but that’s not obvious from your chart.
9 April 2010, 1:17 pmJoseph Fiore:
@Matt: I was thinking exactly the same thing about Lt. Starbuck from BG’s movie/TV series
I’m also looking forward to the complete report. In in the interim, I wondered (and tweeted earlier, asking) if the depth of sourcing/conversation mining capability of these tools solely covers Twitter/Forums/Blogs? Or are these the sources you chose to report on, and excluded the others?
Joseph | RepuMetrix Inc.
9 April 2010, 1:42 pm@RepuTrac
Katie Morse:
Hi from Radian6!
Joseph – speaking about our coverage specifically, we do cover many more areas than just Twitter/Forums/Blogs. I’m happy to chat if you want more details.
James – I’m also looking forward to the full report. Thanks for including us in this review!
Katie Morse
9 April 2010, 5:23 pmCommunity Manager | Radian6
@misskatiemo
Nilesh:
James, thanks for including in Sysomos.
As you pointed out, the result count is often an incorrect measure of how “good” the coverage is. Over 50% of social web is spam, something that skews all analytics and metrics. Here at Sysomos, we pride ourselves in having robust spam filters to ensure data quality as opposed to plain quantity.
Both our products, MAP and Heartbeat, work across hundreds of geographies and languages. This includes segmentation by France, UK and US (I am happy to show more in detail).
Geography functionality again is a tricky issue to access. For example, while IP-based location is easy to get, it is 90% times incorrect. We hence do not use IP, but our own text analysis to infer details on location, age, gender, etc..
Also please note that the Sysomos database goes back 4 years in time.
I look forward to seeing your complete report. If you have any questions, please feel free to reach out to me or tweet to us at @sysomos
thanks,
9 April 2010, 6:58 pmNilesh
Maria Ogneva:
Hi James,
Thanks for including us in the study. I wanted to address the issue of coverage – the result is not consistent with what I am seeing. May I inquire what dates you were considering and what search terms filters you were using? I just ran a report for “Starbucks” for the past 30 days, and got 14.5K mentions across: blogs, Twitter, forums / discussion boards and online news.
If you form your search string as “‘Starbucks’ AND ‘Flat White’ AND ‘Via’ AND ‘Reward Card’ AND ‘Fair Trade’”, you will get articles that include ALL of the above terms in the same mention, whether it’s a tweet, blogpost or a forum discussion. This will narrow down your mentions drastically – that’s a lot of stuff for one tweet.
However, if you set up your terms as “‘Starbucks’ OR ‘Flat White’ OR ‘Via’ OR ‘Reward Card’ OR ‘Fair Trade’”, the system will show you all the articles that have one or more of these items, which will increase your pool to larger than 14k articles.
Additionally, if you want there to be “Starbucks” AND one of the other terms, you can set it up as “Starbucks” AND ["flat white" OR "VIA" OR "reward card" OR "fair trade"] – that will give you a number of mentions somewhere between the two versions above.
Please contact me offline so that we can make sure we are looking at the same thing – my email is mogneva (at) biz360 (dot) com.
- Maria Ogneva, social media director, Biz360
9 April 2010, 8:22 pmConnie Bensen:
Hi James,
Kudos on taking on such a big project!
Every tool has it’s idiosyncracies. As you pointed out each has a bit different way to set up searches. But it goes further than that.
There is no standard for sentiment, filtering or calculating popularity (influence). But that is the beauty of this young industry. Each tool is using it’s own propietary algorithms.
Alterian SM2 doesn’t bring in duplicates and everything that is filtered is available for the user to view & decide on. Filter strengths vary from tool to tool. The one thing that needs to be considered is not necessarily having the ‘most’ results, but rather the most relevant. And that question is dependent on what your objectives are.
Someone else has mentioned this, but the definition of ‘blog’ also varies. Does one consider multi-authored blogs such as TechCrunch and Mashable as a blog? SM2 has them under the category of Media Types-Other. And sometimes if the technology can’t identify that it is a blog then it is added to the ‘Media Types-Other’ category.
We look forward to the next installments in the series.
9 April 2010, 9:29 pmConnie
Director of Community, Alterian
@cbensen
James Turner:
Thanks for the comments guys. Our research is intended to compare the tools without prejudice or prior knowledge of social media monitoring.
We’ll go through all the steps required to for a user the results more meaningful in the full report; we hope it educates people considering the tools!
Also as it’s such a fast changing market, I know that there will be some capabilities in newer releases that address some of our findings. We’d be delighted for you tom contribute to the posts to make them up-to-date.
11 April 2010, 11:05 amJames Turner:
Thanks Nilesh, I’ll check with our Analysts about the date range that was available to us.
11 April 2010, 3:10 pmSocial media monitoring review 2010: Why the location of social media conversations is not always what it seems | FreshNetworks social media agency blog:
[...] Our blog « Social media monitoring review 2010 – Test 1 results [...]
11 April 2010, 3:45 pmMatt:
“Our research is intended to compare the tools without prejudice or _prior knowledge of social media monitoring_.”
11 April 2010, 4:00 pmWho is going to be purchasing access to any of these tools if they don’t know anything about SMM? It also seems like it would bias the results against any level of service and support beyond the bare minimum. The report is called “Social Media Monitoring Review,” not “Social Media Monitoring for Dummies.”
Mikko Kotila:
Very interesting research. Could you share a little more about the methodology/sample:
- did you just use ‘starbucks’ as the search term?
11 April 2010, 6:02 pm- did you confirm relevancy of the results?
- what was the time frame you collected data from?
James Turner:
Matt
Thanks for your comments. I’m pleased you’re looking to FreshNetworks for an expert view! We have many clients that really value our education and assistance in this market.
As a few of the posts have said, this is still a young market and I’m hoping the research benefits the widest possible audience.
We’ll raise the ‘services’ element of the tools in the report so rest assured we’re not trying to bias the results.
(I might rename the introduction section of the report to ‘Social Media Monitoring for Dummies’, thanks!)
11 April 2010, 8:20 pmGiles Palmer:
Hi there James
Good start to the whole comparison.
I have a few comments
1. Can you publish the actual queries you used for each of the systems?
2. Is this for global English only? Can you select only English in all the systems?
3. Twitter differences look odd – can you clarify the twitter search terms used (you might need to speak to the system vendors to check what happens behind the scenes with Twitter)
Finally, you say Brandwatch only goes back 2 months, but that’s the account you have – we have an archive of more than a year
Cheers
13 April 2010, 9:04 amgiles
Social media monitoring and duplication (duplication, duplication!) | FreshNetworks social media agency blog:
[...] saw from the first test in our review that the different tools produced markedly different volumes for the search terms we [...]
13 April 2010, 9:40 pmBUMAGEXCHANGE:
[...] http://www.freshnetworks.com/blog/2010/04/social-media-monitoring-review-2010-test-1-results/ for the full [...]
18 April 2010, 8:33 pmSocial Media Monitoring Tools – 2010 Review (intro) | FreshNetworks social media agency blog:
[...] An overview of our results [...]
21 April 2010, 3:01 pmFreshNetworks Blog: Top five posts in April | FreshNetworks social media blog:
[...] 1. Social media monitoring review 2010: Test 1 results [...]
10 May 2010, 4:06 pmmatt:
you forgot to metion eCairn. They are more affordable tha most of those listed.
11 May 2010, 8:42 pmSocial media monitoring review 2010 – download the final report | FreshNetworks social media blog:
[...] How to set up the tools (search strings, comparison challenges and coverage) [...]
1 June 2010, 10:01 pmSocial Media Monitoring Review 2010 | Digital Asset Management:
[...] How to set up the tools (search strings, comparison challenges and coverage) [...]
3 June 2010, 4:29 amTraining Media Review | Top Social Media Optimization:
[...] Media Review "Training Media Review" image from: http://www.freshnetworks.com/blog/2010/04/social-media-monitoring-review-2010-test-1-results/ "Training Media Review" on TwitterBack with @wardssolicitors this morning. More social media [...]
13 July 2010, 4:01 pmSeven Social Media Monitoring Tools Reviewed, Studied, with Free Download:
[...] How to set up the tools (search strings, comparison challenges and coverage) [...]
1 October 2010, 7:48 am