Is there anyone who has used splunk and logstash/graylog2 on a large scale and can compare the two?
Central Logging with Open Source Software
11–20 of 51 posts
Re: Central Logging with Open Source Software
#12Shameless self promotion but I wrote a blog post about using Logstash, ElasticSearch, and Kibana in production for not only capturing syslog-ng messages but multi-lined Java stack trace errors via log4j.
http://blog.tagged.com/2012/05/grabbing-full-java-stack-trac...
Re: Central Logging with Open Source Software
#13I found this interesting.
Do you have any benchmarking of the speed of ES vs everything else?
Re: Central Logging with Open Source Software
#14I found this interesting. Do you have any benchmarking of the speed of ES vs everything else?
We have a lot of in house expertise in ElasticSearch and choose it for a few reasons: 1) It's easier than Lucence/Sphinx to setup. 2) It's clustering support works out of the box and is so easy to configure it's not funny.
ES is basically a usability wrapper around Lucene. I've heard that Sphinx is better for a single node configuration, it's faster and uses less resources, but clustering with Sphinx is apparently tricky.
The other competitor, which I have no experience with is Solr, but this write-up http://karussell.wordpress.com/2011/05/12/elasticsearch-vs-s... gives an overview of the ES advantages over Solr.
I'm not a Full-Text Search expert, but a number of really smart people at my company evaluated a number of them for one of the most critical pieces of our production site and they chose ElasticSearch.
Re: Central Logging with Open Source Software
#15Is there anyone who has used splunk and logstash/graylog2 on a large scale and can compare the two?
+1 - Kind of the reason I started this blog post.. Besides reporting (which I know my setup lacks), from a techie perspective, what would this setup lack that Splunk provides.
Follow-up, is it _honestly_ worth the licensing to get those features?
Re: Central Logging with Open Source Software
#16I found this interesting. Do you have any benchmarking of the speed of ES vs everything else?
We have a lot of in house expertise in ElasticSearch and choose it for a few reasons: 1) It's easier than Lucence/Sphinx to setup. 2) It's clustering support works out of the box and is so easy to configure it's not funny. ES is basically a usability wrapper around Lucene. I've heard that Sphinx is better for a single node configuration, it's faster and uses less resources, but clustering with Sphinx is apparently tr…
Ok. The one part I don't like in ES is being java and the trouble that usually goes with it (yeah, judging stuff like this is bad, I know :)
For logging searching is the major item at large sites IMO (talking terabytes at least), when you're looking for all occurrences of "item x" over..
"the past week", it may take 1H
"the past month", it may take 10-30H
"the past year", uh, no, you don't do that.
So you gotta use ranges, but it's often hard to guess and you end up missing many log entries just because you don't have the time to search through them.
(obviously loading gigabytes of indexed data takes a while "physically speaking" anyway. I'm guessing ES can distribute the load tho, much like a web search engine does)
Re: Central Logging with Open Source Software
#17Anyone doing this at scale using Scribe/Flume with HDFS and Hive/Pig/MapReduce?
Re: Central Logging with Open Source Software
#18Is there anyone who has used splunk and logstash/graylog2 on a large scale and can compare the two?
+1 - Kind of the reason I started this blog post.. Besides reporting (which I know my setup lacks), from a techie perspective, what would this setup lack that Splunk provides. Follow-up, is it _honestly_ worth the licensing to get those features?
No on the license front. Splunk is way too pricey, IMO.
Thanks for posting this! I've been eyeballing logstash for a while but had not run across Kibana's UI. More fun reading ahead.
Re: Central Logging with Open Source Software
#19A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.
Re: Central Logging with Open Source Software
#20Is there anyone who has used splunk and logstash/graylog2 on a large scale and can compare the two?
I haven't implemented logstash or graylog2 (yet) but I've implemented Splunk multiple times, at multiple companies since back in 2006 and it's a simply fantastic piece of software. Unless the pricing model has changed significantly since the last time I bought it, I don't see it as that expensive. The licensing model is based on daily volume indexed but the licenses are perpetual. It has all the features you'd expect built-in and the reporting/searching/alerting and other integrations are the best that I've seen. I am curious to see what can be accomplished with open source software though.