Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logstash joins Elasticsearch

elasticsearch.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Logstash joins Elasticsearch

#31

Post by noelwelsh »

capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

You probably want to include contact details somewhere. Your profile has none. (I'm not looking for a job right now, but I'm interested in the area.)

Re: Logstash joins Elasticsearch

#32

Post by ape4 »

jpgvm wrote: post #7
JoachimSchipper wrote: post #2

I'm confused. Can someone explain to me why this is so obviously interesting, yet not worth discussing, that it stands - as of 2 hours after submission - at 75 points with zero comments? Honestly, I've never heard of either company, although I obviously wish them the best of luck. Am I just out of touch?

Logstash + Elasticsearch + Kibana is the biggest thing in opensource operational tools since Nagios.

Maybe I a too traditional... but I like KISS when it comes to this kinds of thing.

Re: Logstash joins Elasticsearch

#33

Post by adamlj »

benmmurphy wrote: post #30
jordansissel wrote: post #23

Earlier quoted context omitted.

I don't know what bottlenecks you had when were observing high resource usage in logstash, but, in general, if there's a performance problem, it is a bug, and we can fix it. The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly…

that's good to hear. it could also have been a plugin that we were using that slowed things down. our log files are in csv so i wrote a plugin that uses ruby csv to parse lines and split them into key-value pairs based on String->List[String] hash we have. so it might have been the go csv parser has much better performance than the ruby csv parser.

It sounds very likely that it was Ruby that consumed so much of the CPU.

Re: Logstash joins Elasticsearch

#34

Post by eikenberry »

Keyframe wrote: post #21

I'm currently evaluating elasticsearch and riak for rt analytics of large amount of data. Anyone has similar experience? Maybe even Cassandra, haven't touched it seriously yet.

When looking at elasticsearch remember that it is a secondary datastore used for indexing, searching, querying, etc. Not for a durable long term data store. You'll need another system for that along with a system for re-feeding elasticsearch.

Re: Logstash joins Elasticsearch

#35

Post by davidy123 »

clarkdave wrote: post #3

Logstash, Elasticsearch and Kibana are just fantastic. After being unsatisfied with a whole bunch of Logging As A Service providers (I tried loggly.com, logentries.com and splunkstorm.com) I spent an afternoon setting up Logstash and co and couldn't be happier. There's a neat demo of Kibana here: http://demo.kibana.org/#/dashboard/elasticsearch/Logstash%20... The only thing that isn't fully baked in with this stack i…

it is worth noting there is a Node implementation of logstash.

https://github.com/bpaquet/node-logstash

It is "logstash compatible" (at ElasticSearch, so it works with Kibana) and in my experience very easy to work with, and probably a lot lighter weight than the JRuby version.

Re: Logstash joins Elasticsearch

#36

Post by cmgreen »

quicksilver03 wrote: post #11
benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

Another possible log shipper is nxlog, it compiles to native code and does not have any noticeable impact in terms of CPU or memory usage on my various low-end servers. http://nxlog-ce.sourceforge.net/

I used to use nxlog to collect Windows AD logs to a Linux server. Often, it would end up deadlocking (2-3/week) on the Windows Side and would stop shipping the logs. It was very useful to have the logs but I'm very glad we were able to replace it.

Re: Logstash joins Elasticsearch

#37

Post by jordansissel »

vosper wrote: post #14

For people using this, I'd be interested to know what kind of throughput you're seeing and your cluster size - I'm trying to find something that can handle upwards of 100k small messages per second for a near-realtime analytics platform, and although this is a bit left-field (compared to Cassandra, HBase etc...) it could be a fit.

At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch.

The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes).

During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance.

In terms of capacity planning, this means that we could have a 9x increase in normal event load and still not need to grow the cluster's processing capacity.

Persistent storage is another story. We stored about 300GB/day of events, getting us roughly 45 days of data retention before we would run out of space (2TB * 7 nodes / 300gb/day; roughly 45 days). I'm working on improving storage efficiency of logstash and elasticsearch, too, so retention should improve greatly in the long term.

For other experiences, it's useful to invoke the community and ask what others are done - the #logstash irc channel on freenode is very active as is the logstash-users@googlegroups.com mailling list.

Hope this helps!

Re: Logstash joins Elasticsearch

#38

Post by reinhardt »

eikenberry wrote: post #34
Keyframe wrote: post #21

I'm currently evaluating elasticsearch and riak for rt analytics of large amount of data. Anyone has similar experience? Maybe even Cassandra, haven't touched it seriously yet.

When looking at elasticsearch remember that it is a secondary datastore used for indexing, searching, querying, etc. Not for a durable long term data store. You'll need another system for that along with a system for re-feeding elasticsearch.

Is this really the case? I realize you probably can't express the equivalent of arbitrary SQL queries in elasticsearch but what would prevent it from being used as a primary NoSQL datastore?

Re: Logstash joins Elasticsearch

#39

Post by markelliot »

jordansissel wrote: post #37
vosper wrote: post #14

For people using this, I'd be interested to know what kind of throughput you're seeing and your cluster size - I'm trying to find something that can handle upwards of 100k small messages per second for a near-realtime analytics platform, and although this is a bit left-field (compared to Cassandra, HBase etc...) it could be a fit.

At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch. The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes). During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance…

What's the raw scale of input data for your 300GB/day of stored events? (assuming that's 300GB on disk stored in Elasticsearch)

Re: Logstash joins Elasticsearch

#40

Post by jordansissel »

markelliot wrote: post #39
jordansissel wrote: post #37

Earlier quoted context omitted.

At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch. The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes). During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance…

What's the raw scale of input data for your 300GB/day of stored events? (assuming that's 300GB on disk stored in Elasticsearch)

I think it was roughly 300 million events/day (1kb per event). There is some overhead incurred by logstash (turning a log into json, parsing it into fields) and by elasticsearch (analyzing/indexing data).

In practical terms, and by way of example, a plain text apache access log, fully parsed by logstash (breaking out fields, etc), has historically bloated by quite a bit (6.2x I have measured). Lately, however, with improvements to logstash, better default settings, and elasticsearch being awesome, the 'inflation' number gets down to something more like 1.5x - which isn't bad considering all the awesome you get with it.

Long term, I am working towards making the 'raw data to stored data' ratio something less than 1x.

You can see some experiments I did a year ago on this: https://github.com/jordansissel/experiments/blob/master/elas...

I will repeat these experiments after the next release of logstash, and I expect storage ratios to improve significantly.

Post reply on HN