I'm currently evaluating elasticsearch and riak for rt analytics of large amount of data. Anyone has similar experience? Maybe even Cassandra, haven't touched it seriously yet.
Logstash joins Elasticsearch
21–30 of 60 posts
Re: Logstash joins Elasticsearch
#22I'm confused. Can someone explain to me why this is so obviously interesting, yet not worth discussing, that it stands - as of 2 hours after submission - at 75 points with zero comments? Honestly, I've never heard of either company, although I obviously wish them the best of luck. Am I just out of touch?
To understand the interest, you need to understand the moving parts:
Logstash is a sort of pipeline for data you want to log: you can define multiple inputs, transform/filter the data, and then define multiple outputs.
Example 1: read in your apache logs (input), attach geoip data (transform), and then push the resulting data to elasticsearch (output).
Example 2: read from syslog (input), grep on the input to ignore certain files (filter), then push to graphite for graphing (output).
you can have multiple inputs, multiple transforms/filters, and multiple outputs. You can also chain logstash instances together, so you can have "roll up" logs. Logstash itself is a bit heavy in terms of CPU/RAM (it is written in Java), so there are a few, lighter weight "shippers", and you can ship into a Redis instance to proxy events.
Elasticsearch is a java-based search engine with a great REST API and a _lot_ of features. It is built on top of Lucene. It doesn't have a built in GUI. It also scales out super easily.
Kibana is front-end to Elasticsearch, which lets search/visualize your log events.
Ok, those are the parts, why this is interesting: As other commenters have pointed out, this is a powerful combination for understanding your log data. You can ship logs from apps, services and hosts, visualize what's going on, search, correlate, etc.
Re: Logstash joins Elasticsearch
#23logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…
I don't know what bottlenecks you had when were observing high resource usage in logstash, but, in general, if there's a performance problem, it is a bug, and we can fix it.
The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly 31,000 events/sec parsing apache logs. With logstash 1.2.0.beta1, I can process 102,000 events/sec.
Processing speed will vary greatly by what you are doing with your events and it doesn't make sense to generalize performance characteristics globally, especially with a metric that, alone, doesn't really tell me much (cpu utilization).
If it's slow, it's a bug. We can fix it. :)
Further, you can use hekad with logstash and with elasticsearch (one or both together, it doesn't matter).
In terms of problems solved, logstash helps solve transport and real-time processing problems. In cases where the logstash agent is too resource intensive, the logstash community offers many alternatives on this site: http://cookbook.logstash.net/recipes/log-shippers/
The community (myself included) is very interested in helping logstash be a success for its users, so if you do see performance problems, things that behave weirdly, or anything strange, it's probably a bug, and we can fix it.
The short version of all of this is captured by the project principles listed in the logstash readme: https://github.com/logstash/logstash/#project-principles
<3
Re: Logstash joins Elasticsearch
#24I'm currently evaluating elasticsearch and riak for rt analytics of large amount of data. Anyone has similar experience? Maybe even Cassandra, haven't touched it seriously yet.
ElasticSearch itself should be very good now since they have moved to Lucene 4.0 which brought in lot of improvements in memory usage.
I evaluated elasticsearch for RT analytics. It works wonders for point queries, where your result set is going to be small. Didn't work well for aggregate queries which need to scan lot of data. The biggest problem was field cache in Lucene. Almost all our queries needed to fo faceting which had a big impact on field cache.
Also, I don't know about Riak, but in ES the joins you can do are very limited.
Re: Logstash joins Elasticsearch
#25this is the bestest news i've heard in a long long time ...
[deleted]
Re: Logstash joins Elasticsearch
#26This space is heating up. Cloudera is building a similar stack with Solr - http://www.cloudera.com/content/cloudera/en/campaign/introdu...
Re: Logstash joins Elasticsearch
#27I'm currently evaluating elasticsearch and riak for rt analytics of large amount of data. Anyone has similar experience? Maybe even Cassandra, haven't touched it seriously yet.
ElasticSearch itself should be very good now since they have moved to Lucene 4.0 which brought in lot of improvements in memory usage. I evaluated elasticsearch for RT analytics. It works wonders for point queries, where your result set is going to be small. Didn't work well for aggregate queries which need to scan lot of data. The biggest problem was field cache in Lucene. Almost all our queries needed to fo facetin…
I'll do extensive testing, but I need to scan a lot of data (aggregate basically). I'd be comfortable even with index size in multiples of data size if it delivered RT queries. Have you evaluated anything else?
Re: Logstash joins Elasticsearch
#28I love Logstash+Kibana+Elasticsearch. Holding 410 million log files in a 10 node cluster! Congratulations Jordan!
Re: Logstash joins Elasticsearch
#29logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…
What kind of throughput are you seeing on your cluster, in terms of messages-per-second?
on one of our machines that sends generates logs we do around 2400/s when the application is under heavy use. we have 9 machines that generates logs but they all generate different amounts. we mostly are using heka for generating stats from log files because we are too lazy to instrument the code and we have excellently detailed and formatted logs :) but we do have some logstash machines still pushing stuff to elastic search for low traffic applications we run.
we found that when using logstash even just for pushing stats to statsd it was not performing well enough. i've experimented with hekad pushing to elasticsearch on our staging cluster and performed well enough but we had weird problems showing up in nagios when we were using logstash+elasticsearch in production (checks were timing out even though we were seeing no degradation of performance on the servers). because of this it is quite difficult to get any kind of central log pushing into production. :(
Re: Logstash joins Elasticsearch
#30logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…
I don't know what bottlenecks you had when were observing high resource usage in logstash, but, in general, if there's a performance problem, it is a bug, and we can fix it. The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly…
that's good to hear. it could also have been a plugin that we were using that slowed things down. our log files are in csv so i wrote a plugin that uses ruby csv to parse lines and split them into key-value pairs based on String->List[String] hash we have. so it might have been the go csv parser has much better performance than the ruby csv parser.