Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logstash joins Elasticsearch

elasticsearch.com

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Logstash joins Elasticsearch

#11

Post by quicksilver03 »

benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

Another possible log shipper is nxlog, it compiles to native code and does not have any noticeable impact in terms of CPU or memory usage on my various low-end servers.

http://nxlog-ce.sourceforge.net/

Re: Logstash joins Elasticsearch

#12

Post by netvarun »

This is great news. Our centralized logging system at Semantics3 (https://semantics3.com) is built using Logstash+Kibana+Rsyslog+ElasticSearch. Running off a single EC2 large instance it has been been able to seamlessly aggregate and process logs from about 200-300 instances, processing on average of about 15 GB of log data. We hit some performance bottlenecks (particularly with elasticsearch) when our number of instances went beyond the 300 mark. But that should get fixed once we shard and distribute ElasticSearch.

Looking forward to some really tight integration between the Logstash, ES and Kibana.

Re: Logstash joins Elasticsearch

#13

Post by kossmoboleat »

benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

So heka does the same job as the central logstash server or does it offer a persistence layer, too?

Did you find a usable GUI or web interface to view the logs?

Re: Logstash joins Elasticsearch

#14

Post by vosper »

For people using this, I'd be interested to know what kind of throughput you're seeing and your cluster size - I'm trying to find something that can handle upwards of 100k small messages per second for a near-realtime analytics platform, and although this is a bit left-field (compared to Cassandra, HBase etc...) it could be a fit.

Re: Logstash joins Elasticsearch

#15

Post by kossmoboleat »

quicksilver03 wrote: post #11
benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

Another possible log shipper is nxlog, it compiles to native code and does not have any noticeable impact in terms of CPU or memory usage on my various low-end servers. http://nxlog-ce.sourceforge.net/

Do I have to buy the commercial version to get a web interface or GUI to analyze or browse the logs?

http://log4ensics.com/

Re: Logstash joins Elasticsearch

#16

Post by netvarun »

quicksilver03 wrote: post #11
benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

Another possible log shipper is nxlog, it compiles to native code and does not have any noticeable impact in terms of CPU or memory usage on my various low-end servers. http://nxlog-ce.sourceforge.net/

We've had no issues with rsyslog, which already comes packaged in ubuntu. Runs with no issues on micro instances on the AWS cloud, even at heavy workloads.

Re: Logstash joins Elasticsearch

#19

Post by zimbatm »

kossmoboleat wrote: post #13
benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

So heka does the same job as the central logstash server or does it offer a persistence layer, too? Did you find a usable GUI or web interface to view the logs?

Heka can output events into elasticsearch like logstash. Then you can also use Kibana for the UI.

Re: Logstash joins Elasticsearch

#20

Post by vosper »

benmmurphy wrote: post #8

logstash + elasticsearch are pretty amazing. however, if you are generating a high rate of log entries you may want to consider using mozilla hekad instead ( http://hekad.readthedocs.org/en/latest/ ). on our servers logstash was running around 20% CPU during quite periods while hekad was running around 1-2% CPU. while during busy periods i think logstash was going up to 100% CPU while hekad was sitting around 20-30%…

What kind of throughput are you seeing on your cluster, in terms of messages-per-second?
Post reply on HN