Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logstash joins Elasticsearch

elasticsearch.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Logstash joins Elasticsearch

#51

Post by eikenberry »

reinhardt wrote: post #38
eikenberry wrote: post #34

Earlier quoted context omitted.

When looking at elasticsearch remember that it is a secondary datastore used for indexing, searching, querying, etc. Not for a durable long term data store. You'll need another system for that along with a system for re-feeding elasticsearch.

Is this really the case? I realize you probably can't express the equivalent of arbitrary SQL queries in elasticsearch but what would prevent it from being used as a primary NoSQL datastore?

Primary reason at the moment is that elasticsearch currently has no backup/long term durability story. They plan on something for 1.0, but that doesn't help with the current situation.

Also we've suffered data loss on several occasions with elasticsearch. This has been getting better but is still a concern. Having the external long term datastore and a handy import method made these small hiccups.

Lastly IMO elasticsearch works best, particularly when working with log like data, as a rolling window view into the data. This keeps your elasticsearch cluster to a reasonable size (saving $$) while keeping the ability to re-load old data for exploration when you need it.

Re: Logstash joins Elasticsearch

#52

Post by jordansissel »

mrmondo wrote: post #46

Both Logstash and elasticsearch are great - but they both suffer from the same flaw: they're a pain to deploy and it's a pain to manage their packages.

With logstash, I aim to make it as easy to deploy as possible. That is, in part, why the releases are self-contained jar files with all depenencies built-in (except for java itself). We also started working on shipping rpm/deb packages with recent releases.

Like I always say, if it's hard to use or appears to have major flaws or pains, it's a bug, and we can fix it. Let us know! :)

Re: Logstash joins Elasticsearch

#53

Post by zwily »

ape4 wrote: post #32
jpgvm wrote: post #7

Earlier quoted context omitted.

Logstash + Elasticsearch + Kibana is the biggest thing in opensource operational tools since Nagios.

Maybe I a too traditional... but I like KISS when it comes to this kinds of thing.

Logstash, ES, and Kibana actually are more KISS than any other log searching setup I've tried.

Except for grep of course.

Re: Logstash joins Elasticsearch

#54

Post by stock_toaster »

davidy123 wrote: post #35
clarkdave wrote: post #3

Logstash, Elasticsearch and Kibana are just fantastic. After being unsatisfied with a whole bunch of Logging As A Service providers (I tried loggly.com, logentries.com and splunkstorm.com) I spent an afternoon setting up Logstash and co and couldn't be happier. There's a neat demo of Kibana here: http://demo.kibana.org/#/dashboard/elasticsearch/Logstash%20... The only thing that isn't fully baked in with this stack i…

it is worth noting there is a Node implementation of logstash. https://github.com/bpaquet/node-logstash It is "logstash compatible" (at ElasticSearch, so it works with Kibana) and in my experience very easy to work with, and probably a lot lighter weight than the JRuby version.

oh neat. thanks for the link -- had never run across it before.

Re: Logstash joins Elasticsearch

#55

Post by vosper »

jordansissel wrote: post #37
vosper wrote: post #14

For people using this, I'd be interested to know what kind of throughput you're seeing and your cluster size - I'm trying to find something that can handle upwards of 100k small messages per second for a near-realtime analytics platform, and although this is a bit left-field (compared to Cassandra, HBase etc...) it could be a fit.

At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch. The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes). During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance…

Thanks for the detailed reply! My use case is a stream of distinct, ordered events identified by a UUID, where the first event makes up about 95% of the volume; that is, we don't often receive subsequent events with the same UUID.

The initial event and any subsequent ones tend to arrive close together in time, so the challenge is to find something that can handle a high insertion rate, a relatively low update rate, while providing fast aggregations suitable for charting in a web-frontend. In Riak, Couchbase or HyperDex we'd use a secondary index and do our own math, but Elasticsearch is attractive because it appears to support the kind of queries we're interested in out of the box, in addition to having a good reported write-rate.

Persistence is less of an issue, because after a short period of time (a couple of hours) we would summarise the events into our analytics DB (Infobright) and so we could set a TTL on the data stored in Elasticsearch.

Again, thanks for the response and I'll check out the mailing-list and IRC channel.

Edit: Grammar

Re: Logstash joins Elasticsearch

#56

Post by devopser »

Keyframe wrote: post #27
devopser wrote: post #24

Earlier quoted context omitted.

ElasticSearch itself should be very good now since they have moved to Lucene 4.0 which brought in lot of improvements in memory usage. I evaluated elasticsearch for RT analytics. It works wonders for point queries, where your result set is going to be small. Didn't work well for aggregate queries which need to scan lot of data. The biggest problem was field cache in Lucene. Almost all our queries needed to fo facetin…

I'll do extensive testing, but I need to scan a lot of data (aggregate basically). I'd be comfortable even with index size in multiples of data size if it delivered RT queries. Have you evaluated anything else?

We also checked mongodb. We dropped it mainly because index size was getting too big.

If your data is read-only then Cloudera Impala is worth a try. It's really fast.

Re: Logstash joins Elasticsearch

#57

Post by ook »

JoachimSchipper wrote: post #2

I'm confused. Can someone explain to me why this is so obviously interesting, yet not worth discussing, that it stands - as of 2 hours after submission - at 75 points with zero comments? Honestly, I've never heard of either company, although I obviously wish them the best of luck. Am I just out of touch?

I think any of Logging/Monitoring/Metrics at scale can be thought of as Chicken & Egg problems.

They are important, hard to do well and have a bad habit of only causing issues which swallow engineering time when you are firefighting furiously trying to scale core services.

That's why as someone pointed out separately Spunk is a $5Bn company and people who have had these problems previously are very excited by this news.

(It's also why StatsD&Graphite/OpenTSDB, Riemann/Sensu etc etc are all super interesting)

Re: Logstash joins Elasticsearch

#58

Post by nasalgoat »

capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

The one thing Splunk has going for it over ES is the amount of resources it requires to work at scale.

I needed 12 ES boxes for every one Splunk box to handle the 100MB/day log load of my system, and even then they ran at a high load and searches often failed, and in some cases it took hours for the indexer to catch up.

Re: Logstash joins Elasticsearch

#59

Post by Keyframe »

devopser wrote: post #56
Keyframe wrote: post #27

Earlier quoted context omitted.

I'll do extensive testing, but I need to scan a lot of data (aggregate basically). I'd be comfortable even with index size in multiples of data size if it delivered RT queries. Have you evaluated anything else?

We also checked mongodb. We dropped it mainly because index size was getting too big. If your data is read-only then Cloudera Impala is worth a try. It's really fast.

I was looking at Impala (Cassandra) as well as keeping an eye on Drill progress. My data is write only in ETL stage so it seems it could be the right way. Lots of testing ahead! - thanks

Re: Logstash joins Elasticsearch

#60

Post by jordansissel »

nasalgoat wrote: post #58
capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

The one thing Splunk has going for it over ES is the amount of resources it requires to work at scale. I needed 12 ES boxes for every one Splunk box to handle the 100MB/day log load of my system, and even then they ran at a high load and searches often failed, and in some cases it took hours for the indexer to catch up.

This experience sounds especially bad. Sorry about that.

As mentioned in another comment in this post, I was doing 300gigs of data per day with an elasticsearch cluster size of 7 elasticsearch nodes (16 cores & 16gb ram per node) and load was around 5-10% cpu utilization.

100MB/day is pretty small in terms of log data, I think. If you attempt this again, please invoke the community (elasticsearch's is great!) and see if we can assist you in figuring out what's busted.

Post reply on HN