Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logstash joins Elasticsearch

elasticsearch.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Logstash joins Elasticsearch

#41

Post by kapilvt »

jordansissel wrote: post #37
vosper wrote: post #14

For people using this, I'd be interested to know what kind of throughput you're seeing and your cluster size - I'm trying to find something that can handle upwards of 100k small messages per second for a near-realtime analytics platform, and although this is a bit left-field (compared to Cassandra, HBase etc...) it could be a fit.

At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch. The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes). During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance…

it your taking feature requests, a plugin to archive to s3 would be really nice, for long term data retention. and more props to lumberjack-go port.

Re: Logstash joins Elasticsearch

#42

Post by don_draper »

JoachimSchipper wrote: post #2

I'm confused. Can someone explain to me why this is so obviously interesting, yet not worth discussing, that it stands - as of 2 hours after submission - at 75 points with zero comments? Honestly, I've never heard of either company, although I obviously wish them the best of luck. Am I just out of touch?

Log file analysis is a big deal. This one company (Splunk) alone is worth 5 billion dollars:

http://www.google.com/finance?q=splunk&ei=VwsdUqi2L5qglwON4g...

Re: Logstash joins Elasticsearch

#43

Post by quicksilver03 »

kossmoboleat wrote: post #15
quicksilver03 wrote: post #11

Earlier quoted context omitted.

Another possible log shipper is nxlog, it compiles to native code and does not have any noticeable impact in terms of CPU or memory usage on my various low-end servers. http://nxlog-ce.sourceforge.net/

Do I have to buy the commercial version to get a web interface or GUI to analyze or browse the logs? http://log4ensics.com/

Only if you want.

On my servers I use the open source version of nxlog to collect various logs and forward them to a central nxlog server, which in turn feeds logstash. Behind logstash I have configured elasticsearch as storage and I use kibana as a GUI to search and browse.

Re: Logstash joins Elasticsearch

#44

Post by capkutay »

noelwelsh wrote: post #31
capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

You probably want to include contact details somewhere. Your profile has none. (I'm not looking for a job right now, but I'm interested in the area.)

Ah..I thought the e-mail on my profile page shows up to other users. You can reach me at john.kutay@gmail.com

Re: Logstash joins Elasticsearch

#45

Post by aromasca »

Keyframe wrote: post #27
devopser wrote: post #24

Earlier quoted context omitted.

ElasticSearch itself should be very good now since they have moved to Lucene 4.0 which brought in lot of improvements in memory usage. I evaluated elasticsearch for RT analytics. It works wonders for point queries, where your result set is going to be small. Didn't work well for aggregate queries which need to scan lot of data. The biggest problem was field cache in Lucene. Almost all our queries needed to fo facetin…

I'll do extensive testing, but I need to scan a lot of data (aggregate basically). I'd be comfortable even with index size in multiples of data size if it delivered RT queries. Have you evaluated anything else?

we've done elasticsearch for RT analytics for about a year back in the 0.16 days, it was not stable to say the least for us and we've been using 7x m1.l AMIs at the time..

Today were aggregating the data in hadoop and bringing down only that to elasticsearch basically for a heavy use of the statistical facets.

Re: Logstash joins Elasticsearch

#47

Post by zisaacson »

capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

if you're looking to join Sumo Logic, please feel free to ping me ;)

Re: Logstash joins Elasticsearch

#48

Post by dbarlett »

mrmondo wrote: post #46

Both Logstash and elasticsearch are great - but they both suffer from the same flaw: they're a pain to deploy and it's a pain to manage their packages.

If you're crazy like me and run elasticsearch on Windows, I can't recommend elasticsearch-setup [1] highly enough. Combined with node discovery via the EC2 API [2], it's been rock-solid.

[1] http://ruilopes.com/elasticsearch-setup/

[2] http://www.elasticsearch.org/guide/reference/modules/discove...

Re: Logstash joins Elasticsearch

#49

Post by joshhart »

capkutay wrote: post #5

For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic. What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that…

We built a similar system log-searching system using SenseiDB at LinkedIn. Splunk was outrageously expensive.

It turns out that lucene based systems are pretty good at information retrieval and aren't shackled with all the OLTP requirements most databases have.

Post reply on HN