Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Central Logging with Open Source Software

divisionbyzero.net

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Central Logging with Open Source Software

#21

Post by nl »

zobzu wrote: post #16
reyjrar wrote: post #14

Earlier quoted context omitted.

We have a lot of in house expertise in ElasticSearch and choose it for a few reasons: 1) It's easier than Lucence/Sphinx to setup. 2) It's clustering support works out of the box and is so easy to configure it's not funny. ES is basically a usability wrapper around Lucene. I've heard that Sphinx is better for a single node configuration, it's faster and uses less resources, but clustering with Sphinx is apparently tr…

Ok. The one part I don't like in ES is being java and the trouble that usually goes with it (yeah, judging stuff like this is bad, I know :) For logging searching is the major item at large sites IMO (talking terabytes at least), when you're looking for all occurrences of "item x" over.. "the past week", it may take 1H "the past month", it may take 10-30H "the past year", uh, no, you don't do that. So you gotta use r…

The vast majority of open source "big-data" infrastructure is in Java (Hadoop, HBase, Cassandra, Solr, Elastic Search etc). It works pretty well.

I'm not sure what your question is, but I've experimented with loading netflow data in Solr and I'm averaging sub-2 second query times. That's on a laptop, with a couple of minutes of netflow (around 10Gb).

With proper indexing your search response time shouldn't increase lineally with your data size.

Re: Central Logging with Open Source Software

#22

Post by [deleted] »

ef4 wrote: post #6
Estragon wrote: post #2

This reminds me of something I've been wondering about since the Bitcoinica heist: how do people usually set up secure offline backups which can't be erased using the credentials on the backed-up server? I would probably do something with ssh authorized_keys if I had to make it from scratch, but are there obscure security/reliability risks, and tools which have already mitigated these risks for you?

You pull rather than push. The webserver has no credentials for accessing the backup server. Instead the backup server accesses the webserver. This strategy places higher trust on the backup server, but the backup server is easier to defend -- it only needs connectivity to a small number of other IPs.

[deleted]

Re: Central Logging with Open Source Software

#23

Post by nl »

shaggy wrote: post #20
ashayh wrote: post #11

Is there anyone who has used splunk and logstash/graylog2 on a large scale and can compare the two?

I haven't implemented logstash or graylog2 (yet) but I've implemented Splunk multiple times, at multiple companies since back in 2006 and it's a simply fantastic piece of software. Unless the pricing model has changed significantly since the last time I bought it, I don't see it as that expensive. The licensing model is based on daily volume indexed but the licenses are perpetual. It has all the features you'd expect…

Splunk licenses may be perpetual, but there is much less up-front cost with a yearly license. The license model is fair, though.

(Another happy Splunk user here)

Re: Central Logging with Open Source Software

#24

Post by nl »

Just get Splunk.

I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year.

The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source solution integrates all that.

If you want to compete with Splunk (something I've thought about a few times) then you need to match that. I'd estimate 2 developer for a year to build out those features on top of Solr or ES.

Re: Central Logging with Open Source Software

#25

Post by delinka »

Estragon wrote: post #2

This reminds me of something I've been wondering about since the Bitcoinica heist: how do people usually set up secure offline backups which can't be erased using the credentials on the backed-up server? I would probably do something with ssh authorized_keys if I had to make it from scratch, but are there obscure security/reliability risks, and tools which have already mitigated these risks for you?

1. Don't make your backup server accessible on the public internet. 2. Don't allow shell access from any server that does have access to the public internet. When your web server gets hacked, you don't want your assailant to have the ability to shell around in your network. 3. If you need shell access from outside the network, have a host specifically for this purpose and disallow password authentication (.ssh/authorized_keys indeed) 4. Backup server is write-only. I don't have a hard-and-fast method for enforcing this, but a process (or kernel module?) that watches for incoming backups, moves them immediately, and prevents overwriting existing files seems simple enough.

EDIT: lists on HN- doin it rong

Re: Central Logging with Open Source Software

#26

Post by beagle3 »

swombat wrote: post #3
Estragon wrote: post #2

This reminds me of something I've been wondering about since the Bitcoinica heist: how do people usually set up secure offline backups which can't be erased using the credentials on the backed-up server? I would probably do something with ssh authorized_keys if I had to make it from scratch, but are there obscure security/reliability risks, and tools which have already mitigated these risks for you?

The standard way used to be to use write-only media. For example, if you log to a server which writes the logs incrementally to a DVD writer, you can be fairly certain that the logs won't be erased...

That's only true if you have software that can mount arbitrary past sessions, which is rarely the case. When you put in a dvd, what gets mounted is the latest session -- which is supposed to also include all previous sessions, but doesn't have to.

Re: Central Logging with Open Source Software

#28

Post by packetslave »

nl wrote: post #24

Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…

Yes, except Splunk gets very expensive, very quickly if you want more than the free tier gives you (features or indexing volume). 500mb/day is not all that much when you start shoving everything under the sun into it (and once you've used it, you'll want everything available to it).

Re: Central Logging with Open Source Software

#29

Post by packetslave »

hcarvalhoalves wrote: post #19

A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.

Companies like Orbitz and Etsy that use the heck out of Graphite would be surprised to hear that it's "not really production ready"

Re: Central Logging with Open Source Software

#30

Post by nl »

packetslave wrote: post #28
nl wrote: post #24

Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…

Yes, except Splunk gets very expensive, very quickly if you want more than the free tier gives you (features or indexing volume). 500mb/day is not all that much when you start shoving everything under the sun into it (and once you've used it, you'll want everything available to it).

What do you class as very expensive?

We put multiple orders of magnitude more data than the free tier into Splunk, and it's still a lot cheaper than 2 developer-years.

It is true, though, that if the licensing was cheaper we'd put even more data into it.

Post reply on HN