Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Central Logging with Open Source Software

divisionbyzero.net

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Central Logging with Open Source Software

#31

Post by Estragon »

delinka wrote: post #25
Estragon wrote: post #2

This reminds me of something I've been wondering about since the Bitcoinica heist: how do people usually set up secure offline backups which can't be erased using the credentials on the backed-up server? I would probably do something with ssh authorized_keys if I had to make it from scratch, but are there obscure security/reliability risks, and tools which have already mitigated these risks for you?

1. Don't make your backup server accessible on the public internet. 2. Don't allow shell access from any server that does have access to the public internet. When your web server gets hacked, you don't want your assailant to have the ability to shell around in your network. 3. If you need shell access from outside the network, have a host specifically for this purpose and disallow password authentication (.ssh/author…

Thanks. Great advice.

Re: Central Logging with Open Source Software

#32

Post by flyt »

hcarvalhoalves wrote: post #19

A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.

You set it up once. After you get through that hour long process you've got a great system that is battle tested at many large shops, with plenty of domain knowledge on the Internet.

Re: Central Logging with Open Source Software

#34

Post by heretohelp »

nl wrote: post #24

Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…

Don't listen to this guy if you own more than a couple servers.

Re: Central Logging with Open Source Software

#36

Post by nl »

heretohelp wrote: post #34
nl wrote: post #24

Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…

Don't listen to this guy if you own more than a couple servers.

Why do you say that?

We have a couple of datacenters, so yes, we have more than a couple of servers.

Re: Central Logging with Open Source Software

#37

Post by zobzu »

nl wrote: post #21
zobzu wrote: post #16

Earlier quoted context omitted.

Ok. The one part I don't like in ES is being java and the trouble that usually goes with it (yeah, judging stuff like this is bad, I know :) For logging searching is the major item at large sites IMO (talking terabytes at least), when you're looking for all occurrences of "item x" over.. "the past week", it may take 1H "the past month", it may take 10-30H "the past year", uh, no, you don't do that. So you gotta use r…

The vast majority of open source "big-data" infrastructure is in Java (Hadoop, HBase, Cassandra, Solr, Elastic Search etc). It works pretty well. I'm not sure what your question is, but I've experimented with loading netflow data in Solr and I'm averaging sub-2 second query times. That's on a laptop, with a couple of minutes of netflow (around 10Gb). With proper indexing your search response time shouldn't increase l…

loading 10gb on a traditional hdd takes more than 2s (that's 5gb/s read speed. Nice hard drive.). your data is either in ram and you've a lot of ram, either, it's just not 2s, or its not a 10gb index.

And i'm talking 100gb+ indexes ;-)

Obviously 2min of netflow data ain't much. I would want to see the result over 200h (or more) of netflow data, for example

Re: Central Logging with Open Source Software

#38

Post by mkramlich »

hcarvalhoalves wrote: post #19

A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.

I worked at Orbitz as one of the very first users of Graphite, and yes Orbitz was the first corporate user of it, and it was used heavily in production as far back as say 2008. Graphite is (or at least was then) production ready. It does have documentation but it doesn't need much.

Re: Central Logging with Open Source Software

#39

Post by heretohelp »

nl wrote: post #36
heretohelp wrote: post #34

Earlier quoted context omitted.

Don't listen to this guy if you own more than a couple servers.

Why do you say that? We have a couple of datacenters, so yes, we have more than a couple of servers.

In a situation where one has that much money to blow on something so limited, virtually anything would've sufficed.

We did a trivial test of Splunk at my last company, it's extremely expensive and it's very easy to bump into its limitations. We were able to wreck the poor Splunk server with some rather sundry queries into a dataset that shouldn't have been that big of a deal. Issues that we took back to the company and didn't get any real answer on.

Its popularity leads me to surmise that there is still a lot of money to be made in solving mundane problems. (Which is good news if you're a product-minded programmer)

Re: Central Logging with Open Source Software

#40

Post by mkramlich »

packetslave wrote: post #29
hcarvalhoalves wrote: post #19

A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.

Companies like Orbitz and Etsy that use the heck out of Graphite would be surprised to hear that it's "not really production ready"

Yes I was a senior software engineer on the Ops Arch team, and a coworker of Chris Davis when he was at Orbitz, and both my team and the prod sysadmin/netops folks were the first users of it, anywhere in the world, debutting roughly with the Austin project, which was a big rewrite of the Orbitz/Cheaptickets codebase to support i18n & white label functionality. I can assure everybody it was used in production for a huge travel website at least as far back as the 2008 timeframe. And it stood up very well back then. I'd be surprised if it hasn't gotten even better since.
Post reply on HN