This is a great blog entry on exactly the kind of system I am trying to build.
When we went thru the evaluation for this stack - Elasticsearch came out as the choice for the datastore and querying part. Where we are still not decided is using Flume vs logstash. Have you compared the two? We will be building our own UI ...
Central Logging with Open Source Software
41–50 of 51 posts
Re: Central Logging with Open Source Software
#42Earlier quoted context omitted.
Why do you say that? We have a couple of datacenters, so yes, we have more than a couple of servers.
In a situation where one has that much money to blow on something so limited, virtually anything would've sufficed. We did a trivial test of Splunk at my last company, it's extremely expensive and it's very easy to bump into its limitations. We were able to wreck the poor Splunk server with some rather sundry queries into a dataset that shouldn't have been that big of a deal. Issues that we took back to the company a…
What is extremely expensive for you? We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis.
Without knowing details of exactly what you are doing it's difficult to comment on your problems with queries. It's true that something like Solr gives you more control over the indexing process, so you can optimize it more for specific queries. Splunk tends to rely more on saved searches (and the new search acceleration feature).
Re: Central Logging with Open Source Software
#43Earlier quoted context omitted.
In a situation where one has that much money to blow on something so limited, virtually anything would've sufficed. We did a trivial test of Splunk at my last company, it's extremely expensive and it's very easy to bump into its limitations. We were able to wreck the poor Splunk server with some rather sundry queries into a dataset that shouldn't have been that big of a deal. Issues that we took back to the company a…
What is extremely expensive for you? We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis. Without knowing details of exactly what you are doing it's difficult to comment on your problems with queries. It's true that something like Solr gives you more control over the indexing process, so you can optimize it more for specific queries. Splunk tends to rel…
>We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis.
What are you storing the data with...the etchings on wings of fairies?
>Some blather about Splunk's "saved searches"
We talked to the company, explored every avenue. Our volume of data simply overwhelmed it. (Data from three Apache servers. Lol.)
I am 100% certain you know less than Splunk-The-Company, so our conversation is done here.
Re: Central Logging with Open Source Software
#44Earlier quoted context omitted.
The vast majority of open source "big-data" infrastructure is in Java (Hadoop, HBase, Cassandra, Solr, Elastic Search etc). It works pretty well. I'm not sure what your question is, but I've experimented with loading netflow data in Solr and I'm averaging sub-2 second query times. That's on a laptop, with a couple of minutes of netflow (around 10Gb). With proper indexing your search response time shouldn't increase l…
loading 10gb on a traditional hdd takes more than 2s (that's 5gb/s read speed. Nice hard drive.). your data is either in ram and you've a lot of ram, either, it's just not 2s, or its not a 10gb index. And i'm talking 100gb+ indexes ;-) Obviously 2min of netflow data ain't much. I would want to see the result over 200h (or more) of netflow data, for example
No, querying the data takes less than 2 seconds. I can't remember the load time.
Obviously 2min of netflow data ain't much
Depends where you work...
I just checked, and it was 2Gb of netflow I tested on. That seemed small, so I looked a bit deeper and indeed I was only using a small fraction of our total netflow for that period. Tt was adequate for what I was trying, though.
Re: Central Logging with Open Source Software
#45Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…
Splunk is absurdly priced for normal verbose syslogs for a bunch of hosts. You could preprocess or tune your logging to only send important stuff to Splunk to make up for this.
It's cheap for application-specific logs where each line is relatively high value.
Re: Central Logging with Open Source Software
#46Earlier quoted context omitted.
What is extremely expensive for you? We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis. Without knowing details of exactly what you are doing it's difficult to comment on your problems with queries. It's true that something like Solr gives you more control over the indexing process, so you can optimize it more for specific queries. Splunk tends to rel…
>We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis. What are you storing the data with...the etchings on wings of fairies? >Some blather about Splunk's "saved searches" We talked to the company, explored every avenue. Our volume of data simply overwhelmed it. (Data from three Apache servers. Lol.) I am 100% certain you know less than Splunk-The-Compan…
What are you storing the data with..
It's on a SAN. We'll probably migrate to local disks at some point. The pricing is typical SAN pricing[1].
* Our volume of data simply overwhelmed it. (Data from three Apache servers. Lol.)*
Yeah, well we do a lot more data than that.
[1] Take a look at the NetApp, Dell & EMC prices on http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h..., or look at http://serverfault.com/questions/76725/whats-the-nominal-cos... and you'll be in the right price range.
Re: Central Logging with Open Source Software
#47A little bit of warning, to be fair: he mentions you should quit everything in favor of Graphite. As awesome as Graphite it is, it's not really production ready, and setting it up is an exercise in putting together semi-working software without any documentation.
We collect @2.5 million data points every minute with our Graphite system. While I agree that initial installation is not as easy as "yum install graphite ; /etc/init.d/graphite start", I wouldn't hesitate to call it production ready.
Re: Central Logging with Open Source Software
#48Re: Central Logging with Open Source Software
#49Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source s…
I think it defeats the purpose. Splunk is great but you need to pay for a license.
What's missing is a free as in beer and as in freedom solution that is decent. Mostly because it means we can all commit fixes/updates/etc to it. Including people who can't pay for a product (but are willing to pay for support) such as communities.
Re: Central Logging with Open Source Software
#50Just a note - using TCP logging is dangerous. If the syslog server hangs, clients may block writing to the socket and your whole infrastructure will lock up. See: http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downti...
Hmm, shouldn't this be alleviated with RELP [1]? (which the author suggest you use).
Because, otherwise, since AFAIK rsyslog doesn't support DTLS it means unencrypted log transmission. (For RELP it also means running stunnel anyways, which supports DTLS, and may be a solution)