21–30 of 66 posts
Well logstash is now supporting good persistent queue. https://www.elastic.co/guide/en/logstash/current/persistent-... I don't know why author didn't put correctness tick mark on it.
https://github.com/timberio/vector-test-harness/tree/master/...
Definitely open to feedback on what we're doing wrong.
Flume, but in Rust? https://flume.apache.org/
There already are a lot of projects in this space. While better performance is always great, most are already plenty fast for the majority of use cases. The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing. PS: the Logstash numbers seem suspiciously low. I'd bet it's some…
Yep, I push about 50-100MB/s through a single instance of Logstash (Redis (list) -> S3). That configuration is not in the benchmark table, but surely it's more demanding than TCP -> Blackhole, TCP -> TCP, etc. Regardless, Vector looks very nice and I'll be testing it out :)
Reported an issue in their test harness.
Well logstash is now supporting good persistent queue. https://www.elastic.co/guide/en/logstash/current/persistent-... I don't know why author didn't put correctness tick mark on it.
It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others: https://github.com/timberio/vector-test-harness/tree/master/... Definitely open to feedback on what we're doing wrong.
What is a high level example of why I'd use this?
Hi! I work on Vector. For a motivating example, let's say you have an application fronted by nginx. Using Vector would allow you to ingest your nginx logs off disk, parse them, expose status code and response time distributions to prometheus, and store the parsed logs as JSON on S3. There are obviously plenty of ways to accomplish that same thing today, but we believe Vector is somewhat unique in allowing you to do i…
So is Vector like Splunk forwarder or more than that?
There already are a lot of projects in this space. While better performance is always great, most are already plenty fast for the majority of use cases. The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing. PS: the Logstash numbers seem suspiciously low. I'd bet it's some…
As for the multitudes of inputs/outputs, covering the 95% most-used sources and sinks is a great starting point. I think Vector got that list right in this case.
Earlier quoted context omitted.
Yep, I push about 50-100MB/s through a single instance of Logstash (Redis (list) -> S3). That configuration is not in the benchmark table, but surely it's more demanding than TCP -> Blackhole, TCP -> TCP, etc. Regardless, Vector looks very nice and I'll be testing it out :)
They are using the default config with 1GB memory. Sadly that's absolutely nothing for Logstash. Reported an issue in their test harness.
Earlier quoted context omitted.
They are using the default config with 1GB memory. Sadly that's absolutely nothing for Logstash. Reported an issue in their test harness.
IMO that's totally fair -- 1GB is more than adequate to run a log forwarder. The fact that Logstash can't perform well under such conditions is worthy of consideration when selecting a solution.
But this is far from the possible performance that Logstash offers, so the comparison is at least misleading.
Also, there is no information about the memory usage of the other contestants, which probably don't have memory limits if they don't run on the JVM.
I say all this as someone who stopped using Logstash, in part due to the high requirements - but benchmarks should strive for a fair comparison.
https://github.com/influxdata/telegraf
Biggest thing that pops out to me is LUA engine (seems amazing :) )
I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana. This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations. Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prome…