Freaking A... Rust performance is nuts when done right. Very cool.
Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
11–20 of 66 posts
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#12There already are a lot of projects in this space.
While better performance is always great, most are already plenty fast for the majority of use cases.
The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing.
PS: the Logstash numbers seem suspiciously low. I'd bet it's some JVM config issue. Logstash can come to a crawl if it does not have enough memory.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#13I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana.
This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations.
Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prometheus with same principles.
Low memory footprint,
Rust,
Single binary,
Customizable with a single config file without spending hours in manuals,
Stdin, Stderr -> transform -> Prometheus.
I’m learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one.
Cheers on this open source project. I will contribute whatever I can. Thanks!!
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#14Heka, but in Rust instead of Golang?
There is also a Heka but in C.
https://github.com/mozilla-services/hindsight
Unfortunately deployment of Hindsight isn't as nice as Heka since you need to compile it yourself with all the Lua extensions you need, and the documentation is very disorganized.
Vector looks great on those counts, will be excited to try it if they get features like reliable Vector-Vector transport and more flexible file delimiters.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#15I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana. This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations. Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prome…
What you're asking for kinda exist, it's called mtail, check it out !
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#16I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana. This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations. Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prome…
Thank you! Very glad it looks useful to you.
It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#17Flume, but in Rust?
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#18There already are a lot of projects in this space. While better performance is always great, most are already plenty fast for the majority of use cases. The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing. PS: the Logstash numbers seem suspiciously low. I'd bet it's some…
Yep, I push about 50-100MB/s through a single instance of Logstash (Redis (list) -> S3). That configuration is not in the benchmark table, but surely it's more demanding than TCP -> Blackhole, TCP -> TCP, etc.
Regardless, Vector looks very nice and I'll be testing it out :)
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#19I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana. This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations. Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prome…
What you're asking for kinda exist, it's called mtail, check it out !
We really love the idea of mtail and think it's criminally underused.
A big part of the reason we started building Vector was to integrate that kind of functionality into a larger project, so people wouldn't have to get over the hump of discovering, setting up, and rolling out a whole separate tool.
We're definitely not as mature as mtail yet, but we're working really hard to get there.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#20Well logstash is now supporting good persistent queue.
https://www.elastic.co/guide/en/logstash/current/persistent-...
I don't know why author didn't put correctness tick mark on it.