Congrats to the Timber Team on this one. Ben and Zach were a couple of former colleagues on mine.
Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
41–50 of 66 posts
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#42Seems similar to Veneur (like many other projects mentioned in comments here; didn't realize this space was so crowded!) - down to the first two letters of the name: https://github.com/stripe/veneur Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.
Absolutely, Veneur is something we looked at quite a bit when it popped up. It's clear Stripe was feeling a lot of the same pain points we were when we started building Vector and they've come up with something really impressive.
As you mentioned, it seems they've focused more on metrics out of the gate, while we've spent more of our time on the logging side of things (for now). We're working to catch up on metrics functionality, but interoperability via SSF is an interesting idea!
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#43Just a heads up: There are several figures in your docs where the entirety of the useful information on the page is in the image and they don't have alt tags or any accessible way to get the information (that I can find anyway). e.g. https://docs.vector.dev/use-cases/security-and-compliance
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#44Heka, but in Rust instead of Golang?
There is also a Heka but in C. https://github.com/mozilla-services/hindsight Unfortunately deployment of Hindsight isn't as nice as Heka since you need to compile it yourself with all the Lua extensions you need, and the documentation is very disorganized. Vector looks great on those counts, will be excited to try it if they get features like reliable Vector-Vector transport and more flexible file delimiters.
Hindsight isn't exactly heka in C; it's useful for data processing but not as a general-purpose log shipper. Currently we're using fluentd for the latter.
(Disclosure: I work for Mozilla on the team that runs services used by firefox users and developers)
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#45Earlier quoted context omitted.
It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others: https://github.com/timberio/vector-test-harness/tree/master/... Definitely open to feedback on what we're doing wrong.
This is totally off topic, but holy crap, first message in 11 years. I would have lost my password about a dozen times by now.
That answer was definitely worth the wait.
(there's a "Forgot Password" link somewhere here).
[do you always look at poster's activity? sounds time consuming.]
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#46I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana. This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations. Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prome…
Thank you! Very glad it looks useful to you. It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!
Just a question, are you familiar with work that's been done on the OpenCensus Collector and Agent [0]?
There was discussion earlier this year about creating a design doc for OpenCensus to handle logs. I'm not sure if that got finished, or it was sidelined while the OpenCensus & OpenTracing merger was worked on. Both projects will combine under the OpenTelemetry name.
I've been quite happy with the OpenCensus instrumentation SDKs.
I think the Logs & Metrics space is interesting, especially because there is so much overlap, both are just ways of representing data about an event that occurred in the software.
OpenCensus is fairly widely backed: Google, Microsoft, Etsy, Scalyr...[1]
[0] https://github.com/census-instrumentation/opencensus-service...
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#47Earlier quoted context omitted.
Vector can act as a Splunk forwarder, but is designed to be much more flexible. In addition to forwarding to more storage systems (S3, Elasticsearch, syslog, etc), Vector can do things like sampling logs, parsing them, and aggregating them into metrics. Depending on your needs, this makes it easier to reduce your Splunk volume and reduce costs, transition to something like an ELK stack, etc. We're also working to bui…
Thanks. This is all very interesting. I should try it on our app servers.
Thanks for your interest! And please feel free to get in touch if you have any questions or feel there are things we could do to better support your use case: https://vector.dev/community/
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#48Earlier quoted context omitted.
Thank you! Very glad it looks useful to you. It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!
Just a question, are you familiar with work that's been done on the OpenCensus Collector and Agent [0]? There was discussion earlier this year about creating a design doc for OpenCensus to handle logs. I'm not sure if that got finished, or it was sidelined while the OpenCensus & OpenTracing merger was worked on. Both projects will combine under the OpenTelemetry name. I've been quite happy with the OpenCensus instrum…
I've looked into OpenCensus/OpenTracing/OpenTelemetry (and the apparently unaffiliated OpenMetrics?) a bit, but I'm not as familiar as I'd like to be. It does seem like they're focused primarily on application-level instrumentation and the ability to ship that metrics and tracing data to different backends.
Vector's perspective is that your applications and infrastructure are already emitting all kinds of interesting data via logs, metrics, etc, and the primary challenge is to collect, enrich, and manage the storage of that data. We have no plans to integrate Vector into your application or introduce some kind of Vector-specific method of exporting data.
We'll definitely be watching OpenTelemetry as it moves forward and would very much like to be a compatible part of that ecosystem. To the degree that they use common open standards for their communication protocols, that could just fall out naturally.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#49Congratulations to the Timber team! I've had the pleasure of doing a small amount of contract work for them, have nothing but praise for them all.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#50I wonder how popular posts on hackernews would be if people didn't mention "written in rust" in the title.
Jumping on the hype bandwagon. Don't question it or you get downvoted... oops.