Love Timber.io, so it's great to see some of your innovations coming back out as open source projects. Thanks!
Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
31–40 of 66 posts
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#32How does this compare to telegraf? https://github.com/influxdata/telegraf Biggest thing that pops out to me is LUA engine (seems amazing :) )
Telegraf is nicely done. We spent a lot of time testing solutions in our test harness (https://github.com/timberio/vector-test-harness) and Telegraf was the most impressive of the tools we tested, so kudos to the Influx team on that.
But to answer your question, telegraf is very heavily metrics focused, and their logging support appears to be limited (reducing logs to metrics only). Vector is _currently_ focused on logging with an eye towards metrics, but still has work to do on the metrics front.
For example, we opened the door with the `log_to_metric` transform (https://docs.vector.dev/usage/configuration/transforms/log_t...) to ensure our data model supports metrics, but we still have a lot of work to do when it comes to metrics as a whole. Our end goal is to eventually replace telegraf and be a single, open, vendor neutral solution for both logs and metrics.
Happy to clarify further :)
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#33We use a rather bespoke syslog -> clickhouse log sink (https://github.com/discordapp/punt/tree/clickhouse) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#34Earlier quoted context omitted.
Hi! I work on Vector. For a motivating example, let's say you have an application fronted by nginx. Using Vector would allow you to ingest your nginx logs off disk, parse them, expose status code and response time distributions to prometheus, and store the parsed logs as JSON on S3. There are obviously plenty of ways to accomplish that same thing today, but we believe Vector is somewhat unique in allowing you to do i…
Our company uses Splunk. I am not on admin/ops side so possibly missing details. The way I understand is that there is Splunk forwarder running on our app servers. And then there is Splunk server URL from there I get consolidated logs in browser where I can search and run many other statistical function. So is Vector like Splunk forwarder or more than that?
[deleted]
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#35Earlier quoted context omitted.
Hi! I work on Vector. For a motivating example, let's say you have an application fronted by nginx. Using Vector would allow you to ingest your nginx logs off disk, parse them, expose status code and response time distributions to prometheus, and store the parsed logs as JSON on S3. There are obviously plenty of ways to accomplish that same thing today, but we believe Vector is somewhat unique in allowing you to do i…
Our company uses Splunk. I am not on admin/ops side so possibly missing details. The way I understand is that there is Splunk forwarder running on our app servers. And then there is Splunk server URL from there I get consolidated logs in browser where I can search and run many other statistical function. So is Vector like Splunk forwarder or more than that?
Vector can act as a Splunk forwarder, but is designed to be much more flexible.
In addition to forwarding to more storage systems (S3, Elasticsearch, syslog, etc), Vector can do things like sampling logs, parsing them, and aggregating them into metrics. Depending on your needs, this makes it easier to reduce your Splunk volume and reduce costs, transition to something like an ELK stack, etc.
We're also working to build up the metrics side of Vector's capabilities. In a way, you can think of Vector as a stream processing system for observability data, capable of feeding into a variety of storage backends.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#36Seems similar to Veneur (like many other projects mentioned in comments here; didn't realize this space was so crowded!) - down to the first two letters of the name: https://github.com/stripe/veneur
Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#37Earlier quoted context omitted.
Yep, I push about 50-100MB/s through a single instance of Logstash (Redis (list) -> S3). That configuration is not in the benchmark table, but surely it's more demanding than TCP -> Blackhole, TCP -> TCP, etc. Regardless, Vector looks very nice and I'll be testing it out :)
They are using the default config with 1GB memory. Sadly that's absolutely nothing for Logstash. Reported an issue in their test harness.
I just checked mine and it's running with -Xms1g -Xmx1g and humming away with no problem. I wonder if it's something else?
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#38Earlier quoted context omitted.
Our company uses Splunk. I am not on admin/ops side so possibly missing details. The way I understand is that there is Splunk forwarder running on our app servers. And then there is Splunk server URL from there I get consolidated logs in browser where I can search and run many other statistical function. So is Vector like Splunk forwarder or more than that?
Vector can act as a Splunk forwarder, but is designed to be much more flexible. In addition to forwarding to more storage systems (S3, Elasticsearch, syslog, etc), Vector can do things like sampling logs, parsing them, and aggregating them into metrics. Depending on your needs, this makes it easier to reduce your Splunk volume and reduce costs, transition to something like an ELK stack, etc. We're also working to bui…
Thanks. This is all very interesting. I should try it on our app servers.
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#39We use a rather bespoke syslog -> clickhouse log sink ( https://github.com/discordapp/punt/tree/clickhouse ) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!
Out of curiosity, could you tell us a little more about your log analysis workflow? Once they are in Clickhouse, how do you visualise/search/analyse your logs? What is your equivalent of Kibana?
Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
#40We use a rather bespoke syslog -> clickhouse log sink ( https://github.com/discordapp/punt/tree/clickhouse ) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!
Absolutely, this is likely the next integration we'll be working on. There were a few features schema-wise that we needed support before we started, but we're _very_ close. We'd love beta testers to help us build it out. Feel free to email us if you're interested: vector@timber.io