Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

github.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#61

Post by codepodu »

binarylogic wrote: post #40
jhgg wrote: post #33

We use a rather bespoke syslog -> clickhouse log sink ( https://github.com/discordapp/punt/tree/clickhouse ) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!

Absolutely, this is likely the next integration we'll be working on. There were a few features schema-wise that we needed support before we started, but we're _very_ close. We'd love beta testers to help us build it out. Feel free to email us if you're interested: vector@timber.io

Heyyy you're the one who wrote authlogic! What a blast from the past :) Thank you for your contribution to the Rails community, and congrats on shipping Timber & Vector!

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#62

Post by nickserv »

Could this replace a simple fluentd setup right now or are there still major functionalities missing?

Specifically, I'm ingesting nginx logs in JSON format, cleaning up invalid UTF8 bytes (usually sent in the forwarded-for header to exploit security vulnerabilities), and sending to elasticsearch on an automated 90 day retention policy (daily indexes).

Seems like a fairly common use case for webservers.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#63

Post by binarylogic »

envolt wrote: post #55

Off Topic - Does either of Filebeat or Logstash support config hot reload, as mentioned in the Vector's doc? https://docs.vector.dev/usage/administration/reloading Edit - Found It - https://www.elastic.co/guide/en/logstash/current/reloading-c...

They do, we actually put together a correctness test for this behavior:

https://github.com/timberio/vector-test-harness/tree/master/...

Logstash is not graceful. Our testing shows that they basically shut it down and start it again.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#64

Post by binarylogic »

nh2 wrote: post #53

For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)? If yes, how comes log processing runs at only so low throughput in general? That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth ar…

Ah, I'm amazed that I did not know this. I just updated everything to use MiB. Thanks for pointing that out.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#65

Post by gnufx »

When people say "high performance" about these things, I wonder how they compare, for instance, with the Sandia tools.* One things that matters there is avoiding system noise (jitter) on the monitored systems with transport over RDMA.

* http://ovis.ca.sandia.gov/

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#66

Post by warpspin »

A pity it does not support at least once or exactly once delivery for the Vector sink.

Also, the documentation seems to miss information about which sinks support TLS?

We're currently looking for a distributed-over-the-internet logging setup and are interested in evaluating Rsyslog/RELP alternatives.

Post reply on HN