Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

github.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#52

Post by jhgg »

reacharavindh wrote: post #39
jhgg wrote: post #33

We use a rather bespoke syslog -> clickhouse log sink ( https://github.com/discordapp/punt/tree/clickhouse ) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!

Out of curiosity, could you tell us a little more about your log analysis workflow? Once they are in Clickhouse, how do you visualise/search/analyse your logs? What is your equivalent of Kibana?

We do rollups into bigquery where we have a bunch of dashboards to look at stuff historically.

I did really like Kibana, ultimately, we had to ditch it (because of ditching ES). Of course, this was a good thing, as I more than once degraded ingest the ES cluster by just using Kibana to do some aggressive filtering. Clickhouse handles these without problem.

I think a more complete world view may be to pipe logs into kafka, and ingest them into Clickhouse/Druid for different types of analysis/rollups.

Our current logging volume exceeds ~10b log lines per day now. Clickhouse handles this ingest almost too well (we have 3 16 core nodes that sit at 5% CPU). This is down from a... 20ish node ES cluster that basically set pegged on CPU... and our log volume then was ~1b/day.

For more ad-hoc, we just use the clickhouse-cli to query the dataset directly. We are tangentially investigating using superset with it.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#53

Post by nh2 »

For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)?

If yes, how comes log processing runs at only so low throughput in general?

That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth are capped at (in your case) 76.7Mbit/s. What's the bottleneck?

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#54

Post by amanzi »

nh2 wrote: post #53

For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)? If yes, how comes log processing runs at only so low throughput in general? That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth ar…

The capitalisation is confusing, but "Mi" means "mebi" - either Mib for mebibits or MiB for mebibytes. The correct term for 1024 * 1024 bits is a Mib, and, 1024 x 1024 x8 bits is a MiB.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#56

Post by reacharavindh »

jhgg wrote: post #52
reacharavindh wrote: post #39

Earlier quoted context omitted.

Out of curiosity, could you tell us a little more about your log analysis workflow? Once they are in Clickhouse, how do you visualise/search/analyse your logs? What is your equivalent of Kibana?

We do rollups into bigquery where we have a bunch of dashboards to look at stuff historically. I did really like Kibana, ultimately, we had to ditch it (because of ditching ES). Of course, this was a good thing, as I more than once degraded ingest the ES cluster by just using Kibana to do some aggressive filtering. Clickhouse handles these without problem. I think a more complete world view may be to pipe logs into k…

Thanks for the response. Lot of tips to go research about for me.

I was mentally debating between trying to find a schema for our logs, and store them in a database where it can be queried efficiently from

Vs

Throwing logs into ELasticSearch in a lazy way and let it index the whole thing to enable us do full-text search on logs. But, with a limitation of only have a few days worth of data in ES indexes.

Kibana’s visualisation is what is holding ES up for me. I will look into superset+Clickhouse to see if I can come up with a good analysis front for our log data.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#59

Post by nh2 »

amanzi wrote: post #54
nh2 wrote: post #53

For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)? If yes, how comes log processing runs at only so low throughput in general? That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth ar…

The capitalisation is confusing, but "Mi" means "mebi" - either Mib for mebibits or MiB for mebibytes. The correct term for 1024 * 1024 bits is a Mib, and, 1024 x 1024 x8 bits is a MiB.

That is correct, but it doesn't answer the question whether it's Bytes or bits (which makes an 8x difference).

Given that the reported values don't care about the "m" (which means "milli" -- clearly doesn't make sense for bytes), I don't think we can rely on the casing of the "b" to tell us the answer.

Re: Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

#60

Post by nh2 »

nh2 wrote: post #53

For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)? If yes, how comes log processing runs at only so low throughput in general? That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth ar…

I've asked it here now: https://github.com/timberio/vector/issues/568
Post reply on HN