Vector 0.11 Release: K8s, ARC, and metrics collection
1–10 of 15 posts
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#2Vector still insists that Kafka is for logs only. Vector is a fantastic project, we have so many use cases for it, but the thing that trips us up is not being able to send metrics via Kafka, without transforming them to logs.
Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#3The world of observability seems to be converging on some key ideas. I think Vector's data model where they convert logs into structured events is a key idea that wasn't always obvious. I think the remaining big idea is what do do about high cardinality data. Most solutions pre-aggregate the data which does't tolerate high cardinality tags. Solutions like Honeycomb and Datadog logs are stream based and tolerate high cardinality tags, but with limitations on what can be done with it. It will be interesting if the streaming based solutions become the final standard. Vector warns about high cardinality labels, just like others do. I'm not sure if that is a limitation of Vector or just the Sink.
I think the interesting part is the overlap between observability metrics for operations and expanding in to BI metrics with the same tools.
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#4Vector still insists that Kafka is for logs only. Vector is a fantastic project, we have so many use cases for it, but the thing that trips us up is not being able to send metrics via Kafka, without transforming them to logs. Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.
Thanks for pointing this out! That limitation is largely a holdover from when the Kafka sink was written and our support for accepting multiple data types was not as good as it is now. As things stand today, it should be a pretty simple change to enable this.
I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value this kind of feedback and try to address it as promptly as possible.
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#5I have been running vector in several k8s clusters since July. I'm doing some non-trivial transforms over several different types of sources and sinks.
The config makes this easy, but my favorite part is the fact that the CPU and MEM of the vector processes barely even registers on my metrics charts. I can't even tell you what their actual and requested resources are because I haven't bothered to look in a while.
It's one thing I never have to worry about. I could use more of those.
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#6The world of observability seems to be converging on some key ideas. I think Vector's data model where they convert logs into structured events is a key idea that wasn't always obvious. I think the remaining big idea is what do do about high cardinality data. Most solutions pre-aggregate the data which does't tolerate high cardinality tags. Solutions like Honeycomb and Datadog logs are stream based and tolerate high…
> I think the interesting part is the overlap between observability metrics for operations and expanding in to BI metrics with the same tools.
I built Logflare so you could get your structured logs into BigQuery directly, so now you this :)
Hopefully we'll have a Vector sink soon, but until then I think they support POSTing batches to HTTP endpoints so you can do that and we'll take any JSON and go straight to BQ with it after migrating your schema for you (automatically based on the incoming payload shape).
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#7Yet another project with no clear explanation of what it does. Some sort of conversion proxy I guess?
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#8Yet another project with no clear explanation of what it does. Some sort of conversion proxy I guess?
You want to go to the project page, not just the release notes for this version.
The diagram and succinct summary give you a very high level view, but the Docs link at the top takes you this:
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#9Does Vector replace some uses of Kafka?
Re: Vector 0.11 Release: K8s, ARC, and metrics collection
#10Yet another project with no clear explanation of what it does. Some sort of conversion proxy I guess?
You want to go to the project page, not just the release notes for this version. eg. https://vector.dev/ The diagram and succinct summary give you a very high level view, but the Docs link at the top takes you this: https://vector.dev/docs/about/what-is-vector/
I did that, and it's still just some board room speak about cost reduction, novel data enrichment and not what actual it actually does. What do I use this for?
I get that you want to sell this to executives, but the people that point executives to products to buy are the developers like me, and if don't understand what it does, I can't see executives finding your product.
Just give examples of what on-the-ground problems is solves, converts kafka messages to s3 objects or what?
Lack of novel data enrichment is not a problem that I have, but I could do with something that streams k8s pod logs to kafka or s3 or whatever. Does this solve that?