Live data from Hacker News

Vector 0.11 Release: K8s, ARC, and metrics collection

vector.dev

11–15 of 15 posts

Re: Vector 0.11 Release: K8s, ARC, and metrics collection

#11
post #8

Earlier quoted context omitted.

You want to go to the project page, not just the release notes for this version. eg. https://vector.dev/ The diagram and succinct summary give you a very high level view, but the Docs link at the top takes you this: https://vector.dev/docs/about/what-is-vector/

I did that, and it's still just some board room speak about cost reduction, novel data enrichment and not what actual it actually does. What do I use this for? I get that you want to sell this to executives, but the people that point executives to products to buy are the developers like me, and if don't understand what it does, I can't see executives finding your product. Just give examples of what on-the-ground prob…

Yes, streaming k8s pod logs to Kafka and/or S3 is a great example of when you could use Vector.

The "collect, transform, and route all your logs, metrics, and traces" bit is our most succinct explanation of what Vector does, but I'll admit it's still not as clear as we'd like. To expand it slightly, Vector is a tool to collect observability data (logs, metrics, etc) from wherever it's generated, optionally process that data in-flight, and then forward it to whatever upstream system you'd like to consume it. It does this by providing a variety of different components that you configure into whatever pipeline you need. In your example, you could use our new k8s source and plug it into our Kafka sink, our S3 sink, or both.

Re: Vector 0.11 Release: K8s, ARC, and metrics collection

#12
post #9

Does Vector replace some uses of Kafka?

Vector is designed to work well with systems like Kafka, not to replace them. While it does have a _very_ simple durable queue in the optional disk buffer feature, it is nowhere near the durability, fault tolerance, performance, etc of a full Kafka cluster and we would not recommend thinking of them as the same type of system.

That being said, we do know of a few cases where Vector's in-flight processing and routing capabilities were enough that a full Kafka-based pipeline was no longer needed. This ability to push computation out to the edge and reduce the need for centralized infrastructure is one of the aspects of Vector that we're most excited about.

Re: Vector 0.11 Release: K8s, ARC, and metrics collection

#14
post #4
post #2

Vector still insists that Kafka is for logs only. Vector is a fantastic project, we have so many use cases for it, but the thing that trips us up is not being able to send metrics via Kafka, without transforming them to logs. Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.

Thanks for pointing this out! That limitation is largely a holdover from when the Kafka sink was written and our support for accepting multiple data types was not as good as it is now. As things stand today, it should be a pretty simple change to enable this. I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value…

Very nice tool. I could find a really great use for it, if it supported sftp sinks. Is that in the plan?

Re: Vector 0.11 Release: K8s, ARC, and metrics collection

#15
post #4
post #2

Vector still insists that Kafka is for logs only. Vector is a fantastic project, we have so many use cases for it, but the thing that trips us up is not being able to send metrics via Kafka, without transforming them to logs. Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.

Thanks for pointing this out! That limitation is largely a holdover from when the Kafka sink was written and our support for accepting multiple data types was not as good as it is now. As things stand today, it should be a pretty simple change to enable this. I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value…

Thank you, that’s great. We really do love having Vector available to us, for such a young project it’s amazing that is such a stabil and solid piece of software.
Post reply on HN