ArkFlow: High-performance Rust stream processing engine
1–10 of 37 posts
Re: ArkFlow: High-performance Rust stream processing engine
#2Looks interesting, how does this compare to arroyo and vector.dev?
Re: ArkFlow: High-performance Rust stream processing engine
#3Previous discussion (46 days ago): https://news.ycombinator.com/item?id=43358682
Re: ArkFlow: High-performance Rust stream processing engine
#4seems like a simplified equivalent of https://vector.dev/
a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)
Re: ArkFlow: High-performance Rust stream processing engine
#5Looks very similar to redpanda-connect/benthos
Re: ArkFlow: High-performance Rust stream processing engine
#6[dead]
Re: ArkFlow: High-performance Rust stream processing engine
#7Looks interesting, how does this compare to arroyo and vector.dev?
Also curious about any comparison to Fluvio.
Re: ArkFlow: High-performance Rust stream processing engine
#8seems like a simplified equivalent of https://vector.dev/ a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)
> seems like a simplified equivalent of https://vector.dev/
No? Vector is for observability, to get your metrics/logs, transform them if needed, and put them in the necessary backends. Transformation is optional, and for cases like downsampling or converting formats or adding metadata.
ArkFlow gets data from stuff like databes and message queues/brokers, transforms it, and puts it back in databases and message queues/brokers. Transformation looks like a pretty central use case.
Very different scenarios. It's like saying that a Renault Kangoo is a simplified equivalent of a BTR-80 because both have wheels, engine and space for stuff.
Re: ArkFlow: High-performance Rust stream processing engine
#9seems like a simplified equivalent of https://vector.dev/ a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)
> seems like a simplified equivalent of https://vector.dev/ No? Vector is for observability, to get your metrics/logs, transform them if needed, and put them in the necessary backends. Transformation is optional, and for cases like downsampling or converting formats or adding metadata. ArkFlow gets data from stuff like databes and message queues/brokers, transforms it, and puts it back in databases and message queues…
Its a rust port of Redpanda Connect (benthos), but with less connectors
Re: ArkFlow: High-performance Rust stream processing engine
#10How do you educate people on stream processing? For pipeline like systems stream processing is essential IMO - backpressure/circuit breakers/etc are critical for resilient systems. Yet I have a hard time building an engineering team that can utilize stream processing; Instead of just falling back on synchronous procedures that are easier to understand (But nearly always slower and more error prone)