Stream Processing for Go

I'm the VP of Engineering at Wallaroo Labs. Happy to answer questions. I worked with Andy on this preview release of our Go API. Andy will be on in a while and will also be answering any questions that might come up.

We are getting ready to start work on performance tuning soon. We have a lot of work already planned there.

If you play around with the Go API, we'd love to hear from you. We need your feedback to improve our Go offering.

You can reach us via our user mailing list:

https://groups.io/g/wallaroo

Or our IRC channel:

https://webchat.freenode.net/?channels=#wallaroo

We'd also love to do a short call with folks to discuss their Go stream processing use cases. If you email hello@wallaroolabs.com, we can set that up.

Thanks in advance!

HN was really helpful in helping us develop the new version of our Python API. We hope to get the same results with the Go API.

- How does the wallaroo roadmap compare to other ambitious frameworks such as Flink and Beam ?

I'm not familiar with the roadmaps for Flink and Beam so I can't comment on that. I can however, speak to our roadmap.

For Go in particular, I'm going to be starting on performance improvements very soon. Beyond that, we are waiting for feedback from folks to determine what the best course of action for the Go API is. When we released our Python API, we got a lot of excellent feedback that lead to us switching from a class based API to one that emphasized functions and decorators. We expect that similar evolution will happen with the Go API.

For Wallaroo in general, we're working on improvements to our documentation, our installation experience, Python API improvements, state object migrations (see below), as well as an expanded API to allow you to express more types of dataflows with Wallaroo. There's more beyond that, but you get the idea.

Wallaroo is definitely a younger product than Flink and Beam. We open sourced it in September of 2017 so we have a lot of work to do.

We'd love to hear from folks who are interested in being able to do stream processing in Go or Python without needing to become experts in running the JVM in production. Talking with folks helps us prioritize our work and deliver a product that can be valuable to some folks now and still more in the future as we add features.

- Tools to reason about processing/event time:

Wallaroo has a metrics UI that allows you to see throughput and within Wallaroo latency on a per-pipeline basis as well as per computation. It can be very helpful in spotting bottlenecks.

I'm interested in hearing what other tools you think would be useful. We have a few ideas but are always looking for more things we can build to make the product better.

- Is it possible to save state and restart processes with updated code?

Yes, so long as your data objects don't change. If your data objects change (for example, add a field, remove a field) then not currently. However, we started initial discussion on our approach to that this week and starting work on it in the near future.

The current solution if your data objects change is to get the state of objects out of the system before shutdown and then stream in those new values when you startup with the code that involves a schema change to your data objects. It's far from perfect but works for now until, the schema migration feature is added.

https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-wr...

TCP and Kafka are the currently supported Source and Sink types. We are in the process of improving the Kafka support.

Additionally, we are in the design phase for providing APIS in Go and Python to make it easier for Wallaroo users to provide their own sources and sinks.