Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stream Processing for Go

blog.wallaroolabs.com

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Stream Processing for Go

#31

Post by spooneybarger »

joshlemer wrote: post #30

Does this support advanced windowing and watermarking strategies like are found in Flink?

You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing.

You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-drive...

---

Can you elaborate on what specifically you are referring to as "watermarking strategies" in Flink? We are familiar with Flink's use of the Chandy Lamport algorithm but I'm not sure if that is what you are referring to.

Re: Stream Processing for Go

#32

Post by joshlemer »

spooneybarger wrote: post #31
joshlemer wrote: post #30

Does this support advanced windowing and watermarking strategies like are found in Flink?

You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing. You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-drive... --- Can you elabor…

So in flink you can use a few different bundled in strategies for emitting watermarks, or you can define your own. Here's the documentation on that https://ci.apache.org/projects/flink/flink-docs-release-1.4/...

For example, you can use the bundled MaxOutOfOrdernessTimestampExtractor to emit watermarks that lag behind the maximum timestamp seen so far by some constant amount (say, 50 ms). You could customize this behaviour by implementing the required interface, to emit a watermark, say, whenever we pass an other calendar day. You can also assign strictly-ascending watermarks which you could say is equivalent to maxOutOfOrderness = 0.

You can also choose to emit watermarks based on the content of the stream (see Punctuated Watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/...)

Re: Stream Processing for Go

#33

Post by spooneybarger »

joshlemer wrote: post #32
spooneybarger wrote: post #31

Earlier quoted context omitted.

You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing. You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-drive... --- Can you elabor…

So in flink you can use a few different bundled in strategies for emitting watermarks, or you can define your own. Here's the documentation on that https://ci.apache.org/projects/flink/flink-docs-release-1.4/... For example, you can use the bundled MaxOutOfOrdernessTimestampExtractor to emit watermarks that lag behind the maximum timestamp seen so far by some constant amount (say, 50 ms). You could customize this beh…

Our time based windowing has been implemented yet, so the answer would be: not at this time. It is however on our roadmap.

Re: Stream Processing for Go

#34

Post by felixgallo »

scottlf wrote: post #23
felixgallo wrote: post #17

Earlier quoted context omitted.

the actor model in Erlang and (recently!) in Pony implements something even better: explicit backpressure by deprioritizing senders to busy mailboxes. The model can do this because it includes the scheduler in its scope, which doesn't happen in CSP.

I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect. 1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission betw…

'penalize the sender' has the benefit of being very clean; it doesn't work in >1 machine distributed cases, but then neither do go or pony yet. I'm surprised to discover that it's being removed. I wonder what the thinking is for the local case.

Re: Stream Processing for Go

#35

Post by ramchip »

felixgallo wrote: post #34
scottlf wrote: post #23

Earlier quoted context omitted.

I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect. 1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission betw…

'penalize the sender' has the benefit of being very clean; it doesn't work in >1 machine distributed cases, but then neither do go or pony yet. I'm surprised to discover that it's being removed. I wonder what the thinking is for the local case.

> it doesn't work in >1 machine distributed cases

More generally, it doesn’t work when you have multiple schedulers, which nowadays is true for nearly all Erlang systems.

Post reply on HN