Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming: a skill gap?

charemza.name

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Streaming: a skill gap?

#31

Post by hilbertseries »

Jach wrote: post #17
cultofmetatron wrote: post #2

I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…

> In terms of business value, a cron job running in a high memory vps will more than satisfy and take much less time to develop. Yeah, but I hate it... I've worked with a team where we had a cron job do some batch processing every night, but for some large customers it started taking ~12-15 hours to complete, and certain important user operations are locked while it's running. The solution? Running once per week star…

How long until the cron start takes 70 hours to finish?

Re: Streaming: a skill gap?

#32

Post by gen220 »

We have built a few streaming pipelines at my current place of work, with the standard tech (kafka-confluent, a self-hosted Schema registry, etc) operated for us by SREs.

For our use case (messages never expire, consuming logic is complex and evolves every couple of months necessitating a full replay, we’re the only consumer, few 100k messages per year), we’ve decided that it just isn’t worth the trade offs vs storing temporal data in Postgres.

The place where streaming seems to be most useful for us is for piping data from one team to any other team. That way, no one needs to quibble over database permissions, we can just hand over the topic name and say “have fun”.

For internal workloads, the pain points are the ones where streaming is known to have faults: quick and easy queries on the data in the queue, ergonomics around schema evolution and topic offsets, exactly once delivery with n>1 partitions and shards, Zookeeper crapping out and dropping messages for god knows what reason.

These are all problems that Postgres solves for free.

I think streaming makes sense between teams, or if your workload is significantly different from ours, but that’s just my two cents.

Kafka and friends are for sure useful tools, but they are not the be all end all they’re sometimes made out to be by people who don’t have to deal with them day in and out.

Re: Streaming: a skill gap?

#33

Post by anonymoushn »

nwhitehead wrote: post #24

My favorite is Welford's Algorithm [1] which lets you compute mean and standard deviation in one pass. Every programmer should be aware of it. [1]: https://en.wikipedia.org/wiki/Algorithms_for_calculating_var...

If readers would like to read an implementation, one is here: https://github.com/sharpobject/runningstats

It includes skewness and kurtosis.

Re: Streaming: a skill gap?

#34

Post by mc13 »

nwhitehead wrote: post #24

My favorite is Welford's Algorithm [1] which lets you compute mean and standard deviation in one pass. Every programmer should be aware of it. [1]: https://en.wikipedia.org/wiki/Algorithms_for_calculating_var...

Exactly, i deal with a few of data processing programs, wish my more experienced colleagues told me about these algorithms before, but doesn't this algorithm only compute an approximation and not the exact value ?

Re: Streaming: a skill gap?

#35

Post by Const-me »

I'm not sure it's skill, more likely languages + runtime support.

When I program C#, I use streaming quite a lot, often combined with async-await for I/O. The framework also helps, e.g. all compression/encryption algorithms support asynchronous versions.

When I program C++, I tend to avoid streaming at all cost. The ergonomic is just not there. Technically doable, but will turn the code into callback hell, hard to debug and expensive to support.

Re: Streaming: a skill gap?

#36

Post by anonymoushn »

mc13 wrote: post #34
nwhitehead wrote: post #24

My favorite is Welford's Algorithm [1] which lets you compute mean and standard deviation in one pass. Every programmer should be aware of it. [1]: https://en.wikipedia.org/wiki/Algorithms_for_calculating_var...

Exactly, i deal with a few of data processing programs, wish my more experienced colleagues told me about these algorithms before, but doesn't this algorithm only compute an approximation and not the exact value ?

To about the extent that a 64-bit float can ever be said to contain an exact value, this algorithm will compute the exact value. There's no sampling.

Re: Streaming: a skill gap?

#37

Post by mpweiher »

cultofmetatron wrote: post #2

I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…

Hah! One of my favourite topics: The Gentle Tyranny of Call/Return.

I am still writing it up, but it looks like we are currently stuck in the call/return architectural style or even paradigm. Meaning all our languages offer what are in essence variations of call/return, be they subroutines, procedures, function or methods.

However, a lot of the problems we need to solve or systems we want to build do not conform to this pattern. Probably the majority by now. When we have such a system, we have choice to make, with two bad options on offer: either conform to the system/problem, therefore having something that constantly grates against the language/environment, or conform with call/return and grate against the problem you're trying to solve.

Streaming is an example of this. I presented Standard Object Out: Streaming Objects with Polymorphic Write Streams at DLS '19, which shows some of the nasty effects and the start of a solution.

  https://conf.researchr.org/details/dls-2019/dls-2019/7/Standard-Object-Out-Streaming-Objects-with-Polymorphic-Write-Streams
I also addressed this problem more generally at last summer's ESUG '19, talking about Objective-Smalltalk:

   https://www.youtube.com/watch?v=vrD3TrVuiV0&list=PLJ5nSnWzQXi8DPNpy1jCkjE4yE0WUtDP2&index=53
Objective-Smalltalk makes it possible to express systems (I hesitate to even call them programs) in non-call/return styles (such as dataflow/streaming) as naturally as call/return systems and without giving up interoperability with the (large) call/return world.

Re: Streaming: a skill gap?

#39

Post by cultofmetatron »

Jach wrote: post #17
cultofmetatron wrote: post #2

I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…

> In terms of business value, a cron job running in a high memory vps will more than satisfy and take much less time to develop. Yeah, but I hate it... I've worked with a team where we had a cron job do some batch processing every night, but for some large customers it started taking ~12-15 hours to complete, and certain important user operations are locked while it's running. The solution? Running once per week star…

I would say this is the 5% of the time where it suddenly becomes worth it to streamify. you'll already have code showing what transformations need to be done. The point is to avoid premature optimization.
Post reply on HN