FastSpark: A New Fast Native Implementation of Spark from Scratch • Hacker News Board

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#51

Post by Barraketh » Thu, Oct 31, 2019, 11:14 PM UTC

madhadron wrote: post #48

Barraketh wrote: post #46

Earlier quoted context omitted.

I know that Spark has had a lot of work put into it, but my personal experience with it has been pretty negative. I've spent a lot of time at my job trying to tune it to our workflows (extremely deep queries), with only moderate success. I've just POC'd a custom SQL execution engine that was 200x faster than spark for the same workflows. Now, our requirements are pretty non-standard, but I find it pretty easy to beli…

McSherry et al's paper "Scalability! But at what COST?" is worth reading. A single threaded, single core implementation typically outperforms Spark. The best rule of thumb I'm aware of is: unless you can't fit your computation on a single machine or your jobs are likely to fail before completing from the size and length involved, you are generally better off without Spark or similar systems. And if sampling can get y…

I expected that there is overhead to distributing the computation, but I was surprised by the magnitude of the speedups available.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#52

Post by raja_sekar » Thu, Oct 31, 2019, 11:21 PM UTC

fourthark wrote: post #49

raja_sekar wrote: post #42

Earlier quoted context omitted.

The author of the repo here. It is definitely not orders of magnitude faster. I didn't mention it anywhere also I guess. But yeah, JVM is sometimes a problem for in-memory computing for big data processing. Spark itself tried to address this. This is what their tungsten engine does. They circumvent huge Java Objects by using native types through JNI(sun.misc.Unsafe). This is the reason why Dataframes are generally mu…

> if others see the benefits, it will automatically grow with the help of the community There’s nothing automatic about it, you or someone else will need to put a lot of work into leading the community, merging pull requests, debugging, etc. (Sad to say, promotion too, in a lot of cases.)

I didn't mean it in that way. Instead of sitting idly on my laptop, it might at least be useful for someone, and if it really proves to be beneficial, then people might contribute to it. Yeah, not denying your point, it does require huge effort from some people to get it into a mature production-ready stage.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#53

Post by krcz » Thu, Oct 31, 2019, 11:27 PM UTC

I'm wondering how much could be gained if one used all possible optimizations: e.g. by analyzing the data flow graph - expressed using DSL - and generating native node programs, using CPU thread pinning and user space network stack (like ScyllaDB does [1]).

[1] https://www.scylladb.com/product/technology/

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#54

Post by raja_sekar » Thu, Oct 31, 2019, 11:37 PM UTC

Barraketh wrote: post #46

Joeri wrote: post #8

This sounds too good to be true. If it is this easy to be orders of magnitude faster than spark on JVM, why haven't the spark developers ported spark to native code already?

I know that Spark has had a lot of work put into it, but my personal experience with it has been pretty negative. I've spent a lot of time at my job trying to tune it to our workflows (extremely deep queries), with only moderate success. I've just POC'd a custom SQL execution engine that was 200x faster than spark for the same workflows. Now, our requirements are pretty non-standard, but I find it pretty easy to beli…

It is indeed my opinion too. In non-standard workflows, handcrafted code/application will most likely beat generic frameworks(not true for some cases). I have conflicting thoughts about this. Nowadays industries are very fast-moving, they generally can't afford to do it all for each of their use cases. So they tend to pick up generic frameworks. But I have seen many managers picking the wrong tools for the job and vastly overestimate their future needs. Everyone thinks that they are going to process petabytes of data, and they make the decision to use these generic distributed frameworks from the beginning to avoid the future scale. It rarely happens. Most of the time, they end up spending money on Cloud because making something distributed comes with a lot of redundancy to provide fault tolerance and yet not as performant as single machine performance due for data up to few TBs. Even here, if you take that parquet example, my hand-coded Rust code beats the Rust RDD version by 4x. I guess we can't change this attitude. So it is better to aim for improving these libraries.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#55

Post by missosoup » Thu, Oct 31, 2019, 11:37 PM UTC

maximente wrote: post #31

missosoup wrote: post #12

Earlier quoted context omitted.

What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

looks like dask is python-only, so it's a nonstarter (loser) for already existing JVM code that runs on spark

Spark stacks inevitably end up with PySpark though. It's rework for people who already committed to Spark, sure. And for bigger projects that committed to Spark this change isn't justifiable. But for a greenfield project, choosing Spark is just silly today.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#56

Post by orhmeh09 » Thu, Oct 31, 2019, 11:38 PM UTC

MaxBarraclough wrote: post #47

For the confused: this is about Apache Spark, not the Ada-based SPARK language. [0] Perhaps I'm alone here but I'd prefer the title say Apache Spark explicitly. [0] https://en.wikipedia.org/wiki/SPARK_(programming_language)

Spark (and Apache Spark) is a trademark of the Apache Foundation. If the title were SPARK in all caps, I'd understand, but how often do you read articles about SPARK where the name is written as "Spark?"

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#57

Post by raja_sekar » Thu, Oct 31, 2019, 11:42 PM UTC

madhadron wrote: post #48

Barraketh wrote: post #46

Earlier quoted context omitted.

I know that Spark has had a lot of work put into it, but my personal experience with it has been pretty negative. I've spent a lot of time at my job trying to tune it to our workflows (extremely deep queries), with only moderate success. I've just POC'd a custom SQL execution engine that was 200x faster than spark for the same workflows. Now, our requirements are pretty non-standard, but I find it pretty easy to beli…

McSherry et al's paper "Scalability! But at what COST?" is worth reading. A single threaded, single core implementation typically outperforms Spark. The best rule of thumb I'm aware of is: unless you can't fit your computation on a single machine or your jobs are likely to fail before completing from the size and length involved, you are generally better off without Spark or similar systems. And if sampling can get y…

In my experience too I observed that distributed code introduces a lot of redundancy and it requires a lot of data to beat the performance of a single-threaded/single machine implementation. Check out McSherrys' Timely Dataflow, it is truly an amazing piece of work.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#58

Post by missosoup » Thu, Oct 31, 2019, 11:45 PM UTC

sandGorgon wrote: post #28

missosoup wrote: post #12

Earlier quoted context omitted.

What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?

tl;dr we use it for a similar set of tasks that one would use Airflow for.

Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc.

I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#59

Post by wmf » Thu, Oct 31, 2019, 11:54 PM UTC

krcz wrote: post #53

I'm wondering how much could be gained if one used all possible optimizations: e.g. by analyzing the data flow graph - expressed using DSL - and generating native node programs, using CPU thread pinning and user space network stack (like ScyllaDB does [1]). [1] https://www.scylladb.com/product/technology/

A lot. https://www.weld.rs/

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#60

Post by Nursie » Thu, Oct 31, 2019, 11:55 PM UTC

MaxBarraclough wrote: post #47

For the confused: this is about Apache Spark, not the Ada-based SPARK language. [0] Perhaps I'm alone here but I'd prefer the title say Apache Spark explicitly. [0] https://en.wikipedia.org/wiki/SPARK_(programming_language)

Nor Spark (Sparkjava), the http framework!