FastSpark: A New Fast Native Implementation of Spark from Scratch

#1

Post by mkj » Thu, Oct 31, 2019, 3:35 AM UTC

FastSpark: A New Fast Native Implementation of Spark from Scratch
medium.com

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#2

Post by mindcrime » Thu, Oct 31, 2019, 4:44 AM UTC

That's mondo righteous. I think I may have finally found a reason to learn Rust.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#3

Post by gok » Thu, Oct 31, 2019, 4:51 PM UTC

I'm kind of surprised it took this long for someone to do this. It was clear very early on that the JVM was a bad match for what Spark was trying to do.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#4

Post by splix » Thu, Oct 31, 2019, 5:11 PM UTC

> You’ve reached the end of your free member preview for this month

:( I guess I can read Medium posts only during first couple of days in a month

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#5

Post by missosoup » Thu, Oct 31, 2019, 5:12 PM UTC

"You’ve reached the end of your free member preview for this month"

Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#6

Post by vojta_letal » Thu, Oct 31, 2019, 5:17 PM UTC

splix wrote: post #4

> You’ve reached the end of your free member preview for this month :( I guess I can read Medium posts only during first couple of days in a month

Anonymous windows have found a second use-case.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#7

Post by ieatpies » Thu, Oct 31, 2019, 5:21 PM UTC

missosoup wrote: post #5

"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Try incognito mode

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#8

Post by Joeri » Thu, Oct 31, 2019, 5:34 PM UTC

This sounds too good to be true. If it is this easy to be orders of magnitude faster than spark on JVM, why haven't the spark developers ported spark to native code already?

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#9

Post by truth_seeker » Thu, Oct 31, 2019, 5:37 PM UTC

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#10

Post by madenine » Thu, Oct 31, 2019, 5:49 PM UTC

truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

depends what you’re doing. For querying large datasets? 100% with you.

For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out