Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FastSpark: A New Fast Native Implementation of Spark from Scratch

medium.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#61

Post by missosoup »

riku_iki wrote: post #32
missosoup wrote: post #24

Earlier quoted context omitted.

Yeah that's not going to work for what people call analytics workloads today. PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.

> hell not suitable to fan out ML workloads depends on the scale? Not everyone processes petabytes of data. > PG might play the role of the slow You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.

> I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.

There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should.

When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node IO eclipses single node perf, which is why Spark is used despite being a terribly inefficient thing (although in the case of Spark, it's so inefficient that for interim sized workloads you'd actually be better off vertically scaling a single node and using something else). PG won't help at all with these tasks.

Also, that smorgasbord of extensions GP listed isn't offered by any cloud vendor as a managed service afaik, meaning you must roll and manage your own. Depending on your needs, that might be a show stopper.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#62

Post by missosoup »

raja_sekar wrote: post #36
missosoup wrote: post #5

"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Sorry about that. I will just use Github to host my content hereafter. For now, you can use this friend link. https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...

Thanks for that. Now that I've had a chance to read through it, a question:

The examples seem to be implemented in pure Rust. No one is going to port their Spark jobs to Rust in the shot term. Have you evaluated perf with Python etc?

If you're still seeing significant speedups, you might want to bottle this up and seek VC because a managed service along the lines of 'databricks but 10x faster' would certainly get traction.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#63

Post by raja_sekar »

missosoup wrote: post #62
raja_sekar wrote: post #36

Earlier quoted context omitted.

Sorry about that. I will just use Github to host my content hereafter. For now, you can use this friend link. https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...

Thanks for that. Now that I've had a chance to read through it, a question: The examples seem to be implemented in pure Rust. No one is going to port their Spark jobs to Rust in the shot term. Have you evaluated perf with Python etc? If you're still seeing significant speedups, you might want to bottle this up and seek VC because a managed service along the lines of 'databricks but 10x faster' would certainly get tra…

It is in a very initial POC stage and distributed mode is pretty basic, but it is moving faster than I expected. Python integration is definitely one of the primary objectives as I suspect that no one is going to learn Rust for this, although I feel that it is not that hard. In fact, it can have a better integration story with python than Spark as Rust has good C interop. Regarding performance, yeah it is pretty good from what I have seen for CPU intensive tasks and once blockmanager is implemented with compression and other optimizations like Spark, shuffle tasks also will improve. There are a lot of unnecessary allocations here than I would prefer just to keep it in safe Rust as much as possible and there is still plenty of optimizations possible here. I am doing this in my free time only. I feel that it is too early to compare witn Spark given how many features Spark has. Maybe in a couple of months after it matures a bit and if there is enough traction for this, then we can look for sponsors.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#64

Post by Barraketh »

raja_sekar wrote: post #54
Barraketh wrote: post #46

Earlier quoted context omitted.

I know that Spark has had a lot of work put into it, but my personal experience with it has been pretty negative. I've spent a lot of time at my job trying to tune it to our workflows (extremely deep queries), with only moderate success. I've just POC'd a custom SQL execution engine that was 200x faster than spark for the same workflows. Now, our requirements are pretty non-standard, but I find it pretty easy to beli…

It is indeed my opinion too. In non-standard workflows, handcrafted code/application will most likely beat generic frameworks(not true for some cases). I have conflicting thoughts about this. Nowadays industries are very fast-moving, they generally can't afford to do it all for each of their use cases. So they tend to pick up generic frameworks. But I have seen many managers picking the wrong tools for the job and va…

I completely agree that we need better generic libraries. I was mostly commenting that I really believe that there are huge wins that can be achieved in the "generic distributed execution engine" space, and that people shouldn't be intimidated by the work that has already gone into spark.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#65

Post by riku_iki »

missosoup wrote: post #61
riku_iki wrote: post #32

Earlier quoted context omitted.

> hell not suitable to fan out ML workloads depends on the scale? Not everyone processes petabytes of data. > PG might play the role of the slow You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.

> I believe highly optimized C code in PG can be significantly faster than Scala inside Spark. There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should. When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node I…

> like running input through GloVe and pushing ngrams to a temporal store

why exactly you think PG will not do this well?

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#66

Post by vamega »

missosoup wrote: post #18
ai_ja_nai wrote: post #17

Earlier quoted context omitted.

Very interesting. Can't find references to "Perfect", though; could you please point to a link?

https://www.prefect.io Not the most SEO-friendly choice of name. Great product though.

Are you using their cloud product? The core/open source product doesn't have a way to persist schedule data.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#67

Post by koolba »

jdminhbg wrote: post #19
ieatpies wrote: post #7

Earlier quoted context omitted.

Try incognito mode

You can also use Reader Mode on Safari, which not only avoids the modals and popups but gets rid of the top and bottom bars as well. Long-click on the Reader Mode button and you can set it to always use it on medium.com.

> Long-click on the Reader Mode button and you can set it to always use it on medium.com.

Learning this just made my day!

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#68

Post by sandGorgon »

missosoup wrote: post #58
sandGorgon wrote: post #28

Earlier quoted context omitted.

Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?

tl;dr we use it for a similar set of tasks that one would use Airflow for. Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc. I'll say one thing though. The Perfect website…

This is super interesting!

Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler.

How has your experience been ?

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#70

Post by wenc »

aabbcc1241 wrote: post #69

I like to see people re-implement things and share their (better) results. Even if the results isn't better than the 'battle-tested' existing solutions, at least we can learn something in the process.

Scylla [1] for instance is a C++ rewrite and a drop-in replacement of JVM-based Cassandra, and from what I've read is fairly stable and performs much faster.

[1] https://www.scylladb.com/

Post reply on HN