Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FastSpark: A New Fast Native Implementation of Spark from Scratch

medium.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#11

Post by missosoup »

ieatpies wrote: post #7
missosoup wrote: post #5

"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Try incognito mode

Why?

This is an economy where content competes for clicks, not clicks competing for content. The author of that content wants me to see it, Medium doesn't want me to see it. I don't care enough to try to circumvent their arrangement.

Given the number of votes on my root comment, it seems neither do most people.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#12

Post by missosoup »

madenine wrote: post #10
truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out

What does spark win at exactly?

Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#13

Post by tandav »

ieatpies wrote: post #7
missosoup wrote: post #5

"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Try incognito mode

or (sign out & remove medium cookies)

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#14

Post by nxpnsv »

truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

If you want a more complicated transform on lots of data your fancy sql won’t help.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#15

Post by mvitorino »

truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

How would you efficiently keep continuously updated complex metrics (aggregations, windowed functions, etc) calculated on top of unbounded/streaming data using a database? I'm not saying that Spark is the ideal solution, but there are a set of problems that require tools such as Spark.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#16

Post by aloknnikhil »

truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#17

Post by ai_ja_nai »

missosoup wrote: post #12
madenine wrote: post #10

Earlier quoted context omitted.

depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out

What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

Very interesting. Can't find references to "Perfect", though; could you please point to a link?

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#18

Post by missosoup »

ai_ja_nai wrote: post #17
missosoup wrote: post #12

Earlier quoted context omitted.

What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

Very interesting. Can't find references to "Perfect", though; could you please point to a link?

https://www.prefect.io

Not the most SEO-friendly choice of name. Great product though.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#19

Post by jdminhbg »

ieatpies wrote: post #7
missosoup wrote: post #5

"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.

Try incognito mode

You can also use Reader Mode on Safari, which not only avoids the modals and popups but gets rid of the top and bottom bars as well. Long-click on the Reader Mode button and you can set it to always use it on medium.com.

Re: FastSpark: A New Fast Native Implementation of Spark from Scratch

#20

Post by truth_seeker »

madenine wrote: post #10
truth_seeker wrote: post #9

Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)

depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out

Say for an example, I am using PostgreSQL 12 + CitusDB extension

Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all)

Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding

Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension OR Using Logical replication for stream computation

ML -> PG support variety of statistics functions. It also supports PL/R and PL/Python extension to interface with ML libraries.

Also, there are various kinds of Foreign Data Wrappers supported by PG - https://wiki.postgresql.org/wiki/Foreign_data_wrappers

Post reply on HN