31–40 of 93 posts
Earlier quoted context omitted.
depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out
What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.
Earlier quoted context omitted.
Say for an example, I am using PostgreSQL 12 + CitusDB extension Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all) Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension…
Yeah that's not going to work for what people call analytics workloads today. PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.
depends on the scale? Not everyone processes petabytes of data.
> PG might play the role of the slow
You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.
I'm kind of surprised it took this long for someone to do this. It was clear very early on that the JVM was a bad match for what Spark was trying to do.
A lot of people using PySpark are moving to Dask for significantly faster performance. Dask is also built for kubernetes - which is a huge deployment win. Spark is still in-between yarn and kubernetes.
"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.
"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.
"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.
The author has chosen to monetize this article. They have the choice to make it free to everyone.
The examples are all in Rust, so it’s very hard to make a non-toy demo. Usually if one uses the RDD API, one has some sort of library code that’s already in Java or Python and it’s impractical to port that code to make the job run. Or more likely, somebody will write an initial version using that code and then port / optimize the job later. Native dependencies usually mean you’ll need docker. Spark pre-dates docker,…
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.
Earlier quoted context omitted.
If you want a more complicated transform on lots of data your fancy sql won’t help.
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
It's also graph analysis and ML models.