21–30 of 93 posts
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
How would you efficiently keep continuously updated complex metrics (aggregations, windowed functions, etc) calculated on top of unbounded/streaming data using a database? I'm not saying that Spark is the ideal solution, but there are a set of problems that require tools such as Spark.
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
If you want a more complicated transform on lots of data your fancy sql won’t help.
Earlier quoted context omitted.
How would you efficiently keep continuously updated complex metrics (aggregations, windowed functions, etc) calculated on top of unbounded/streaming data using a database? I'm not saying that Spark is the ideal solution, but there are a set of problems that require tools such as Spark.
One example could be PipelineDB extension for PostgreSQL http://docs.pipelinedb.com/introduction.html
Earlier quoted context omitted.
depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out
Say for an example, I am using PostgreSQL 12 + CitusDB extension Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all) Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension…
PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
A Spark inspired framework written in modern C++.
Earlier quoted context omitted.
depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out
What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.
Spark is still in-between yarn and kubernetes.
Native dependencies usually mean you’ll need docker. Spark pre-dates docker, and just relatively recently added the Kubernetes runner, which makes dockerized jobs easy. But historically it hasn’t been easy to run a job in a containerized environment with the native dependencies you need. You can ship native deps with your job, but that’s not easy, especially if you need a rebuild with each job.
The main advantage of Spark is flexibility and interoperability. You save time by not having to write something optimized on day 1 (for something you might throw away). And you get SQL support, something Beam / Hadoop don’t have (certainly not for Python). There are lots of benchmarks where Spark SQL is not a winner, but the point is Spark will help you save development time.