71–80 of 93 posts
A lot of people using PySpark are moving to Dask for significantly faster performance. Dask is also built for kubernetes - which is a huge deployment win. Spark is still in-between yarn and kubernetes.
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.
You can use the following options individually or in combination.
Option 1 : Pipeline DB extension (PostgreSQL)
Option 2 : Service broker in commercial SQL databases or building PUSH/PULL queue if not supported. There are many libraries in each programming language which tries to do that. Also see option 4.
Option 3 : Using CDC or Replication for synchronous or asynchronous streamed computation on single or multi node cluster
Option 4 : Transducers. For example, you can compose many sql functions or procedures to act on a single chunk of data instead of always doing async streamed computation after each stage of transformation.
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
distributed in-memory computing for massive datasets to big to fit into vertically scaled memory. generic tabular files, not tables. delta lake.
Earlier quoted context omitted.
One example could be PipelineDB extension for PostgreSQL http://docs.pipelinedb.com/introduction.html
https://www.pipelinedb.com/blog/pipelinedb-is-joining-conflu...
Earlier quoted context omitted.
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
Earlier quoted context omitted.
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
Spark != SQL It's also graph analysis and ML models.
ML models - I already mentioned how to uplift R and Python functions to SQL function. even if you are not using PostgreSQL many other databases help you with uplifting and interfacing with existing ML libraries through FFI
A lot of people using PySpark are moving to Dask for significantly faster performance. Dask is also built for kubernetes - which is a huge deployment win. Spark is still in-between yarn and kubernetes.
Spark is still potentially faster for SQL-like workloads due to the existence of a query optimizer. Dask works at a different level of abstraction and does not have a query optimizer.
That's apples to oranges - because dask does not expose a SQL syntax that needs a query optimiser.
Also pyspark has the additional issue of serialisation between python and jvm. Turns out that just getting rid of that is a huge performance boost.
Earlier quoted context omitted.
> I believe highly optimized C code in PG can be significantly faster than Scala inside Spark. There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should. When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node I…
> like running input through GloVe and pushing ngrams to a temporal store why exactly you think PG will not do this well?