81–90 of 93 posts
Earlier quoted context omitted.
If you want a more complicated transform on lots of data your fancy sql won’t help.
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
Earlier quoted context omitted.
distributed in-memory computing for massive datasets to big to fit into vertically scaled memory. generic tabular files, not tables. delta lake.
Yes those features help and all of the distributed SQL databases have data and query cache.
Earlier quoted context omitted.
> like running input through GloVe and pushing ngrams to a temporal store why exactly you think PG will not do this well?
Tell me how you'd do it and I'll tell you why it won't work :)
Then you join first and second table.
Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline.
Earlier quoted context omitted.
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
Sharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
Earlier quoted context omitted.
Tell me how you'd do it and I'll tell you why it won't work :)
gloves are stored in table: token -> vector. Function tokenizes text and store in another table: texd_id, token Then you join first and second table. Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline.
Correct. PG has no place in this workload other than being the final store for the model output. And even then, you'd be using a column store like Redshift or Clickhouse. PG not even suitable for the ngram counters because its ingest rates are way too slow to keep up with a fanned out model spitting out millions of ngrams per second in addition to everything else going on in the pipeline.
You -could- probably do it all in PG. But that'd be a silly esoteric challenge exercise and not something anyone would try on a project. I am sure you recognise that.
Earlier quoted context omitted.
gloves are stored in table: token -> vector. Function tokenizes text and store in another table: texd_id, token Then you join first and second table. Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline.
> Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline. Correct. PG has no place in this workload other than being the final store for the model output. And even then, you'd be using a column store like Redshift or Clickhouse. PG not even suitable for the ngram counters because its ingest rates are way too slow to keep up with a fanned out model spitting out millions of ng…
Earlier quoted context omitted.
> Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline. Correct. PG has no place in this workload other than being the final store for the model output. And even then, you'd be using a column store like Redshift or Clickhouse. PG not even suitable for the ngram counters because its ingest rates are way too slow to keep up with a fanned out model spitting out millions of ng…
I would say "fanned out model spitting out millions of ngrams per second" is much more unusual exercise comparing to using PG for ETL workload.
But you have a year worth of historical data that you want to work with. If you're able to process 1m ngrams per second, it'll take a couple of days to get through that. You probably want to get closer to 10m/s if you're tweaking your model and want to iterate reasonably quickly. Of course there's ways to optimise all that and batch it and whatnot, but basically any big data tasks with the need to work on historical data and iterate on their models, quickly end up with kafka clusters piping millions of messages per second to keep those iteration times productive.
Ultimately this post is about Spark, and the comment that started this was someone listing PG 'replacements' for traditional ML pipeline components. If you need Spark, you're at scales where PG has no place.
Earlier quoted context omitted.
I would say "fanned out model spitting out millions of ngrams per second" is much more unusual exercise comparing to using PG for ETL workload.
A typical twitter post will have about 50 2/3/4-grams. Let's ignore skipgrams. The twitter decahose will throw about 600 of these at you per second. That's 30k barebones ngrams per second to keep with the decahose. But you have a year worth of historical data that you want to work with. If you're able to process 1m ngrams per second, it'll take a couple of days to get through that. You probably want to get closer to…
Also in typical ML pipeline as I mentioned you can generate ngrams in input function of your model (Dataset API in TF), you don't need to store it somewhere.
Earlier quoted context omitted.
Spark is still potentially faster for SQL-like workloads due to the existence of a query optimizer. Dask works at a different level of abstraction and does not have a query optimizer.
Are you talking about the spark SQL catalyst optimiser ? That's apples to oranges - because dask does not expose a SQL syntax that needs a query optimiser. Also pyspark has the additional issue of serialisation between python and jvm. Turns out that just getting rid of that is a huge performance boost.
Most operations on dataframe-like objects can be described in SQL operations. Spark supports these operations and Catalyst can optimize query plans for these.
You are correct in that Dask does not optimize for this because Dask operations are more primitive hence it does not have the correct level of abstraction to do query optimization, only task graph optimization. Which reinforces my point that if you have a SQL-like workload on Dask.dataframes, chances are Dask may not outperform Spark.
I'm kind of surprised it took this long for someone to do this. It was clear very early on that the JVM was a bad match for what Spark was trying to do.
this sounds far too simplistic, so .. reference?