51–60 of 169 posts
Earlier quoted context omitted.
Any specifics about which services?
AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.
Earlier quoted context omitted.
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which i…
Earlier quoted context omitted.
You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…
A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...
Earlier quoted context omitted.
That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which i…
No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.
It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…
Especially for those of us who get paid for developing, designing, supporting, architecturing, meetinging, catering, conferencing and maintaining everything around it :-)
It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…
Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?
It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…
Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.
This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Earlier quoted context omitted.
Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.
If I send 1MiB of data by packing it up into messages of 10 byte each, it will ikely be slower than sending 10MiB in a single message. Messages == Overhead. Envelopes, packing, unpacking, parsing, assembling, etc. all eat up cycles.
Earlier quoted context omitted.
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.