Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

41–50 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#41

Post by fritkot »

Foobar8568 wrote: post #28
bpodgursky wrote: post #11

Earlier quoted context omitted.

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...

Snowflake and DV have no interdependency whatsoever. Snowflake is just a database. Whether you use DV to model the data inside of it or dimensional modelling or "big wide tables" is completely up to you, there's nothing about it that requires or benefits DV in particular.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#42

Post by StreamBright »

rxin wrote: post #12
bpodgursky wrote: post #11

Earlier quoted context omitted.

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)

The biggest selling point of Snowflake for most of the customers is that they do not need to maintain the infrastructure.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#43

Post by fritkot »

rxin wrote: post #12
bpodgursky wrote: post #11

Earlier quoted context omitted.

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)

Of course you would say that it's more performant and flexible ...TCP-DS was just a PR ploy

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#45

Post by ricardobeat »

grenran wrote: post #5

Walking to the store is faster than catching a plane

I assume this is mocking the article as obvious, but it starts from someone using Hadoop to process 1.75GB of data.

The default advice is that you should catch a plane to go anywhere, and the author is showing why you should walk.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#46

Post by ed_elliott_asc »

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares?

Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong.

It comes with a well documented security mode and a support contract when needed.

Developers can be onboarded quickly and work code reviewed and managed.

Logging and diagnostics are available and you can report on metrics easily.

That isn’t true with custom data pipelines written in shell scripts.

The value isn’t in the pure execution time, it is in everything around it.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#47

Post by jbverschoor »

ed_elliott_asc wrote: post #46

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…

Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale.

It's the same narrative people use for many complexity or slow things

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#48

Post by spullara »

Puts wrote: post #23
wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.

I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#49

Post by belter »

rxin wrote: post #12
bpodgursky wrote: post #11

Earlier quoted context omitted.

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)

Probably an oversight on your part, but I would argue would be elegant to disclose you are one of the co-founders.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#50

Post by wodenokoto »

ed_elliott_asc wrote: post #46

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…

Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands?

Or is that more of a kubernetes thing?

Post reply on HN