Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accidentally exponential behavior in Spark

heap.io

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Accidentally exponential behavior in Spark

#31

Post by mjburgess »

thundergolfer wrote: post #23
d_burfoot wrote: post #20

There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…

Exactly correct. I’ve got a post in the works called “Elegy for Hadoop” that traces the history back to the early 2000s and arrives at the present day where you can easily get on-demand instances with 500Gb of RAM and use it for only your application’s lifetime. If you want 1000Gb instead of 500gb it does not cost 5x it costs 2x, significantly invalidating the “need to use excess commodity hardware” premise of the di…

Is there anything you can say about Spark for Data Engineering (/ETL) ?

The most common reason for spark use today is ETL+DataLakes (ie., cloud object stores and ETL in/out).

It seems actual analysis is happening in fast databases that receive data from the object stores.

can anyone here comment on this paradigm?

Re: Accidentally exponential behavior in Spark

#33

Post by KptMarchewa »

ngc248 wrote: post #30
d_burfoot wrote: post #20

There are two lessons you could learn from this episode: 1. Use shallow trees and the clever workaround presented in the article. 2. Don't use Spark for tasks that require complex logic. People should trace out the line of reasoning that leads them to use tools like Spark. It is convoluted and contingent - it goes back to work done at Google in the early 2000s, when the key to getting good price / performance was usi…

Spark is still the best for stream processing use cases and if you have enough volume of data coming in something like spark is still the best for batch processing. '

>Spark is still the best for stream processing use cases

No, Flink is much better.

Re: Accidentally exponential behavior in Spark

#34

Post by KptMarchewa »

throwaway81523 wrote: post #25
IvanVergiliev wrote: post #14

Post author here. Let me know if you have any questions!

Would be nice if title said Apache Spark instead of just Spark, since there are other programs like Spark/Ada also called Spark.

There is also Java web application framework called Spark. Nowadays everyone just call it Sparkjava.

Re: Accidentally exponential behavior in Spark

#35

Post by KptMarchewa »

thundergolfer wrote: post #24
paulbaumgart wrote: post #21

Earlier quoted context omitted.

Is there an alternative you’d recommend?

Check out Frank McSherry’s COST (Configuration that Outperforms a Single Thread) and see if you are just better off with a single fat machine[1]. 1. https://www.usenix.org/system/files/conference/hotos15/hotos...

Premise of the article is very true, but the comparison itself is very biased and dishonest.

Graph problems are famously hard to scale horizontally, and represent very small percent of what people use those big data systems for. Especially if you can fit the data in RAM...

Anyway, if you're able to run your workload on a single machine, then definitely do it.

Re: Accidentally exponential behavior in Spark

#36

Post by thundergolfer »

KptMarchewa wrote: post #35
thundergolfer wrote: post #24

Earlier quoted context omitted.

Check out Frank McSherry’s COST (Configuration that Outperforms a Single Thread) and see if you are just better off with a single fat machine[1]. 1. https://www.usenix.org/system/files/conference/hotos15/hotos...

Premise of the article is very true, but the comparison itself is very biased and dishonest. Graph problems are famously hard to scale horizontally, and represent very small percent of what people use those big data systems for. Especially if you can fit the data in RAM... Anyway, if you're able to run your workload on a single machine, then definitely do it.

I basically agree with you. Linked COST because of the premise and the upshot of the paper, which is totally valid.

Re: Accidentally exponential behavior in Spark

#37

Post by xentripetal »

mjburgess wrote: post #31
thundergolfer wrote: post #23

Earlier quoted context omitted.

Exactly correct. I’ve got a post in the works called “Elegy for Hadoop” that traces the history back to the early 2000s and arrives at the present day where you can easily get on-demand instances with 500Gb of RAM and use it for only your application’s lifetime. If you want 1000Gb instead of 500gb it does not cost 5x it costs 2x, significantly invalidating the “need to use excess commodity hardware” premise of the di…

Is there anything you can say about Spark for Data Engineering (/ETL) ? The most common reason for spark use today is ETL+DataLakes (ie., cloud object stores and ETL in/out). It seems actual analysis is happening in fast databases that receive data from the object stores. can anyone here comment on this paradigm?

I don't have much insight into spark but I've been using Dataflow/beam for ETL. Been a pretty good experience. follows the style of spinning up compute to process as needed then shutdown.

Re: Accidentally exponential behavior in Spark

#38

Post by IvanVergiliev »

mjburgess wrote: post #32
IvanVergiliev wrote: post #14

Post author here. Let me know if you have any questions!

Is there anything you can say here about why you're running this query in spark? Supposing spark is your ETL machinery... would it not make more sense to ETL this into a database?

Definitely. One of the primary benefits we get out of Spark is the ability to decouple storage and compute, and to very easily scale out the compute.

Our main Spark workload is pretty spiky. We have low load during most of the day, and very high load at certain times - either system-wide, or because a large customer triggered an expensive operation. Using Spark as our distributed query engine allows us to quickly spin up new worker nodes and process the high load in a timely manner. We can then downsize the cluster again to keep our compute spend in check.

And just to provide some context on our data size, here's an article about how we use Citus at Heap - https://www.citusdata.com/customers/heap . We store close to a petabyte of data in our distributed Citus cluster. However, we've found Spark to be significantly better at queries with large result sets - our Connect product syncs a lot of data from our internal storage to customers' warehouses.

Post reply on HN