Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 95 posts

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#51

Post by theLiminator »

robertnishihara wrote: post #27

I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…

Curious if you know how well Ray works with multithreaded python libraries? For example, when using jax with ray, I have to ensure the import ordering imports ray first, as forking a threaded process leads to deadlocks in Python. Do you know how to ensure that ray handles forking the python interpreter correctly?

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#52

Post by layoric »

thedood wrote: post #29
igmor wrote: post #24

Can you share any data on how big of a cluster is running Ray jobs?

From the blog post, the largest individual Ray cluster that was observed running a production compaction job in Q1 had 26,846 vCPUs and ~210TiB of RAM. This is roughly equivalent to a Ray cluster composed of 839 r5.8xlarge EC2 nodes (w/ 32 vCPUs and 256GiB RAM per node).

For those interested, this would be at a cost of:

- ~$1691/hour on demand

- ~$1065/hour reserved

- ~$521/hour spot

Not including any related data transfer costs.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#53

Post by robertnishihara »

theLiminator wrote: post #51
robertnishihara wrote: post #27

I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…

Curious if you know how well Ray works with multithreaded python libraries? For example, when using jax with ray, I have to ensure the import ordering imports ray first, as forking a threaded process leads to deadlocks in Python. Do you know how to ensure that ray handles forking the python interpreter correctly?

Multi-threaded libraries (e.g., numpy and PyTorch on CPUs come to mind) are well supported. In scenarios where many processes are each running heavily multi-threaded computations, it can help to pin specific processes to specific cores (e.g., using tools like psutil) to avoid contention.

The scenario where a Ray task forks is probably not very well supported. You can certainly start a subprocess from within a Ray task, but I think forking could easily cause issues.

You can definitely use Ray + Jax, but you probably need to avoid forking a process within a Ray worker.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#54

Post by robertnishihara »

zacmps wrote: post #48
robertnishihara wrote: post #27

I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…

Super cool to see you here. I've also looked at ray for running data pipelines before (at much much smaller scales) for the reasons you suggest (unstructured data, mixed CPU/GPU compute). One thing I've wanted is an incremental computation framework (i.e., salsa [1]) built on ray so that I can write jobs that transparently reuse intermediate results from an object store if their dependents haven't changed. Do you kno…

Other folks have built data processing libraries on top of Ray: Modin and Daft come to mind.

But I'm not aware of anything exactly like what you're referring to!

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#55

Post by fragmede »

wenc wrote: post #25
100pctremote wrote: post #19

Earlier quoted context omitted.

Most people don't directly query or otherwise operate on raw CSV, though. Large source datasets in CSV format still reign in many enterprises, but these are typically read into a dataframe, manipulated and stored as Parquet and the like, then operated upon by DuckDB, Polars, etc., or modeled (E.g. DBT) and pushed to an OLAP target.

There are folks who still directly query CSV formats in a data lake using a query engine like Athena or Spark or Redshift Spectrum — which ends up being much slower and consuming more resources than is necessary due to full table scans. CSV is only good for append only. But so is Parquet and if you can write Parquet from the get go, you save on storage as well has have a directly queryable column store from the start…

how about using Sqlite database files as an interchange format?

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#56

Post by nubinetwork »

robertnishihara wrote: post #27

I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…

> this is not a typical ray use case

Must be good enough if you're willing to dogfood it though?

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#57

Post by p0rkbelly »

whoevercares wrote: post #39
mannyv wrote: post #3

Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

Is it really AWS? I don’t recall any service called BDT

The second paragraph discusses that BDT is an internal team at Amazon Retail. They used AWS and Ray to do this.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#58

Post by qwerp »

mannyv wrote: post #3

Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

our plans are measured in centuries

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#59

Post by wenc »

fragmede wrote: post #55
wenc wrote: post #25

Earlier quoted context omitted.

There are folks who still directly query CSV formats in a data lake using a query engine like Athena or Spark or Redshift Spectrum — which ends up being much slower and consuming more resources than is necessary due to full table scans. CSV is only good for append only. But so is Parquet and if you can write Parquet from the get go, you save on storage as well has have a directly queryable column store from the start…

how about using Sqlite database files as an interchange format?

I haven't thought about sqlite as a data interchange format, but I was looking at deploying sqlite as a data lake format some time ago, and found it wanting.

1. Dynamically typed (with type affinity) [1]. This causes problems with there are multiple data generating processes. The new sqlite has a STRICT table type that enforces types but only for the few basic types that it has.

2. Doesn't have a date/time type [1]. This is problematic because you can store dates as TEXT, REAL or INTEGER (it's up to the developer) and if you have sqlite files from > 1 source, date fields could be any of those types, and you have to convert between them.

3. Isn't columnar, so complex analytics at scale is not performant.

I guess one can use sqlite as a data interchange format, but it's not ideal.

One area sqlite does excel in is as a application file format [2] and that's where it is mostly used [3].

[1] https://www.sqlite.org/datatype3.html

[2] https://www.sqlite.org/appfileformat.html

[3] https://en.wikipedia.org/wiki/SQLite#Notable_uses

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#60

Post by spott »

whoevercares wrote: post #41

I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet . Of course it didn’t exist 4 years ago, but would it cheaper to build?

It sounds like a lot of the value here was in a more appropriate level of abstraction, not in the speed of the compute itself.

Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.

Post reply on HN