Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 95 posts

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#11

Post by thedood »

whoiscroberts wrote: post #9

Ray user here, what language actors are they using? Ray support Python Java and cpp actors…

We wrote the compactor in Python but, as noted in my previous response to quadrature, most of the performance sensitive code is written in C++ and Rust (but still invoked from Python).

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#13

Post by thedood »

esafak wrote: post #12

Are we talking about big data ETL here? I did not know Ray was suited for it.

This is a specialized ETL use-case - similar to taking a single SQL query and creating a dedicated distributed application tailored to run only that query. The lower-level primitives in Ray Core (tasks and actors) are general purpose enough to make building this type of application possible, but you'll be hard pressed to quickly (i.e., with less than 1 day of effort) make any arbitrary SQL query or dataframe operation run with better efficiency or scale on Ray than on dedicated data processing frameworks like Spark. IMO, the main value add of frameworks like Spark lies more in unlocking "good enough" efficiency and scale for almost any ETL job relatively quickly and easily, even if it may not run your ETL job optimally.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#14

Post by sdfsdfadf »

mannyv wrote: post #3

Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

[flagged]

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#15

Post by wenc »

mannyv wrote: post #3

Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

Most people are moving away from CSV for big datasets, except in exceptional cases involving linear reads (append only ETL). CSV has one big upside which is human readability. But so many downsides: poor random access, no typing, no compression, complex parser needing to handle exceptions.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#17

Post by taeric »

frankjr wrote: post #16

> From the typical Amazon EC2 customer’s perspective, this translates to saving over $120MM/year on Amazon EC2 on-demand R5 instance charges. Does the sales team know about this? /jk

This assumes they aren't reinvesting the money back to other tangible improvements in the services, though?

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#19

Post by 100pctremote »

wenc wrote: post #15
mannyv wrote: post #3

Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!

Most people are moving away from CSV for big datasets, except in exceptional cases involving linear reads (append only ETL). CSV has one big upside which is human readability. But so many downsides: poor random access, no typing, no compression, complex parser needing to handle exceptions.

Most people don't directly query or otherwise operate on raw CSV, though. Large source datasets in CSV format still reign in many enterprises, but these are typically read into a dataframe, manipulated and stored as Parquet and the like, then operated upon by DuckDB, Polars, etc., or modeled (E.g. DBT) and pushed to an OLAP target.
Post reply on HN