Video from the author deep diving this. https://www.youtube.com/watch?v=h7svj_oAY14
Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
21–30 of 95 posts
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#22> From the typical Amazon EC2 customer’s perspective, this translates to saving over $120MM/year on Amazon EC2 on-demand R5 instance charges. Does the sales team know about this? /jk
Amazon sales and business teams are constantly focused on reducing costs for customers and they celebrate this internally too.
I have seen dozens of big ticket “we saved this customer $xxxK/year” posts on slack and other internal venues, the customer obsession is real.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#23I was in a workshop that taught me Ray. It was interesting to know that the people who started Spark were also involved in making Ray.
This is not badmouthing either project just an observation and if you architected one task you would be good at attacking the same problem better .
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#24Can you share any data on how big of a cluster is running Ray jobs?
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#25Earlier quoted context omitted.
Most people are moving away from CSV for big datasets, except in exceptional cases involving linear reads (append only ETL). CSV has one big upside which is human readability. But so many downsides: poor random access, no typing, no compression, complex parser needing to handle exceptions.
Most people don't directly query or otherwise operate on raw CSV, though. Large source datasets in CSV format still reign in many enterprises, but these are typically read into a dataframe, manipulated and stored as Parquet and the like, then operated upon by DuckDB, Polars, etc., or modeled (E.g. DBT) and pushed to an OLAP target.
There are folks who still directly query CSV formats in a data lake using a query engine like Athena or Spark or Redshift Spectrum — which ends up being much slower and consuming more resources than is necessary due to full table scans.
CSV is only good for append only.
But so is Parquet and if you can write Parquet from the get go, you save on storage as well has have a directly queryable column store from the start.
CSV still exists because of legacy data generating processes and dearth of Parquet familiarity among many software engineers. CSV is simple to generate and easy to troubleshoot without specialized tools (compared to Parquet which requires tools like Visidata). But you pay for it elsewhere.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#26I was in a workshop that taught me Ray. It was interesting to know that the people who started Spark were also involved in making Ray. This is not badmouthing either project just an observation and if you architected one task you would be good at attacking the same problem better .
Well spark was really a showcase Project for mesos when it was created. Now everyone knows a lot more
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#27I'm one of the creators of Ray. A few thoughts :)
1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14
2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training, inference, and unstructured data processing.
3. We have a data processing library called Ray Data for ingesting and processing data, often done in conjunction with training and inference. However, I believe in this particular use case, the heavy lifting is largely done with Ray's core APIs (tasks & actors), which are lower level and more flexible, which makes sense for highly custom use cases. Most Ray users use the Ray libraries (train, data, serve), but power users often use the Ray core APIs.
4. Since people often ask about data processing with Ray and Spark, Spark use cases tend to be more geared toward structured data and CPU processing. If you are joining a bunch of tables together or running SQL queries, Spark is going to be way better. If you're working with unstructured data (images, text, video, audio, etc), need mixed CPU & GPU compute, are doing deep learning and running inference, etc, then Ray is going to be much better.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#28> From the typical Amazon EC2 customer’s perspective, this translates to saving over $120MM/year on Amazon EC2 on-demand R5 instance charges. Does the sales team know about this? /jk
Amazon sales and business teams are constantly focused on reducing costs for customers and they celebrate this internally too. I have seen dozens of big ticket “we saved this customer $xxxK/year” posts on slack and other internal venues, the customer obsession is real.
Even the reduced cost is still ten times what it costs on a traditional provider, though, right?
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#29Can you share any data on how big of a cluster is running Ray jobs?
From the blog post, the largest individual Ray cluster that was observed running a production compaction job in Q1 had 26,846 vCPUs and ~210TiB of RAM. This is roughly equivalent to a Ray cluster composed of 839 r5.8xlarge EC2 nodes (w/ 32 vCPUs and 256GiB RAM per node).
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#30Anyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.
Some of the initial differentiators are described at the bottom of our design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c... . But yes, controlling file I/O was also an important part of this since it allowed us to (1) run more targeted downloads/reads of only the Parquet row groups and columns participating in compaction and (2) track dirty/clean files to skip unnecessary re-writes of "clean"…
This is one of the first times I've heard of people using Daft in the wild. Would you be able to elaborate on where Daft came in handy?
Edit: Nvm, I kept reading! Thanks for the interesting post!