Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
1–10 of 95 posts
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#2Absolutely insane work. So much data you’d think they would come up with a custom solution instead of using the “newest available toolkit” but I understand how much of a mess dealing with that much data is.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#3Crazy that the project took almost 4 years end-to-end, and it's still ongoing.
I had no idea anything at AWS had that long of an attention span.
It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#4Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!
They reference parquet files, not sure if it's only CSV or CSV even figures in that heavily other than the first iteration before migrating to spark
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#5Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!
Hi mannyv - one of the devs that worked on the migration here. It has been a pretty long project - approached with caution due to the criticality of keeping our BI datasets healthy - but the preliminary results produced year-over-year kept looking promising enough to keep after it. =)
Also, we mostly have Parquet data cataloged in S3 today, but delimited text is indeed ubiquitous and surprisingly sticky, so we continue to maintain some very large datasets natively in this format. However, while the table's data producer may prefer to write delimited text, they are almost always converted to Parquet during the compaction process to produce a read-optimized table variant downstream.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#6Absolutely insane work. So much data you’d think they would come up with a custom solution instead of using the “newest available toolkit” but I understand how much of a mess dealing with that much data is.
Hi Narhem - one of the devs that worked on the migration here. The data volume, and subsequent compute power required to process it, is actually one of the things that led us to Ray (or Ray Core specifically) since it had the distributed computing primitives (tasks and actors) that we needed to build out our envisioned solution with very few compromises. One thing we DIDN'T want to do was just throw another one-liner SQL statement running on a new data processing framework at the problem, since that leads us back to the problems we had with Spark - not enough low-level control for such an important problem.
In short, after evaluating our options, Ray seemed to strike the best balance between the one efficiency extreme of, say, building out custom "compaction-optimized" hardware/clusters, and the other maintainability extreme of just letting the latest managed cloud service run a 1-liner SQL statement for us without ever looking under the hood.
Regardless, I expect both our existing solution and the distributed compute frameworks leveraged to deliver it to continue to evolve over time.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#7Anyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#8Anyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.
Some of the initial differentiators are described at the bottom of our design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c.... But yes, controlling file I/O was also an important part of this since it allowed us to (1) run more targeted downloads/reads of only the Parquet row groups and columns participating in compaction and (2) track dirty/clean files to skip unnecessary re-writes of "clean" files that weren't altered by compaction. Also, just better leveraging catalog metadata (e.g., primary key indexes if available) to filter out more files in the initial scan, and to copy clean files into the compacted variant by reference (when supported by the underlying catalog format).
The trick with doing compaction in Python was to ensure that the most performance-sensitive code was delegated to more optimal C++ (e.g, Ray and Arrow) and Rust (e.g., Daft) code paths. If we did all of our per-record processing ops in pure Python, compaction would indeed be much slower.
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#9Ray user here, what language actors are they using? Ray support Python Java and cpp actors…
Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
#10Anyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.
Some of the initial differentiators are described at the bottom of our design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c... . But yes, controlling file I/O was also an important part of this since it allowed us to (1) run more targeted downloads/reads of only the Parquet row groups and columns participating in compaction and (2) track dirty/clean files to skip unnecessary re-writes of "clean"…
Thanks a lot for the explanation. Sounds a lot like how Pyspark allows for declarative definitions of computation that is actually executed in Java.