Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#81

Post by immibis »

sofixa wrote: post #68
immibis wrote: post #28

Earlier quoted context omitted.

Even the reduced cost is still ten times what it costs on a traditional provider, though, right?

Wildly depends. How many people would it take to set up, deploy, maintain, secure, back up something like S3, RDS, SQS similarly redundantly based on top of just raw compute? For some organisations, it will make sense to internalise and DIY everything. For many others, AWS/equivalent enables them to break down internal silos (instead of having to file a ticket to get storage team to provision storage for your new fan…

If we ignore data egress fees, S3 is a rare exception to the rule that all of AWS is 10 times overpriced.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#82

Post by jonmoore »

zacmps wrote: post #48
robertnishihara wrote: post #27

I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…

Super cool to see you here. I've also looked at ray for running data pipelines before (at much much smaller scales) for the reasons you suggest (unstructured data, mixed CPU/GPU compute). One thing I've wanted is an incremental computation framework (i.e., salsa [1]) built on ray so that I can write jobs that transparently reuse intermediate results from an object store if their dependents haven't changed. Do you kno…

I asked the same question to one of the core devs at a recent event and he (1) said that some people in finance have done related things and (2) suggested using the Ray slack to connect with developers and power users who might have helpful advice.

I agree this is a very interesting area to consider Ray for. There are lots of projects/products that provide core components that could be used but there’s no widely used library. It feels like one is overdue.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#83

Post by PeterCorless »

I remember when everyone shifted from Apache Hadoop to Apache Spark. This seems like a possibly similar sea change. I am not sure if many other users will embrace Ray over Spark, but this is a sign that people are looking to either improve Spark on some fundamental levels, or are going to reach out to new technologies to resolve their problems. Cool stuff.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#84

Post by e28eta »

I wonder what they’re doing to combat the growth rate of their data. A 13x speed up, or 82% cost reduction is great, but doesn’t seem significant enough compared to the growth of the business and (my assumption) demand for adding new data sources and added data to existing sources.

Like, if the current latency is ~60 minutes for 90% of updates, will it ever be better than that? Won’t it just slowly degrade until the next multi-year migration?

PS: this article was infuriating to read on iPad - it kept jumping back to the top of the page and couldn’t figure out why

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#85

Post by thedood »

jaychia wrote: post #61

I work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now! We love Ray, and are excited about the awesome ecosystem of useful + scalable tools that run on it for model training and serving. We hope that Daft can complement the rest of the Ray ecosystem to enable large scale ETL/analytics to also run on your existing Ray clusters. If you have an existing Ray cluster s…

Good to see you here! It's been great working with Daft to further improve data processing on Ray, and the early results of incorporating Daft into the compactor have been very impressive. Also agree with the overall sentiment here that Ray clusters should be able to run best-in-class ETL without requiring a separate cluster maintained by another framework (Spark or otherwise). This also creates an opportunity to avoid many inefficient, high-latency cross-cluster data exchange ops often run out of necessity today (e.g., through an intermediate cloud storage layer like S3).

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#86

Post by jedberg »

parhamn wrote: post #32

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is m…

Training a foundation model that understands shopping queries. 25 years of shopping data goes a long way.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#87

Post by thedood »

e28eta wrote: post #84

I wonder what they’re doing to combat the growth rate of their data. A 13x speed up, or 82% cost reduction is great, but doesn’t seem significant enough compared to the growth of the business and (my assumption) demand for adding new data sources and added data to existing sources. Like, if the current latency is ~60 minutes for 90% of updates, will it ever be better than that? Won’t it just slowly degrade until the…

As noted in the bottom of the design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c..., we also improved the runtime efficiency of compaction from O(nlogn) to O(n). However, a lot of this also comes down to making intentional data engineering decisions to control how physical data is laid out (and retained) across files to keep reads/writes as localized as possible. For example, we found that grouping records according to the date they were last updated to be very helpful, as outlined in our 2022 Ray Summit talk: https://youtu.be/u1XqELIRabI?t=1589.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#88

Post by LarsDu88 »

layoric wrote: post #52
thedood wrote: post #29

Earlier quoted context omitted.

From the blog post, the largest individual Ray cluster that was observed running a production compaction job in Q1 had 26,846 vCPUs and ~210TiB of RAM. This is roughly equivalent to a Ray cluster composed of 839 r5.8xlarge EC2 nodes (w/ 32 vCPUs and 256GiB RAM per node).

For those interested, this would be at a cost of: - ~$1691/hour on demand - ~$1065/hour reserved - ~$521/hour spot Not including any related data transfer costs.

Wow that is not NEARLY as expensive as I would have imagined considering the scale of the data involved.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#90

Post by artemisart »

LarsDu88 wrote: post #88
layoric wrote: post #52

Earlier quoted context omitted.

For those interested, this would be at a cost of: - ~$1691/hour on demand - ~$1065/hour reserved - ~$521/hour spot Not including any related data transfer costs.

Wow that is not NEARLY as expensive as I would have imagined considering the scale of the data involved.

That's also $41k / 26k / 13k per day or $1.2M / 767k / 375k per month!
Post reply on HN