Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 95 posts

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#61

Post by jaychia »

I work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now!

We love Ray, and are excited about the awesome ecosystem of useful + scalable tools that run on it for model training and serving. We hope that Daft can complement the rest of the Ray ecosystem to enable large scale ETL/analytics to also run on your existing Ray clusters. If you have an existing Ray cluster setup, you absolutely should have access to best-in-class ETL/analytics without having to run a separate Spark cluster.

Also, on the nerdier side of things - the primitives that Ray provides gives us a real opportunity to build a solid non-JVM based, vectorized distributed query engine. We’re already seeing extremely good performance improvements here vs Spark, and are really excited about some of the upcoming work to get even better performance and memory stability.

This collaboration with Amazon really battle-tested our framework :) happy to answer any questions if folks have them.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#62

Post by hiyer »

We chose Ray over Spark in my previous company mostly because we were a Python shop and Ray is Python-native (though it's implemented in C++ I believe). It worked very well for us even for real-time queries - though we were obviously nowhere near the scale that AWS is at.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#63

Post by mr_toad »

parhamn wrote: post #32

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is m…

Customer retention, infrequent purchases, and time series forecasts all benefit from having at least several years of data.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#65

Post by physicsguy »

gregw2 wrote: post #33
parhamn wrote: post #32

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is m…

If you have seasonal demand patterns, you generally need three years history to do good predictive analytics. I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now u…

Yeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#67

Post by bigcat12345678 »

fs111 wrote: post #26
zitterbewegung wrote: post #23

I was in a workshop that taught me Ray. It was interesting to know that the people who started Spark were also involved in making Ray. This is not badmouthing either project just an observation and if you architected one task you would be good at attacking the same problem better .

Well spark was really a showcase Project for mesos when it was created. Now everyone knows a lot more

Spark was never a showcase for mesos

Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce

Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea

In the time mesos was invented, Google had its own mesos, which is a similar project by learning wrong lesson from Borg. That thing is Omega.

Eventually everyone decided that Borg was right all along, thus the kubernetes

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#68

Post by sofixa »

immibis wrote: post #28
andrewxdiamond wrote: post #22

Earlier quoted context omitted.

Amazon sales and business teams are constantly focused on reducing costs for customers and they celebrate this internally too. I have seen dozens of big ticket “we saved this customer $xxxK/year” posts on slack and other internal venues, the customer obsession is real.

Even the reduced cost is still ten times what it costs on a traditional provider, though, right?

Wildly depends. How many people would it take to set up, deploy, maintain, secure, back up something like S3, RDS, SQS similarly redundantly based on top of just raw compute?

For some organisations, it will make sense to internalise and DIY everything. For many others, AWS/equivalent enables them to break down internal silos (instead of having to file a ticket to get storage team to provision storage for your new fancy RabbitMQ cluster, multiplied by 10 for networking, security AV bullshit, etc.) and just consume everything as a service. It's much faster, and in some cases even maybe cheaper.

Cf. Dropbox that started on S3, and a decade afterwards, migrated to self-managed datacenters.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#69

Post by fs111 »

bigcat12345678 wrote: post #67
fs111 wrote: post #26

Earlier quoted context omitted.

Well spark was really a showcase Project for mesos when it was created. Now everyone knows a lot more

Spark was never a showcase for mesos Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea In the time mesos was invented, Google…

I have no idea what any of the google tech has to do with anything here.

Quoting from the original spark paper:

> Spark is built on top of Mesos [16, 15], a “cluster operat- > ing system” that lets multiple parallel applications share > a cluster in a fine-grained manner and provides an API > for applications to launch tasks on a cluster

https://people.csail.mit.edu/matei/papers/2010/hotcloud_spar...

Note how Matei Zaharia - the inventor of spark - is also on the mesos paper:

https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#70

Post by rubenvanwyk »

dekhn wrote: post #42
jgalt212 wrote: post #36

Slightly flip, but it's interesting that no one believes in or brags about cost savings via statistical sampling techniques these days.

well, I can save money by eating only lentils, but I prefer a richer diet. As do BI folks in a highly profitable company.

Winning comment.
Post reply on HN