Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

aws.amazon.com

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#71

Post by refset »

smfjaw wrote: post #44
parhamn wrote: post #32

Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is m…

I work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting

Yep backtesting is critical step in the deployment of new trading algorithms.

Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#72

Post by refset »

robertnishihara wrote: post #49
dekhn wrote: post #43

Earlier quoted context omitted.

Speaking as a distributed computing nerd, Ray is definitely one of the more interesting and exciting frameworks I've seen in a while. It's one of those systems where reading the manual, I can see that I'm not going to have to learn anything new, because the mental model resembles so many distributed systems I've worked with before (I dunno about anybody else, but tensorflow is an example of a distributed system that…

I'm glad you find it exciting! Our intention from the start was for Ray to be general purpose. And the core Ray APIs are quite general (basically just scheduling a Python function somewhere in a cluster or instantiating a Python class as a process somewhere in the cluster). We had AI use cases in mind from the start, since we were grad students in AI. But the generality has really been important since AI workloads en…

Is there anybody trying to build a SQL database on Ray yet? Asking for a friend.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#74

Post by jgalt212 »

dekhn wrote: post #42
jgalt212 wrote: post #36

Slightly flip, but it's interesting that no one believes in or brags about cost savings via statistical sampling techniques these days.

well, I can save money by eating only lentils, but I prefer a richer diet. As do BI folks in a highly profitable company.

It's a terribly inelegant and inefficient solution that no one should be "proud" of. The only time you need N=all is for the general ledger.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#77

Post by PROMISE_237 »

robertnishihara wrote: post #54
zacmps wrote: post #48

Earlier quoted context omitted.

Super cool to see you here. I've also looked at ray for running data pipelines before (at much much smaller scales) for the reasons you suggest (unstructured data, mixed CPU/GPU compute). One thing I've wanted is an incremental computation framework (i.e., salsa [1]) built on ray so that I can write jobs that transparently reuse intermediate results from an object store if their dependents haven't changed. Do you kno…

Other folks have built data processing libraries on top of Ray: Modin and Daft come to mind. But I'm not aware of anything exactly like what you're referring to!

[dead]

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#78

Post by dekhn »

fs111 wrote: post #69
bigcat12345678 wrote: post #67

Earlier quoted context omitted.

Spark was never a showcase for mesos Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea In the time mesos was invented, Google…

I have no idea what any of the google tech has to do with anything here. Quoting from the original spark paper: > Spark is built on top of Mesos [16, 15], a “cluster operat- > ing system” that lets multiple parallel applications share > a cluster in a fine-grained manner and provides an API > for applications to launch tasks on a cluster https://people.csail.mit.edu/matei/papers/2010/hotcloud_spar... Note how Matei Z…

The RAD lab folks who built Mesos were aware of Borg and how it approached the problem of schedling a bunch of different jobs on a collection of disparate hardware. Prior to borg, most large-scale clusters were managed with batch queue software, while borg and mesos are more from the "service management"- a collection of jobs that run concurrently, with priority levels used to preempt lower-priority jobs to allow higher-priority jobs to schedule and run "immediately".

The need for this pops up for nearly every large scale data processing enterprise- with k8s replacing mesos, yarn, and other systems as the cluster scheduler du-jour.

One of the big advantages of a service scheduler versus a batch queue is that you can implement a batch queue on top of a service scheduler much more easily than you can implement a service scheduler on top of a batch queue.

Re: Amazon's exabyte-scale migration from Apache Spark to Ray on EC2

#80

Post by disgruntledphd2 »

jgalt212 wrote: post #74
dekhn wrote: post #42

Earlier quoted context omitted.

well, I can save money by eating only lentils, but I prefer a richer diet. As do BI folks in a highly profitable company.

It's a terribly inelegant and inefficient solution that no one should be "proud" of. The only time you need N=all is for the general ledger.

> The only time you need N=all is for the general ledger.

If you're predicting for each user, you need all of the data.

And generally you probably wouldn't want to sample too much for BI as it could lead to people making wrong decisions.

But yeah, in general sampling rocks and is super effective.

Post reply on HN