91–95 of 95 posts
Earlier quoted context omitted.
Wow that is not NEARLY as expensive as I would have imagined considering the scale of the data involved.
That's also $41k / 26k / 13k per day or $1.2M / 767k / 375k per month!
I wonder what they’re doing to combat the growth rate of their data. A 13x speed up, or 82% cost reduction is great, but doesn’t seem significant enough compared to the growth of the business and (my assumption) demand for adding new data sources and added data to existing sources. Like, if the current latency is ~60 minutes for 90% of updates, will it ever be better than that? Won’t it just slowly degrade until the…
As noted in the bottom of the design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c... , we also improved the runtime efficiency of compaction from O(nlogn) to O(n). However, a lot of this also comes down to making intentional data engineering decisions to control how physical data is laid out (and retained) across files to keep reads/writes as localized as possible. For example, we found that gr…
I'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training,…
> this is not a typical ray use case Must be good enough if you're willing to dogfood it though?
Data processing workloads are quite common on Ray, especially with unstructured data.
Also, I work on Ray, which is the underlying framework used here, but all the work in the post was done by the Amazon team.
Has anyone found a good ELI5 site for all the different AI toolsets. Eg Ray, Databricks, Notebooks etc
Some VCs try to categorize tools. Github's "awesome" lists are an alternative: https://github.com/mahseema/awesome-ai-tools