exa-d is our internal data processing framework that stores the web in S3. It helps deal with the complexity of data at (web) scale using specific design decisions like declarative typed dependencies and enabling sparse updates.
Exa-d: How to store the web in S3
1–6 of 6 posts
Re: Exa-d: How to store the web in S3
#2[deleted]
Re: Exa-d: How to store the web in S3
#3hi will! super nicely written, nice look under the hood of your processing. as an orchestration guy i always wondered why everyone seems to converge on using Ray, and as a secondary thought, how well is Anyscale capturing the Ray market.
if i were doing what you do i might set up a lot of rate limits/anomaly detection in case some weird unintended invalidation causes a weird spike in your dependency graphs. is there good practice there for anomaly detection other than "setup a bunhc of dashboards and be on call"?
Re: Exa-d: How to store the web in S3
#4Opening this page makes my (quite beefy) machine grind to a halt! Almost all CPU threads and the GPU jump up to 80% usage
Re: Exa-d: How to store the web in S3
#5Opening this page makes my (quite beefy) machine grind to a halt! Almost all CPU threads and the GPU jump up to 80% usage
Would be funny if blog visitors were the distributed compute nodes.
Re: Exa-d: How to store the web in S3
#6hi will! super nicely written, nice look under the hood of your processing. as an orchestration guy i always wondered why everyone seems to converge on using Ray, and as a secondary thought, how well is Anyscale capturing the Ray market. if i were doing what you do i might set up a lot of rate limits/anomaly detection in case some weird unintended invalidation causes a weird spike in your dependency graphs. is there…
Ray is the future