Predicting the future of distributed systems
blog.colinbreck.com
Predicting the future of distributed systems
1–10 of 19 posts
Re: Predicting the future of distributed systems
#2I dunno, Postgres seems pretty solid.
Re: Predicting the future of distributed systems
#3A new programming model for distributed computing is desperately needed. Something between a full operational system a la Temporal, but without the extreme operational overhead + a sane cooperative runtime like Golang.
I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
Re: Predicting the future of distributed systems
#4This was on the front page 3 days ago #deng
Re: Predicting the future of distributed systems
#5I don't buy this "object storage + Iceberg is all you need for OLAP" hype. If the application has sustained query load, it makes sense to provision servers rather than go pure serverless. If there are already provisioned servers, it makes sense to cache data on them (either in their memory or SSDs) to avoid round-trips to object storage. This is the architecture of the latest breed of OLAP databases such as Databend and Firebolt, as well as the latest iterations of Redshift's and Snowflake's architecture. Also, this is the approach of the newest breed of vector databases, such as Turbopuffer.
For OLAP use cases with real-time data ingestion requirements, object-storage-only approach also leads to write amplification. Therefore, I don't think that architectures like Apache Pinot, Apache Paimon, and Apache Druid are going anywhere.
Another problem with "open table formats" like Iceberg, Hudi, and Delta Lake is their slow innovation speed.
I've recently argued about this at greater length here: https://engineeringideas.substack.com/p/the-future-of-olap-t...
Re: Predicting the future of distributed systems
#6A new programming model for distributed computing is desperately needed. Something between a full operational system a la Temporal, but without the extreme operational overhead + a sane cooperative runtime like Golang. I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
Ray? Terrible name, so I cannot find anything but a Go debugger? Do you have a link?
Re: Predicting the future of distributed systems
#7A new programming model for distributed computing is desperately needed. Something between a full operational system a la Temporal, but without the extreme operational overhead + a sane cooperative runtime like Golang. I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
Ray? Terrible name, so I cannot find anything but a Go debugger? Do you have a link?
Re: Predicting the future of distributed systems
#8A new programming model for distributed computing is desperately needed. Something between a full operational system a la Temporal, but without the extreme operational overhead + a sane cooperative runtime like Golang. I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
Wouldn't you say the Unison programming language is the closest so far?
Re: Predicting the future of distributed systems
#9Having superficial knowledge about data storage/DBMS, perhaps I am just spewing nonsense. But I always imagine that a compaction layer for the long-tail/old data should be a standard, where less accessed(sometimes once a month) is pushed to a s3(but still queryable) like storage.
Re: Predicting the future of distributed systems
#10A new programming model for distributed computing is desperately needed. Something between a full operational system a la Temporal, but without the extreme operational overhead + a sane cooperative runtime like Golang. I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
> A new programming model for distributed computing is desperately needed
The article mentions actor-model frameworks like Akka. Is that not like Ray?
At work we use and maintain something similar called Cloud Haskell (confusingly implemented in a package called distributed-process: https://github.com/haskell-distributed/distributed-process) and I have to say that using it is a breeze.