I'd like to add that I'm seeing more and more companies unifying synchronous and asynchronous APIs. With the concept of GraphQL Federation, it's possible to "extend" Entities by defining their (primary) keys in a GraphQL Schema. If we're combining this with Async APIs, e.g. NATS or Kafka, we can enable teams to build APIs around events, while still being able to distribute the implementation of how certain fields can be resolved. The Federation Router then joins the Stream with additional data from synchronous services, a very powerful pattern I believe. I wrote a bit more on the topic here: https://wundergraph.com/blog/distributed_graphql_subscriptio...
Predicting the Future of Distributed Systems
21–30 of 51 posts
Re: Predicting the Future of Distributed Systems
#22Things I have come to know about distributed systems:
The S3 API (object storage) is the accepted storage API, but you do not need AWS (but they are very good at this).
The Kafka API is the accepted stream/ buffer/ queue API, but you do not need Confluent.
SQL is the query language, but you do not need a relational database.
Re: Predicting the Future of Distributed Systems
#23Something I anticipate is smarter storage that can do some filtering on push down predicates. There's compute on the storage nodes that is being wasted today.
I was kinda expecting BigQuery to do this under the hood, but it seems like they don't, which is a shame. BigQuery isn't faster than, say, trino on gcs, even though Google could do some major optimisations here.
Re: Predicting the Future of Distributed Systems
#24Things I have come to know about distributed systems: The S3 API (object storage) is the accepted storage API, but you do not need AWS (but they are very good at this). The Kafka API is the accepted stream/ buffer/ queue API, but you do not need Confluent. SQL is the query language, but you do not need a relational database.
I'd argue that a lot of people are moving from Kafka to NATS. NATS and Kafka serve different purposes and for many use cases related to APIs, NATS has a lot more to offer, like like wildcard topic topologies.
Re: Predicting the Future of Distributed Systems
#25> Programming Models
If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence.
I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing it. TBQH, I'm disenchanted by the fact that it seems to get any recognition you need to have VC's and a three-year timeline, short-term goals, and a plan to be done by then or move on.
If you're serious about future infra, then it's all here:
https://github.com/purpleidea/mgmt/
Looking for coding help for some of the harder bits that people might wish to add, and for people to take it into production and find issues that we've missed.
Re: Predicting the Future of Distributed Systems
#26Something I anticipate is smarter storage that can do some filtering on push down predicates. There's compute on the storage nodes that is being wasted today. I was kinda expecting BigQuery to do this under the hood, but it seems like they don't, which is a shame. BigQuery isn't faster than, say, trino on gcs, even though Google could do some major optimisations here.
I also wonder if Athena does this with AWS. Parquet supports pushdown. But i would suspect, pushdown predicates would mean that the file storage unit has to have some logic to execute custom code, bringing back the code to the data. The promise of spark, once. It would be a huge win, definitly. Hmmm.
But opens up also a threat vector. And you have competing users running their predicates. So one has to think also about queues and pipelining and so on. But probably also solvable, just like on any multiuser system.
Interesting.
Re: Predicting the Future of Distributed Systems
#27Things I have come to know about distributed systems: The S3 API (object storage) is the accepted storage API, but you do not need AWS (but they are very good at this). The Kafka API is the accepted stream/ buffer/ queue API, but you do not need Confluent. SQL is the query language, but you do not need a relational database.
I'd argue that a lot of people are moving from Kafka to NATS. NATS and Kafka serve different purposes and for many use cases related to APIs, NATS has a lot more to offer, like like wildcard topic topologies.
it's the Kafka API, not Kafka itself, that I see as having become the standard.
Re: Predicting the Future of Distributed Systems
#28> Programming Models If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence. I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing i…
I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works (https://talos.dev).
Re: Predicting the Future of Distributed Systems
#29> Programming Models If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence. I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing i…
I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works ( https://talos.dev ).
I would be hesitant to claim "the world is moving to" anything, really. Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time.
And not only because of legacy systems that are hard to migrate to a modern platform. At my place of work there are workloads that can easily run on Kubernetes and it would be wise to do so. On the other hand there are systems that are not designed to run in a container and there is frankly no need to, because not everything needs to scale up and down or be available 100% of the time at all costs.
I think configuration management systems like mgmt (or Ansible and Puppet) are here to stay.
Re: Predicting the Future of Distributed Systems
#30Earlier quoted context omitted.
I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works ( https://talos.dev ).
I would be hesitant to claim "the world is moving to" anything, really. Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time. And not only because of legacy systems that are hard to migrate to a modern platform. At my place of work there are workloads that can easily run on Kubernetes and it would be wise to do so. On…
Not disagreeing with you there; technology lingers for many years. But in terms of market share and mind share, configuration management has shrank in dominance and I suspect it will continue to do so.