Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Predicting the Future of Distributed Systems

blog.colinbreck.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Predicting the Future of Distributed Systems

#41

Post by choeger »

cyberax wrote: post #17
mikepurvis wrote: post #15

Earlier quoted context omitted.

It boggles my mind that people accept architectures where the only dev story is a duplicate cloud instance of the required services. Being able to bring the whole application up locally should be an absolute non-negotiable.

> Being able to bring the whole application up locally should be an absolute non-negotiable. This usually doesn't work that well for larger systems with services split between multiple teams. And it's not typically the RAM/CPU limitations that are the problem, but the amount of configuration that needs to be customized (and, in some cases, data). Sooner or later, you just start testing with the other teams' productio…

> Sooner or later, you just start testing with the other teams' production/staging environments rather than deal with local incompatibilities.

That's probably about the time when your development pace goes downhill.

I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to have a mock of it. That policy increases the development effort by at least a factor of two (you always have to create the mock alongside the thing), but it's just a linear increase.

Re: Predicting the Future of Distributed Systems

#42

Post by mikepurvis »

choeger wrote: post #41
cyberax wrote: post #17

Earlier quoted context omitted.

> Being able to bring the whole application up locally should be an absolute non-negotiable. This usually doesn't work that well for larger systems with services split between multiple teams. And it's not typically the RAM/CPU limitations that are the problem, but the amount of configuration that needs to be customized (and, in some cases, data). Sooner or later, you just start testing with the other teams' productio…

> Sooner or later, you just start testing with the other teams' production/staging environments rather than deal with local incompatibilities. That's probably about the time when your development pace goes downhill. I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to have a mock of it. That policy increases the development effort by at least a fact…

In theory it should be the cloud providers themselves maintaining the locally-runnable stand-ins for their services, but as it stands you basically either get it as a third party effort (MinIO for S3) or in cases where the service is just a hosted version of some existing OSS product (Postgres for RDS).

Either way, once the local version exists, then the job becomes maintaining all the infrastructure that lets you bring up the pieces, populate them with reasonable state and wire them into whatever the bits are that are being actively hacked-on.

Re: Predicting the Future of Distributed Systems

#43

Post by cyberax »

choeger wrote: post #41
cyberax wrote: post #17

Earlier quoted context omitted.

> Being able to bring the whole application up locally should be an absolute non-negotiable. This usually doesn't work that well for larger systems with services split between multiple teams. And it's not typically the RAM/CPU limitations that are the problem, but the amount of configuration that needs to be customized (and, in some cases, data). Sooner or later, you just start testing with the other teams' productio…

> Sooner or later, you just start testing with the other teams' production/staging environments rather than deal with local incompatibilities. That's probably about the time when your development pace goes downhill. I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to have a mock of it. That policy increases the development effort by at least a fact…

> That's probably about the time when your development pace goes downhill.

Oh, absolutely. But at this point, your team is probably around several dozen people and you have a product with paying customers. This naturally slows the development speed, however you organize the development process.

> I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to have a mock of it. That policy increases the development effort by at least a factor of two (you always have to create the mock alongside the thing), but it's just a linear increase.

The problem is, you can't really recapture the actual behavior of a service in a mock.

To give you an example, DynamoDB in AWS has a local mock in-memory DB for testing and development. It has nearly the same functionality, but stores all the data in RAM. So the simulated global secondary indexes (something like table views in classic SQL databases) are updated instantly. But on the real database it's eventually consistent, and it can take a fraction of a second to update.

So when you try to use your service in production, it can start breaking under the load.

Perhaps, we need better mocks that also simulate the behavior of the real services for delays, retries, and so on.

Re: Predicting the Future of Distributed Systems

#44

Post by __turbobrew__ »

singron wrote: post #14
jauntywundrkind wrote: post #7

Earlier quoted context omitted.

OTel being a capture & ingest only specification is kind of messed up. There's no attempt from what I can tell for how to query or present stored data; it's just an over-the-wire specification, & that drastically limits usable scope. It means vendors each get to make their own services & backends & tools, but it's greviously limiting the effort as a whole, makes even an open spec like OTel a one-way door. Ideally OTe…

Otel works as a standard since there isn't any need to innovate at that level. Despite the over complications it has, all the implementations largely have the same requirements, and it's useful to instrument everything the same way. Querying unfortunately has lots of room for innovation, and it's really hard to nail down in a spec especially when the vendors all want to compete.

Otel is nice and all but I still think you are best off going 100% all in prometheus. Prometheus is so common that it has become a de-facto standard in metrics.

At BigCo we have migrated a number of internal things to Otel but I don’t think it has been worth the effort.

So many projects come with prometheus metrics, dashboards, and alerts out of the box that it becomes hard to use anything else. When I pick some random helm chart to install you can almost guarantee that is comes with prometheus integrations.

With grafana mimir you can now scale easily to a few billion metrics streams so a lot of the issues with the old model of prometheus have been fixed.

Like you said I don’t think there is much to innovate on in this area, which is a good thing.

Re: Predicting the Future of Distributed Systems

#45

Post by __turbobrew__ »

Pushing as much down to the infra sounds like aws lambda and friends. You basically upload a zip or container and say, “just run this business code somewhere, I don’t care”. OCI bundles are basically a two day door at this point, you can build them with many tools, and run them with many other tools.

It works great for stateless things, but not so great for stateful things. I guess this plays into state being persisted in object storage or DBs, this allows the application to be stateless.

Re: Predicting the Future of Distributed Systems

#46

Post by filterfish »

purpleidea wrote: post #25

> Programming Models If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence. I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing i…

> ... closed-loop feedback systems ...

It's good to actually see even a mention of control theory.

My degree was electronics and control theory and whilst I've only had one job that involved either electronics or control theory I often think about software in these terms: I genuinely think that as an industry we need to seriously consider the systems we build in control theoretic terms.

Re: Predicting the Future of Distributed Systems

#47

Post by KAKAN »

cyberax wrote: post #43
choeger wrote: post #41

Earlier quoted context omitted.

> Sooner or later, you just start testing with the other teams' production/staging environments rather than deal with local incompatibilities. That's probably about the time when your development pace goes downhill. I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to have a mock of it. That policy increases the development effort by at least a fact…

> That's probably about the time when your development pace goes downhill. Oh, absolutely. But at this point, your team is probably around several dozen people and you have a product with paying customers. This naturally slows the development speed, however you organize the development process. > I think it's an interesting idea to consider: If some team interfaces with something outside of its control, they need to…

This reminds me of an article I read somewhere (probably here in HN) wherein people implementing Banking Services just straight up test the API in Production after a few cycles of mock development, due to constantly having to deal with edge cases not present in the dev env.

Re: Predicting the Future of Distributed Systems

#48

Post by pjmlp »

hnthrow289570 wrote: post #32
karmarepellent wrote: post #29

Earlier quoted context omitted.

I would be hesitant to claim "the world is moving to" anything, really. Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time. And not only because of legacy systems that are hard to migrate to a modern platform. At my place of work there are workloads that can easily run on Kubernetes and it would be wise to do so. On…

>Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time. I think there is even a widening talent gap where you can't get people excited about doing something that maybe should have been done years ago (assuming VM -> containers makes sense for a thing). The salary needs to go higher for things that are less beneficial to…

Even Microsoft themselves have a bunch of products that still require .NET Framework.

SharePoint CSM, Dynamics, SQL Server CLR, Visual Studio extensions, Office AddIns.

Re: Predicting the Future of Distributed Systems

#49

Post by pjmlp »

purpleidea wrote: post #36
lifty wrote: post #28

Earlier quoted context omitted.

I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works ( https://talos.dev ).

> I think the world is moving to more immutable systems Mgmt doesn't care whether or not you want to build your system to be immutable, that's up to you! Mgmt let's you glue together the different pieces with a safe, reactive, distributed DSL. Regarding your Talos comment, Kubernetes makes building things so complicated, so no, I don't think it will win out long term.

There is a reason why most cloud now sell managed Kubernetes.

Re: Predicting the Future of Distributed Systems

#50

Post by pjmlp »

mikepurvis wrote: post #15
awkii wrote: post #4

I think the author has a point with one-way doors slowing down the adoption of distributed systems. The best way to build two way doors is to push for industry adoption of a particular API. In theory the backend of these APIs matter little to me, the developer, so long as they are fast and consistent. Some examples that come to mind is that Apache Beam is a "programming model" for Data pipelines, Akka is a "programmi…

It boggles my mind that people accept architectures where the only dev story is a duplicate cloud instance of the required services. Being able to bring the whole application up locally should be an absolute non-negotiable.

We are back to timesharing days, in better clothing, and that is non negotiable from management point of view.
Post reply on HN