Do you have a moment to talk about our Lord and Saviour, Erlang/OTP?
Fragile narrow laggy asynchronous mismatched pipes kill productivity
11–20 of 91 posts
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#12I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems.
It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME.
-----
I would like to use something like git to store the source code to every piece of software I use, and the binaries. That is, most of a whole Linux distro.
I have been loosely following some "git for binary data" projects for a number of years. I looked at IPFS like 5 years ago but it seems to have gone off the rails. The dat project seems to have morphed into something else?
Are there any new storage projects in that vein? I think the OP is identifying a real problem -- distributed systems are unreliable, and you can get a lot done on a single machine. But we are missing some primitives that would enable that. Every application is littered with buggy and difficut network logic, rather than having a single tool like git (or rsync) which would handle the problem in a focused and fast way.
It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue.
Sort of related is a fast distro here I'm looking at: https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#13Just reading the title I assumed it was a post about business processes and communication between teams. Because this is how working for a big corp sometimes feel.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#14Just reading the title I assumed it was a post about business processes and communication between teams. Because this is how working for a big corp sometimes feel.
It's a bit of Conway's law: the software structure mirrors the organization structure.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#15Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the…
[deleted]
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#16I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#17Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the…
As a developer who's trying to move on to either a management or product role, failure modes is one of the things I want to emphasize in that role. The sad fact is that so many product owners really don't understand how software works or gets built, and as such, they are unequipped to reason about such things.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#18> Sometimes a distributed system is unavoidable, such as if you want extreme availability or computing power, but other times it’s totally avoidable. But so much of our sales pitch involves these shiny cloud systems. Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"?
>Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"?
I did many times. Many practical businesses are looking how to solve their real problems in reliable and cheapest way. They mostly do not give a flying hoot about all those buzzwords and super duper newest tech. They just ask how much, how exactly will it work, what are hidden cost, how will it be maintained and they will also look at your profile (clients, references, examples of finished projects etc).
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#19I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization.
These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data.
I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller problem domain than the one described in the article.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#20I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource expertise.
The vision of the service mesh is to make unreliability and security no longer the job if the application binary. Even without a service mesh, you can put a lot of common functionality into a reverse proxy. Personally I am loving OpenResty for the simplicity of writing adapters and oauth at the proxy layer with good performance.