I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works me…
Fragile narrow laggy asynchronous mismatched pipes kill productivity
61–70 of 91 posts
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#62I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works me…
Also, git-annex uses reflink copies whenever possible, on zfs, btrfs, or apfs. Also, since people were talking about p2p and git, git-annex does this amazing trick for syncing directly to other git-annex repos, even with the checked out branch. There is no need at all for a seperate server.
I have used git-annex for years on OSX, and have not found it be deficient in any way compared to linux.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#63So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some…
There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it.
This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined with how the shell worked being driven by people who really really understood language and came up with one that made sense to them for interacting with computers... was, is still somewhat, wonderful.
What's missing today is exactly what you mention. People designing tool sheds with materials often shoddily designed for cathedrals.
Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find.
The favourite example of this is - perhaps apocryphal, but entirely believable - is replacing dozens of nodes using fancy big data tools with one node running sed/awk,etc.
One thing is clear, nowhere I've been has had the tools readily available, the documentation clear and forthcoming, and the scale in the right range for projects.
I found myself recently solving a problem with Hashicorp Vault using GCP to verify identity of machines wanting secrets. It was a stretch goal which had been on my plate for six months, every once in a while I would try to go back and figure out how to make it work, and months and months and months after trying, I put it together and it worked perfectly. The documentation to lead to this understanding had to be read out of order on several different pages with some lucky guesses to arrive at the solution, which in the end was just a few steps easily explained. Afterwards the documentation was fine and made perfect sense. Before I grokked the issue the documentation just seemed like a bunch on nonsense which led me to believe what I wanted to do wasn't possible in a constrained security environment.
That is the kind of problem I solve all the time as someone with a decade of DevOps,SysAdmin,whatever experience behind me. Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#64I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance. Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it c…
If there's anything I've learned across my career, it's to avoid distributed databases unless absolutely necessary, and if it is necessary, then spend a bunch of time trying to make sure you got it right. And then even after that you probably got it wrong.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#65So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some…
If you sort of turn that idea around in your head, software will always end up trying to barely work on the shittiest hardware.
Think about it. Chips work on 30 nm. What's our next step, to make 30 nm more solid and reliable? No, it's to go to 20 nm or 10 nm and get the same software to work.
The same goes with systems stuff. Do we work on having much more reliable cpus? No, we work on using lots more less-reliable cpus (or gpus) with more unreliable storage over less reliable interconnects and networks and get the software to (imperfectly) abstract away the problems.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#66>probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed.
Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic.
At the end of the day, if your customer is on the other end of the internet, then all of those complaints apply. If you solve that by running an app on their device, then oh boy are you going to have fun testing.
I prefer scaling out. The stackoverflow peeps prefer scaling up. There are some great write-ups about how they scale. I found this [1] one after some quick googling, but I am certain there are more. So it's really about choose your poison.
>I think people should be more willing to try and write performance-sensitive code as a (potentially multi-threaded) process on one machine in a fast language if it’ll fit rather than try and distribute a slower implementation over multiple machines.
Sure. I once replaced a system that ran on 10 32-core machines with one that ran on one with four cores on one machine and did the work in the same time. Another time I had 96 cores, more threads, and I replaced it with one that had three threads and was faster.
But both of those solutions were evolutionary dead-ends. The tasks were very specific, and not subject to change. The first one was a single C file. The latter was actually java, but with hand-rolled hash tables and optimistic locks. The first one I doubt I could follow it now.
My point is, you can have understandable systems that good people (as opposed to geniuses) can work on, evolve and adapt, and that have well understood failure modes and scaling cliffs. Or you can have bonkers code that everyone is afraid to touch, and which fails in production when it hit a cliff you didn't know about and now your site is dead for eight days.
If you can strike a good balance, then you'll probably have some combination of distributed, and brute force.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#67Really like the curiosity and thought behind this article. Couple of thoughts: >probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic. At the end of the day, if your customer is on the other end of the internet, then all of those comp…
Deeply false equivalence.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#68Of course they do. But there's no alternative. No matter how fast or beefy your server is, these days if your product becomes a success, 99% of the time it will outgrow what's possible on a single server. (Not to mention needs for redundancy, geographic latency, etc.) And by the time you see the trend heading upwards so you can predict what day that will happen, you already won't have the time for the massive rewrite…
[1] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...
Also, I don't think your post deserved a down-vote.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#69Hah, I saw the title "Fragile narrow laggy asynchronous mismatched pipes kill productivity" and thought it was about the pitfalls of trying to coordinate remote teams across disparate time zones.
Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity
#70There's a point here about microservices.