Fragile narrow laggy asynchronous mismatched pipes kill productivity

Of course they do. But there's no alternative.

No matter how fast or beefy your server is, these days if your product becomes a success, 99% of the time it will outgrow what's possible on a single server. (Not to mention needs for redundancy, geographic latency, etc.) And by the time you see the trend heading upwards so you can predict what day that will happen, you already won't have the time for the massive rewrite necessary.

So yes, it's tons slower to write distributed servers/systems. But what other choice do you usually have?

Though, as much as possible, you can try to avoid the microservices route, and integrate everything as much as possible into monolithic replicable "full-stack servers" that never talk to each other, but rather rely entirely on things like cloud storage and cloud database. Where you're paying your cloud provider $$$ to not fail, rather than handle it yourself. Sometimes this will work for you, sometimes it won't.

Not sure how well if fits your use-case, but I've been very happy with git-lfs in combination with a NAS I have at home. The NAS is just mounted as a normal network drive and available to use with LFS via lfs-filestore[0] and available on the go via the builtin DynDNS + VPC of the NAS.

I've been using it for a repo where I store all my university/academic stuff like lectures (recordings), PDFs of books and papers, Anki decks, etc., and it has now grown to be ~120GB big.

Biggest issue was that I'm always running low on disk space with my laptop, and git-lfs doesn't have a good built-in way to only checkout part of the files on your machine, so I built a small tool to make that easier[1]. Since I've been using that it's been a pretty smooth ride.

[0]: https://github.com/sinbad/lfs-folderstore

[1]: https://github.com/hobofan/lfs-unload

I've seen a handful of applications that attempt to "scale" by going down the micro-service route in a completely flawed way. Only to end up with a tangled mess that's impossible to reliably debug. All progress halts.

There's nothing inherently wrong about your statement, just that it's still far to easy to write shitty distributed system, and so much easier to push that complexity off onto the OS or even network layer itself.

Why should I the application developer care about the way my user's data enters my DB? This should be tightly abstracted away, and traced/logged accordingly. Leave it to me the systems developer to get the details right, and share the fruits of my labor with everyone.

I can imagine no system more deserving of shared resources than network technology. Try and imagine a world without TCP/IP, do you not end up with something similar?

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses.

Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago.

Then there are some folks/organizations building cathedrals, and the associated tooling (react, angular, maven, etc) and all the rest of us look up in awe and thing "well I guess if I want to be that good I need to use those tools on this dog house."

But your dog house doesn't have the need to host parties, provide security, or even real weather protection other than a roof to keep the rain and sun out. Yet we all try to build our dog houses in ways that might be better if they are one day converted to a proper living quarters but likely will never have a need for running Water or windows.

Hah, I saw the title "Fragile narrow laggy asynchronous mismatched pipes kill productivity" and thought it was about the pitfalls of trying to coordinate remote teams across disparate time zones.