> How else are you going to to create increasingly complex high-level software if not by gluing together pieces that have already been written? Are you going to create your data layer from scratch every time?
No, but I understand why you’d think that. Both the author and commenters make strange conclusions about the situation.
The problem is much easier to phrase than that. We don’t have good building blocks and tools. What does good mean? It means reliable and performant implementations, predictable behavior, well designed API surfaces (does one thing well), ability to debug and inspect, among other things.
The basic tools are already like this. Standard libraries, compilers, unix tools, file systems, certain battle tested databases. They’re all open source & move very slowly.
However, we also have countless proprietary cloud services, dependency hell in languages like JS and Rust, services scattered over different networks, regions and vendors.
So.. how did we end up here? I think it’s a combination of different reasons:
- More data both in total and per time unit. Much more people online. The world also developed towards ad tech, ML and video which demands a lot more than text and images.
- Horizontal scaling is the most cost-effective (due to physics) which prompts orders of magnitude more complex distributed systems.
- Cloud is the only option for many and gets pioneered by massive rent-seeking cynical corporations, so we lost both FOSS and any serious opportunities for standardization and simplicity. The IBM years are back.
- Money is so heavily involved, so ad-tech giants slurp up the talent and use them for market dominance, tech contributions become a side effect.
- Package managers and GitHub makes distribution easy, but results in a ginormous amount of dependencies and vendors.
So how do we fix it? We need to change culture and attitudes. A couple of tricks: Choose your deps carefully. Prefer FOSS, check issues – are these good authors? Is the API surface good? Well documented? Only use what you need, don’t let trendy blog posts FOMO you into using anything. You probably don’t need click metrics, A/B testing frameworks, minification, uglification, code splitting. You probably don’t need microservices either, or 99.999% uptime. Chances are you don’t even need multiple machines, or more than basic monitoring.