Ask HN: Have we screwed ourselves as software engineers?

Hmm, incompetent folks exist and need to be let go. However, I think it is a slightly different category than RDD.

No one has given you a good explanation for why to move to the cloud?

Let me fix that for you! :)

I work for a Fortune 200 company who has their own data centers and has for decades. We just completed the construction of two new data centers about four years ago at a cost of $110 million dollars. Those new data centers are now at 70% of their capacity for power requirements. You should take a look at the specs on Intel's latest server-class processors. It's insane! We're talking half horse power and higher for each processor - and you want blades full of these things and then a rack of those blades and a row of those racks! Data centers now consume more power than industrial manufacturing! Most maddening of all - most of that power is going to go to heat, and guess what? You need giant chiller units to cool the place down. The power costs alone are insane.

You also lose agility. Need a new server? Go to your favorite cloud console and provision it. Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?

You already know you need to staff to manage and patch your servers but you're also going to need staff to procure and manage your software licenses - and those licenses aren't cheap! Don't forget the ongoing tail - you pay for the license and then you get to pay 20% per annum forever after for that license. Those licenses also restrict your agility - you're pressured to use the software that's licensed in order to get better value. Try working with a procurement-based architecture!

You think you're going to avoid the software licensing hell by using open source? Good luck with that! That means you are responsible for the packaging, distribution, and support of whatever it is you're utilizing. That's more staff you need. They will inevitably miss patching some critical vulnerability that's going to land you in the news!

I can go on, and on, and on. But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above! We have ultimate flexibility and agility, and we can quickly utilize managed open source platforms to solve business problems. On-prem? You lose all of that.

Faster, cheaper, better - that's why people are moving to the cloud.

It seems to me the 'problem' has very little to do with software engineering and mostly just with marketing a significantly wider gradient en skills and quality.

"Back in the day" you really couldn't get things done if you didn't actually know how things worked. Today, you can do a little learning (which is still a dangerous thing) and based on low quality requirements create a buzzword-friendly applications with 1000 dependencies you neither know about nor have to check for. That was however just a side-effect of something that is a good thing: composability.

But that dives into the technical side of things, in reality, the marketing of technologies as a product and the involvement of human middleware (management) in things they have no business being involved in causes most of the perceived problems. That is not something that is really caused purely by software engineers, nor can it be solved by just them.

Two ways to go about it could be:

  1. Having all the human overhead go through the same requirements and QA process as everything else
  2. Be better at marketing your own solution (but make sure it has the correct technical and business underpinnings)

This doesn't work in legacy hierarchical work environments, and you're essentially just screwed if you are stuck in one of those. Best to either stop worrying about the technology in one of those situations, or move on to somewhere else.

Well, some things got easier and other things got harder. Some of the harder things are just due to higher consumer expectations or more competition, but, harder they are.

The easier things are mostly obvious. Better languages, open source libs etc. Things that are harder now than before:

1. UI. The web is not a good UI platform, sorry. Designing UI in the 90s was easier, except for the need to do manual memory management if you weren't using Visual Basic. Partly because there was little expectation of branded UI, so you could easily re-use large control libraries that came with the OS which were/are pretty feature complete and well documented.

2. Cross language interop. Microsoft had this nailed. COM was a beast, but it worked and there was an actual real market of cross-language, auto-bound objects and GUI controls (COM objects, OCX controls). There was actually a thriving ecosystem of languages on Windows which are now mostly forgotten (Delphi, FoxPro, VB6, Paradox, Visual Prolog etc). Nowadays cross-language interop is a joke. Transpile to JavaScript or go via a C FFI, maybe, if you're lucky.

3. Expectation of supporting multiple platforms. This causes a lot of dysfunction because the lowest common denominator is really low, especially if you treat the browser as a "platform". In the 90s Microsoft had a monopoly. That had its own problems, but, it meant you could write a Windows app, once, and everyone would accept it. Nowadays you want web+mobile, and "web" is not really a platform in the sense Windows was. If you're doing anything serious you still need a desktop app.

4. Over-specialization/staffing. Back then everyone was "full stack". The developer was also a DBA, at least to some extent, and you could just throw up a quick GUI Windows app that connected directly to the database. The DBA would then manage security and backup. It wasn't really a full time job, at least not on a per app basis. Nowadays even simple projects feel absurdly over-staffed. Do you really need a backend guy, a frontend guy and a devops guy for an ordinary LOB app? Probably not.

5. Process overkill. Waterfall is underrated. It got a bad rap because it requires you to understand your customer, and for your customer to understand what they actually want, and for them to not change their mind every five minutes. Not always possible. Nonetheless, agile has become some kind of monster. Half the words in the average agile methodology are made up and a lot of it is really questionable. If your business domain weren't totally unstable and your users weren't totally incompetent (often the case!), then you could sit down and write an actual spec, which people would read and sign off on, and then you could build it. And it'd work, and after the initial debugging / shakeout period, people would be happy. Many of those apps never broke! Just imagine the output of the typical agile web AWS-based web stack teams today lasting 20 or 30 years without a dozen rewrites along the way. Very hard to imagine that.

I believe simpler solutions will occur through the ballyhooed "decentralization". What actually decentralizing tech entails is a rethink of the types of business that are in play and therefore the technologies needed for them.

Right now the assumptions are heavily entangled with the platforms: we know that our audiences are on "X". Therefore "X" becomes part of our strategy, and our end goal(from an evolutionary success standpoint) is to make them standardize on our "Y". This happens from all parties: devs, consumers, forms, governments.

Thus when I boot up Windows I'm confronted with a cacaphony of different updaters, notifications, etc. All of them trying to exploit the platform I'm currently using to pull me deeper into some other platform.

If we fast forward this process, we can see that the nature of computing ecosystems is to be a jungle that exceeds understanding. And in jungles, there are a multitude of niches. Compatibility is situational. Although one could point to apex predators, they don't exactly "rule" the jungle.

Which means that the appropriate goal to achieve in a software's lifecycle is most likely a sustainable niche that only needs to know about a few things. But our industry is not doing this yet. Why? Because software has not eaten the software industry yet.

That is the culmination of all these decades of churning on code: eventually we end up with software that is better at coordinating information and activities for society than any human-mediated organization could be. And you look at the technologies we have, and assume there's a logistical function to them, and it's like: OK, maybe AI can do that. Maybe blockchain can do that. Maybe cloud and no-code frameworks can do that. Maybe if you bodge those things together, you end up in a place where the professional developer isn't dealing with as many details, like photography vs painting. And if that's really the case then you don't have to write nearly as much of an app: it will start hooking into the ecosystem readily, instantly presenting the views on information that you need and filtering noise for you.

We haven't had a really fundamental realignment of the economy since the end of World War II. And if you look at movies from then, the economy that emerges is sensible within its concepts of how economies should move forward: information was still expensive and while many novel things could be mass produced, you needed firm structures to coordinate them(Newspapers multiple times a day! Icebox and milk deliveries! Mail-order houses!), and you needed a new set of infrastructure to animate this action. Highways, supermarkets, shipping containers, and TV were all representative of where the world was going: an ecosystem of "products and services". And you could learn what products and services a city had by walking through the phone book and making calls.

But over the past few decades, it's saturated into an "attention economy". There are so many goods available that you'll never know about all of them, so the information systems have to take up the task of digesting it and leading us towards our best lives. So the task of making software simpler is also a task of making economic coordination simpler. And we are still going to be using the products and services framing for some time, but it's likely to get weird.