Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software Infrastructure 2.0: A Wishlist (2021)

erikbern.com

Post reply on HN
Print view View original

71–80 of 85 posts

Re: Software Infrastructure 2.0: A Wishlist (2021)

#71

Post by latchkey »

015a wrote: post #50

Here's something very specific I've been thinking about recently. I think Google Cloud Cloud Run is obscenely ahead of its time. Its a product that's adjacent to so many competitors, yet has no direct competitor, and has managed to drive a stake into that niche in a way that makes it such a valuable product. Its serverless, but not "Lambda Serverless" or "Vercel Serverless" which forces you to adopt an entirely diffe…

Cloud Run is great, but imho Cloud Functions are even better. It is just a simple http handler.

The entire deployment can be easily defined in github actions. Combine that with Cloud Tasks and a Cloud SQL Postgres instance and you have a near infinitely scalable solution.

I ran a system like this where over 30k servers across 7 different data centers all over the US, were hitting cloud function endpoints 24/7 with 30-50+ RPS and I never had a single failure or outage over multiple years. Even better, the whole thing never cost more than about $100/month.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#72

Post by samsquire »

I enjoyed this post, thank you.

I'm encouraged by the same ideas.

Sometimes you just want something that stays running and doesn't go down and can scale to zero and scale upwards, ideally with revenue.

I kind of want a special mega HTTP form endpoint which I can define a pipeline from, that can go to database and cause background jobs and goes into a mega API automatically.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#73

Post by pdimitar »

throwawaaarrgh wrote: post #22

The one thing I want, that doesn't exist, and won't for at least 10 years: immutable infrastructure. Oh, the concept exists. I can make some infrastructure mostly-immutable, myself. But the cloud doesn't give me it out of the box. What the cloud gives me are APIs. If I write software to call those APIs, predict what the allowed values are, predict the failures I might see, write about 5,000 lines of code to handle th…

Yeah, sadly true. While I am not a platform engineer I've witnessed their plight many times and I truly sympathize.

Now more than ever because I started making an effort to self-host much more than before... the amount of scripts I have to write just to achieve idempotency, nevermind immutability, is staggering, and I am already questioning my approach. Will likely start making use of ZFS or BTRFS snapshots, or I don't know, I'll just start snapshotting manually the entire filesystem on my Linux machines (like store all dir/file paths with their sizes and modification dates; it's a start and you can diff against such "snapshots").

I am just not comfortable with running commands and not having an idea what and where changed. It's insane that everyone is just accepting this! I am not okay with it, I want to see an exact breakdown on what changed and where and how.

IMO working on this and bringing it to the mainstream is loooong overdue.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#74

Post by pdimitar »

fbergen wrote: post #18

I would love to have what we were sold as a “truly” serverless (even though the name doesn’t mean no server) - CloudRun did a good job, but the autoscaling is too slow to not pay for idle - Lambda is great, but I want to run way more complex workloads than simple functions

Somebody mentioned Google Cloud Functions and I instantly bookmarked the service to check it later. Looks to be a pretty solid deal.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#75

Post by pdimitar »

fbergen wrote: post #29
phrotoma wrote: post #13

> You know how crappy software is crappy in ways that are so blatantly obvious to the user that you wonder why it was released? It has crossed my mind several times recently that I want a word to describe this exact state of affairs. Where a thing has a defect so blatant that it is evident to any user that the creator of the thing has never tried using it. Eg. an airbnb with no towels in it. What's the word for this…

Yet still people are using it? Otherwise it’s called an MVP and a promise of plugging the holes

Come on now, you were never told in your career, not once, "we use Microsoft Teams for communication here"? Ever?

Most crappy software exists because of inertia and corporate policies. If people truly had a choice stuff like MS Teams could be phased out by the end of the next quarter.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#76

Post by pdimitar »

Kon-Peki wrote: post #41
crabbone wrote: post #35

Earlier quoted context omitted.

I want to expand on this :) When I have to describe to people who don't work with me my interactions with developers (especially of the crappy code like that) from a standpoint of someone who represents the QA side of things... I describe to them my interactions with my five y.o. son: Me: How as school? Son: Goooood! Me: Did you behave? Son: Yes! Me: Did the teacher send you into timeout? Son: Yes... Me: So how come?…

Not knowing anything other than what you wrote, it sounds like your organization has leadership problems. People don't know why their job exists, they don't know what your organization is actually trying to accomplish, how any individual person fits into it, why the day-to-day things someone does helps, etc. Nothing anyone does with software will help.

> it sounds like your organization has leadership problems

That's like saying "sounds like the Sun is going to rise tomorrow again". Most companies have leadership problems, it's kind of ingrained in Homo Sapiens to fight for a cozy position and then become a gatekeeper of their own mediocrity, to the detriment of their rulers.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#77

Post by pdimitar »

crabbone wrote: post #35
phrotoma wrote: post #13

> You know how crappy software is crappy in ways that are so blatantly obvious to the user that you wonder why it was released? It has crossed my mind several times recently that I want a word to describe this exact state of affairs. Where a thing has a defect so blatant that it is evident to any user that the creator of the thing has never tried using it. Eg. an airbnb with no towels in it. What's the word for this…

I want to expand on this :) When I have to describe to people who don't work with me my interactions with developers (especially of the crappy code like that) from a standpoint of someone who represents the QA side of things... I describe to them my interactions with my five y.o. son: Me: How as school? Son: Goooood! Me: Did you behave? Son: Yes! Me: Did the teacher send you into timeout? Son: Yes... Me: So how come?…

> And, my guess is that the reason for it is that nobody really wants to work too hard.

There is certainly a lot of that but it gets even worse: you get actively punished for doing good work in many companies: you end up making other people work like asking managers around for product requirements (that are of course barely written somewhere, if at all) or reminding that sysadmin that they half-arsed the job of the deployment and now must add another k8s resource, or asking another dev why did they do X with the Y library... you want to make sure not to screw something up but you just end up annoying them.

And sadly these things get brought up on meetings. And many over-zealous managers will scold you because they don't like the boat rocked (even if they would actually welcome their initiative; but that assumes they'd have made an effort to understand the situation which is not a given).

It's no surprise that many talented people just end up checking in, doing the bare minimum, and clocking out. The equation is extremely easy to solve: "work X*3, get scolded, don't get promotions, accumulate hostility in colleagues" vs. "work X and have peace and quiet".

Re: Software Infrastructure 2.0: A Wishlist (2021)

#78

Post by pdimitar »

crabbone wrote: post #30

If I didn't know better, I'd think I'm reading one of those cheesy LinkedIn advertorials... To someone who dedicated their professional life to infrastructure all of these wishes read mostly irrelevant, with a strong proprietary advertising flavor. At every turn of a sentence I expected to find a mention of some commercial product this article was going to promote. Well, at least it doesn't seem to do that, not openl…

> Well... yeah, sure, why not... but it's not very important. Lots of other goals will overshadow this one.

For you. For me having to tinker with a repo full of YAML files just to have a Kafka topic provisioned (like it just happened to me this week) can and has killed motivation to the point of not working at all after, for a day or two.

This stuff should be blindingly obvious, to the point a trained monkey should be able to do it.

I have the feeling that many agents are working against such a goal though. Vested interests and all.

You even kinda sorta agree with me by qualifying your statement with this, right after the previous quote:

> Also, if we are talking in the context of whatever-as-a-service, there's very little incentive to work on the speed aspect as long as it not taking ages.

Maybe to me time_it_should_take == X and to you X times 3 is fine, but in the end the brain schemata is the same: have it take LongEnough™ (subjective value) and the person responsible simply checks out mentally.

If I were a CTO or an IT manager I'd be very worried about stuff like this.

> But having a worthless QA department isn't really something to wish for in Infrastructure 2.0. I don't see how this is a good goal.

This is IMO not at all related to the article, nowadays QA depts are removed either because leadership wants to save money or because iteration would grind to a crawl, and many businesses need the next feature the next Wednesday. Nothing to do with infra management I'd think.

Though don't get me wrong, QA is hugely important per se. But I wonder if proper end-to-end automated frontend testing (e.g. with Playwright) won't eventually make them truly extinct. Who knows. I don't.

> This is such an MBA b/s... Just give it a break.

I'll always despise MBA speak but the point of programmer productivity is important. I have no problem churning out features and fixing bugs but give me a slow bureaucratic process and you'll find out what it's like to pay a salary to somebody who pushes to the GitHub repo 5 times a month with diffs like +30-20.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#79

Post by throwawaaarrgh »

pdimitar wrote: post #73
throwawaaarrgh wrote: post #22

The one thing I want, that doesn't exist, and won't for at least 10 years: immutable infrastructure. Oh, the concept exists. I can make some infrastructure mostly-immutable, myself. But the cloud doesn't give me it out of the box. What the cloud gives me are APIs. If I write software to call those APIs, predict what the allowed values are, predict the failures I might see, write about 5,000 lines of code to handle th…

Yeah, sadly true. While I am not a platform engineer I've witnessed their plight many times and I truly sympathize. Now more than ever because I started making an effort to self-host much more than before... the amount of scripts I have to write just to achieve idempotency, nevermind immutability, is staggering, and I am already questioning my approach. Will likely start making use of ZFS or BTRFS snapshots, or I don…

I think it's that few people can see its potential. When I first started using immutable infra like 10 years ago, and saw how many problems it solved, my mind was blown. Until I saw the difference myself, it just looked like some trivial CS concept.

It's not apparent that problems X, Y and Z will be solved by immutability. Once it's applied everywhere, whole classes of problems just disappear. But until people see the problems disappear, they won't implement it. Catch-22.

Re: Software Infrastructure 2.0: A Wishlist (2021)

#80

Post by pdimitar »

throwawaaarrgh wrote: post #79
pdimitar wrote: post #73

Earlier quoted context omitted.

Yeah, sadly true. While I am not a platform engineer I've witnessed their plight many times and I truly sympathize. Now more than ever because I started making an effort to self-host much more than before... the amount of scripts I have to write just to achieve idempotency, nevermind immutability, is staggering, and I am already questioning my approach. Will likely start making use of ZFS or BTRFS snapshots, or I don…

I think it's that few people can see its potential. When I first started using immutable infra like 10 years ago, and saw how many problems it solved, my mind was blown. Until I saw the difference myself, it just looked like some trivial CS concept. It's not apparent that problems X, Y and Z will be solved by immutability. Once it's applied everywhere, whole classes of problems just disappear. But until people see th…

True, plus not many devs are directly exposed to the problems and thus the will to fix the problem never has a chance to materialize.

One of the best-oiled teams I was in had devs and sysadmins work together closely. If Jim made a huge Python mess out of its small throwaway project (that the CEO needed because he wanted a nice chart for an investor meeting) that required several virtual environments and a particular (older) version of something then the sysadmin had the power to call him out and question his methods. While not many programmers appreciate that, those that do make for a more positive workplace IMO.

RE: idempotency / immutability in general, I heard about Nix many times but I have been put off every time I tried it: cutesy (and rather dumb) terminology like pills and flakes and such, a Haskell dialect the world really did not need, tight binding between things (forgot which at this point, sorry), and the list kept growing until I just gave up. With all their quirkiness and edge cases my scripts still beat the pants off of Nix for my own goals. I mean, pacman/yay have a flag that says "only install this package if not already installed" so... ¯\_(ツ)_/¯

But I really do want something like Nix (and no, not Guix either). Not only for packages -- for the entire system. I want to be able to plug an USB drive and issue a command that says "show me new devices plugged in the last 5 minutes, or last time I checked".

We don't have stuff like that. Or if we do, I am blissfully unaware of it. Can't we just start writing them and push their adoption? Every sysadmin team invents magic from scratch. Surely we can and should collectively do better...

Post reply on HN