Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

cep.dev

Post reply on HN
Print view View original

81–90 of 247 posts

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#81

Post by nothrabannosir »

gouggoug wrote: post #29
nine_k wrote: post #21

Earlier quoted context omitted.

Any opinion on Pulumi?

Not an opinion on Pulumi specifically, but an opinion on using imperative programming languages for infrastructure configuration: don't do it. (This includes using things like CDKTF) Infrastructure needs to be consistent, intuitive and reproducible. Imperative languages are too unconstrained. Particularly, they allow you to write code whose output is unpredictable (for example, it'd be easy to write code that creates…

Couldn't disagree more.

I have written both TF and then CDKTF extensively (!), and I am absolutely never going back to raw TF. TF vs CDKTF isn't declarative vs imperative, it's "anemic untyped slow feedback mess" vs "strong typesystem, expressive builtins and LSP". You can build things in CDKTF that are humanly intractable in raw TF and it requires far less discipline, not more, to keep it from becoming an unmaintainable mess. Having a typechecker for your providers is a "cannot unsee" experience. As is being able to use for loops and defining functions.

That being said, would I have preferred a CDKTF in Haskell, or a typed Nix dialect? Hell yes. CDKTF was awful, it was just the least bad thing around. Just like TF itself, in a way.

But I have little problems with HCL as a compilation target. Rich ecosystem and the abstractions seem sensible. Maybe that's Stockholm syndrome? Ironically, CDKTF has made me stop hating TF :)

Now that Hashicorp put the kibosh on CDKTF though, the question is: where next...

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#83

Post by ses1984 »

UltraSane wrote: post #75
dangus wrote: post #68

Earlier quoted context omitted.

Whether or not your VPC can have subnets in multiple regions is entirely unrelated to security.

I meant failure blast radius. Having isolated regions is a core part of the AWS reliability design. AWS has had entire regions fail but these failure have always been isolated to a single region. Global VPCs must rely on globally connected routers that can all fail in ways AWS regional VPCs can't.

If you need global HA to the extent that you're worried about global VPC failure modes, you're going to have to spend a lot of effort to squeeze uptime to the max regardless of where you deploy.

Undersea cable failures are probably more likely than a google core networking failure.

In AWS a lot of "global" things are actually just hosted in us-east-1.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#84

Post by SkiFire13 »

Tostino wrote: post #66
x3n0ph3n3 wrote: post #39

Earlier quoted context omitted.

Yeah, but ansible is one of the nine circles of hell and its support for various AWS services beyond EC2 and S3 is near nonexistant.

I have mixed feelings about it. On my first startup, I used ansible to automate all of the manual workflows and server setup that we had done. Everything was just completely manual and in people's heads before, and translating it to ansible was a pain in the ass to say the least. I don't think it would have been any easier to translate it to something else though. It ended up working fine and we had a solid system th…

> on a set of VPS provided by some terraform scripts

You ended up needing Terraform too for the infrastructure though. At that point why not just use Terraform?

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#85

Post by nottorp »

jasonpeacock wrote: post #14
unethical_ban wrote: post #9

Earlier quoted context omitted.

What do you use if not slack? OPs advice is standard best practice. Respect peoples time by not expecting immediate response, and use team or function based channels as much as possible. Other options are email of course, and what, teams for instant messages?

The bullet points for using Slack basically describe email (and distribution lists). It’s funny how we get an instant messaging platform and derive best practices that try to emulate a previous technology. Btw, email is pretty instant.

If you work in a team, email is limited to the people you cc: while a convo in a slack channel can have people you didn't think of jump in* with information.

See the other point in the article about discouraging one on one private messages and encouraging public discussion. That is the main reason.

* half a day later or days later if you do true async, but that's fine.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#86

Post by dijit »

nottorp wrote: post #85
jasonpeacock wrote: post #14

Earlier quoted context omitted.

The bullet points for using Slack basically describe email (and distribution lists). It’s funny how we get an instant messaging platform and derive best practices that try to emulate a previous technology. Btw, email is pretty instant.

If you work in a team, email is limited to the people you cc: while a convo in a slack channel can have people you didn't think of jump in* with information. See the other point in the article about discouraging one on one private messages and encouraging public discussion. That is the main reason. * half a day later or days later if you do true async, but that's fine.

I am neutral in this particular topic, so don’t think I’m defending or attacking or anything.

But aren’t mailling lists and distribution groups pretty ubiquitous?

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#87

Post by [deleted] »

kolja005 wrote: post #37

>Since the database is used by everyone, it becomes cared for by no one. Startups don’t have the luxury of a DBA, and everything owned by no one is owned by infrastructure eventually. This post was a great read. Tangent to this, I've always found "best practices" to be a bit of a misnomer. In most cases in software and especially devops I have found it means "pay for this product that constrains the way that you do t…

[deleted]

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#88

Post by gib444 »

Infra guys doing DBA is a nightmare in my experience (usually clueless and it gets loved less than more sexy parts of infra). Devs too

Hire a DBA ASAP. They need to reign in also the laziness of all other developers when designing and interacting with the DB. The horrors a dev can create in the DB can take years to undo

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#89

Post by sleepychu »

SkiFire13 wrote: post #79
dangus wrote: post #26

Earlier quoted context omitted.

As a counterpoint, I find our AWS super team to be a mix of 40% helpful, 40% “things we say are going over their head,” 20% attempting to upsell and expand our dependence. It’s nice that we have humans but I don’t think it’s a reason to choose it or not. GCP’s architecture seems clearly better to me especially if you are looking to be global. Every organization I’ve ever witnessed eventually ends up with some kind of…

> Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare. What are these struggles? The product I work on uses AWS and we have ~5 accounts (I hear they used to be more TBF) but nowadays all the infrastructure is on one of them and the other are for some niche stuff (tech support?). I could see how going overboard with many accounts co…

We were saved by the bell when they announced the increased account limit for S3 buckets (1M buckets, now, 1k I think before).

Just before they announced that I was working on creating org accounts specifically to contain S3 buckets and then permitting the primary app to use those accounts just for their bucket allocation.

AWS themselves recommend an account per developer, IIRC.

It's as you say, some policy or limitation might require lots of accounts and lots of accounts can be pretty challenging to manage.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#90

Post by velocity3230 »

jrjeksjd8d wrote: post #4

I see you regret Datadog but there's no alternative - did you end up homebrewing metrics, or are you just living with their insane pricing model? In my experience they suck but not enough to leave.

LGTM stack?
Post reply on HN