> everything owned by no one is owned by infrastructure eventually
Whoa, now there is a truth bomb. I've seen this happen a bunch, but never put it this succinctly before.
Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
151–160 of 247 posts
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#152Earlier quoted context omitted.
As a counterpoint, I find our AWS super team to be a mix of 40% helpful, 40% “things we say are going over their head,” 20% attempting to upsell and expand our dependence. It’s nice that we have humans but I don’t think it’s a reason to choose it or not. GCP’s architecture seems clearly better to me especially if you are looking to be global. Every organization I’ve ever witnessed eventually ends up with some kind of…
> Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare. What are these struggles? The product I work on uses AWS and we have ~5 accounts (I hear they used to be more TBF) but nowadays all the infrastructure is on one of them and the other are for some niche stuff (tech support?). I could see how going overboard with many accounts co…
5 accounts would be heaven if that could be my environment.
I have almost 40 AWS accounts on my login portal.
Two accounts per product, one for development environments and one for production environments, every new company acquisition has their own accounts, then we have accounts that solely exist to help traverse accounts or host other ops stuff.
Maybe you don’t see issues with everything in one account but my company would.
I don’t really think they’re following current best practices but that’s a political issue that I have no control over, and I think if you went back enough years you’d find that we followed AWS’ advice at the time.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#153Earlier quoted context omitted.
Whether or not your VPC can have subnets in multiple regions is entirely unrelated to security.
I meant failure blast radius. Having isolated regions is a core part of the AWS reliability design. AWS has had entire regions fail but these failure have always been isolated to a single region. Global VPCs must rely on globally connected routers that can all fail in ways AWS regional VPCs can't.
I believe you’re likely misunderstanding Google’s architecture.
The routing isn’t centralized, it’s distributed. The VPCs are a logical abstraction, not a centralized dependency.
If you have a region/AZ going down in your global VPC, the other ones are still available.
I think it’s also not that much of an advantage for AWS to be able to say its outages are confined to a region. That doesn’t help you very much if their architecture makes architecting global services more difficult in the first place. You’re just playing region roulette hoping that your region isn’t affected. Outages frequently impact all/multiple AZs.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#154Earlier quoted context omitted.
> Particularly, they allow you to write code whose output is unpredictable Is that an easy mistake to make and a hard one to recover from, in your experience? The way you have to bend over backwards in Terraform just to instantiate a thing multiple times based on some data really annoys me..
> Is that an easy mistake to make and a hard one to recover from, in your experience? If you're alone in a codebase? Probably not. In a company with many contributors of varying degrees of competence (from your new grad to your incompetent senior staff), yes. In large repositories, without extremely diligent reviewers, it's impossible to prevent developers from creating the most convoluted anti-patterny spaghetti cod…
> but their constrained nature makes it easier to reign in the zealous junior developer who just discovered OOP and insists on trying it everywhere...
I don't think this is true anymore. Junior devs of today seem to be black pilled on OOP.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#155> Multiple applications sharing a database [0]
> Regret
Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side.
Can anyone else share their experience with this decision?
[0] https://cep.dev/posts/every-infrastructure-decision-i-endors...
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#156Earlier quoted context omitted.
I meant failure blast radius. Having isolated regions is a core part of the AWS reliability design. AWS has had entire regions fail but these failure have always been isolated to a single region. Global VPCs must rely on globally connected routers that can all fail in ways AWS regional VPCs can't.
I believe you’re likely misunderstanding Google’s architecture. The routing isn’t centralized, it’s distributed. The VPCs are a logical abstraction, not a centralized dependency. If you have a region/AZ going down in your global VPC, the other ones are still available. I think it’s also not that much of an advantage for AWS to be able to say its outages are confined to a region. That doesn’t help you very much if the…
Regions and Availability Zones are very different things. Regions are much more isolated.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#157Earlier quoted context omitted.
Any opinion on Pulumi?
Not an opinion on Pulumi specifically, but an opinion on using imperative programming languages for infrastructure configuration: don't do it. (This includes using things like CDKTF) Infrastructure needs to be consistent, intuitive and reproducible. Imperative languages are too unconstrained. Particularly, they allow you to write code whose output is unpredictable (for example, it'd be easy to write code that creates…
The imperative trite just comes off as geriatric. There are better arguments you can use here which you have shared below. One of which I agree with.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#158Pagerduty: They haven't yet hit that point where PD doubles the prices for them. Or they don't have everyone on the platform, it will be their next Datadog (too expensive)
PagerDuty's pricing trajectory is following the exact same playbook as Datadog. Start cheap enough that teams adopt it without finance approval, then jack up per-seat pricing once it's embedded in every runbook and escalation policy.
The insidious part with on-call tooling specifically is that switching costs are higher than almost any other category. Your escalation chains, schedules, integrations with monitoring, incident templates, post-mortem workflows - it all becomes organizational muscle memory. Migrating monitoring backends is a weekend project compared to migrating on-call routing.
What I've seen work: teams that treat on-call routing as a thin layer rather than a platform. If your schedules live in something portable (even a YAML file synced to whatever tool) and your alert routing is OpenTelemetry-native, swapping the actual dispatch tool becomes manageable. The teams that get locked in are the ones who build their entire incident process inside PD's UI.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#159> Picking Terraform over Cloudformation: Endorse I, too, prefer McDonald's cheeseburgers to ground glass mixed with rusty nails. It's not so much that I love Terraform (spelled OpenTofu) as that it's far and away the least bad tool I've used in the space.
CDK is far better than Terraform.
I don't see it that way. CDKs, Terraform, Pulumi all have their use cases and trade offs. I use a mix of them.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#160> Multiple applications sharing a database [0] > Regret Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side. Can anyone else share their experience with this decision? [0] https://cep.dev/posts/every-infrastructure-decision-i-endors...
Accept that eventually you'll have multiple databases, so it makes sense to plan from that from the start and get in place the mechanisms for the databases to talk to each other.