Earlier quoted context omitted.
We've used ECS back when we were on AWS, and now GCE. We didn't have to invent any homegrown orchestration tool. Our infra is hundreds of VMs across 4 regions. Can you give an example of what you needed to do?
Really? What deploys your code now? I'm SRE, walk me through high level. How do I roll back?
Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
231–240 of 247 posts
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#232It seems excessive and expensive. Is this what most startups are doing these days?
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#233It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.
googlers in their infinite wisdom have built a startup ecosystem for gcp that assigns "startups" to entry level new hires who are scrambling to figure out how to be managers of accounts while learning how to talk to humans, because googlers are generally not used to interacting with humans, just code, and that is the result of the programmatic hiring/screening process. each newly hired 20-something is also assigned 3…
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#234Earlier quoted context omitted.
I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.
We've only raised a handful of support cases with GCP the past 5 years, but we happened to raise one this week and they've put us onto a preview feature that solves the problem we were facing - I'm suddenly wondering if we should be trying our luck with support more often instead of figuring it out ourselves.
1: yes, I know the sla’s are usually a joke
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#235> Picking Terraform over Cloudformation: Endorse I, too, prefer McDonald's cheeseburgers to ground glass mixed with rusty nails. It's not so much that I love Terraform (spelled OpenTofu) as that it's far and away the least bad tool I've used in the space.
Any opinion on Pulumi?
IMO pulumi is a huge gain of productivity when you know what you're doing. Cons: * It's plagued by bugs and the pulumi-kubernetes provider is not getting enough attention from the pulumi team (they're always working on compatibility with yet another language instead of focusing on one thing) * You end up with your very specific/personal codebase instead of having a generic/standard thing
Still, no regrets, I saved so much time thanks to pulumi!
With a bigger team: * Oboarding people takes more time * You end up with code quality issues. Most "DevOps" people aren't devs, sadly.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#236Earlier quoted context omitted.
Not an opinion on Pulumi specifically, but an opinion on using imperative programming languages for infrastructure configuration: don't do it. (This includes using things like CDKTF) Infrastructure needs to be consistent, intuitive and reproducible. Imperative languages are too unconstrained. Particularly, they allow you to write code whose output is unpredictable (for example, it'd be easy to write code that creates…
Couldn't disagree more. I have written both TF and then CDKTF extensively (!), and I am absolutely never going back to raw TF. TF vs CDKTF isn't declarative vs imperative, it's "anemic untyped slow feedback mess" vs "strong typesystem, expressive builtins and LSP". You can build things in CDKTF that are humanly intractable in raw TF and it requires far less discipline, not more, to keep it from becoming an unmaintain…
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#237Earlier quoted context omitted.
> Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare. What are these struggles? The product I work on uses AWS and we have ~5 accounts (I hear they used to be more TBF) but nowadays all the infrastructure is on one of them and the other are for some niche stuff (tech support?). I could see how going overboard with many accounts co…
We were saved by the bell when they announced the increased account limit for S3 buckets (1M buckets, now, 1k I think before). Just before they announced that I was working on creating org accounts specifically to contain S3 buckets and then permitting the primary app to use those accounts just for their bucket allocation. AWS themselves recommend an account per developer, IIRC. It's as you say, some policy or limita…
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#238Earlier quoted context omitted.
Accept that eventually you'll have multiple databases, so it makes sense to plan from that from the start and get in place the mechanisms for the databases to talk to each other.
the method for databases to talk to each other is via? if we're not talking about replicas, we're talking about coordination at the app level, right?
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#239Pagerduty: They haven't yet hit that point where PD doubles the prices for them. Or they don't have everyone on the platform, it will be their next Datadog (too expensive)
PagerDuty's pricing trajectory is following the exact same playbook as Datadog. Start cheap enough that teams adopt it without finance approval, then jack up per-seat pricing once it's embedded in every runbook and escalation policy. The insidious part with on-call tooling specifically is that switching costs are higher than almost any other category. Your escalation chains, schedules, integrations with monitoring, i…
Anyone implementing PD paging into their company: I highly advise you add a template/interface on top of PD. This will make swapping out later much easier.
Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)
#240As a non infra guy I'll say this. I'm curious about Linear. At my own company I vibecoded my own project management app against the JIRA API because I can't stand our version of JIRA. It's too many clicks, too many things to remember and it's unintuitive.
If you have the power to do so, get rid of JIRA immediately. There are like 10 competitors that are all dramatically better. I would personally recommend https://www.shortcut.com which is very well designed, and also made some really sensible improvements over the time that we used it.