Live data from Hacker News

Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

cep.dev

181–190 of 247 posts

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#181

> Multiple applications sharing a database [0] > Regret Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side. Can anyone else share their experience with this decision? [0] https://cep.dev/posts/every-infrastructure-decision-i-endors...

Separate! You lose the flexibility to move logic between the application and the database when the database is its own API.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#182
post #21

> Picking Terraform over Cloudformation: Endorse I, too, prefer McDonald's cheeseburgers to ground glass mixed with rusty nails. It's not so much that I love Terraform (spelled OpenTofu) as that it's far and away the least bad tool I've used in the space.

Any opinion on Pulumi?

manage the infrastructure with infrastructure tools - manage the application with application tools. they are not the same thing. you do not need to change the oil on your cars seats...drivetrains and interiors are different worlds joining together to achieve the goal of moving humans around.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#183
post #150

Earlier quoted context omitted.

I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.

if you need to bump a quota above the predetermined range of what googlers think is "normal" usage (which is far too low to run anything at scale)you have to talk to a human to negotiate the quota bump. why? because googlers in their infinite engineering wisdom use "gcp quotas" not as a cost optimization guardrail for customers benefit, but to inform google on when and how much metal they need to buy for their datace…

I have to defend the Googlers here (I work at a different hyperscaler). Teams / services need to optimize their COGS. That means optimizing infrastructure cost. A lot of pay as you go service may not have any base cost to customers but they require some infrastructure to be provisioned. Without quotas you can have a lot of provisioned infrastructure which does not produce any revenue to even collectively break even. Just yesterday this a decision we evaluated again in my team. As a team we cannot afford an unlimited quota - both because of what that would do to our bottom line and because we can't necessarily obtain all the quotas we need ourselves to provision enough capacity for our dependencies. It's a difficult trade off requiring manual intervention.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#184
post #110

Earlier quoted context omitted.

Curious from you or others when FaaS isn’t possible? What criteria do you look for to decide or migrate off?

not possible: - workloads over 15m for lambda last time I checked, unsure on other providers - if you are looking to do anything stateful possible but not ideal/inconveniences: - cold starts can hamper latency sensitive apps (language dependant + there are things you can do) - if you have consistent traffic its not very good value for money - if you value local debugging

Those are obvious. Sorry but I was interested in more nuance given the article.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#187
post #147

Earlier quoted context omitted.

This is true. But at the same time people need to understand that most companies will never hit that certain point. It's a matter of if, not when. Everyone tries to plan for a world where they've become one of the hyperscalers. Better to optimize for the much more likely scenarios.

We were not a hyperscaler, we were boring company that you never heard of. Database is still 40TB with 3200 stored procedures.

I've dealt with postgres DBs larger than that in size though with no stored procedures and have never run into such problems. Except for a single table in a single DB at one stop, and that was a special case of people being extra stupid.

Granted, DB size isn't the best metric to be using here in terms of performance, but it's the one you used.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#188

Pagerduty: They haven't yet hit that point where PD doubles the prices for them. Or they don't have everyone on the platform, it will be their next Datadog (too expensive)

I agree, 2-3x pricing because you have more people always felt like a cash grab, similar to an SSO Tax. We also have a lot of complex Pagerduty configurations and their APIs are painful. Why do timestamps drift, templates get updated but don't show drift, and identifiers are not the same between UI and API. I regret implementing within Terraform and would rather just let teams manage their own on-call sadly.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#189

> Multiple applications sharing a database [0] > Regret Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side. Can anyone else share their experience with this decision? [0] https://cep.dev/posts/every-infrastructure-decision-i-endors...

He doesn't want to manage the database the way he manages the rest of his infrastructure. All of his bullet points apply to other components as well, but he's absorbed the cost of managing them and assigning responsibilities.

- Crud accumulates in the [infrastructure thingie], and it’s unclear if it can be deleted.

- When there are performance issues, infrastructure (without deep product knowledge) has to debug the [infrastructure thingie] and figure out who to redirect to

- [infrastructure thingie] users can push bad code that does bad things to the [infrastructure thingie]. These bad things may PagerDuty alert the infrastructure team (since they own the [infrastructure thingie]). It feels bad to wake up one team for another team’s issue. With application owned [infrastructure thingies], the application team is the first responder.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#190

I think we're making a mistake by shoving all of this into the cloud rather than building tooling around local agents (worktrees, containers, as mentioned as "difficult" in the post). I think as an industry we just reach for cloud like our predecessors reached for IBM, without critical thought about what's actually the right tool for the job. If you can manage docker containers in a cloud, you can manage them on your…

The thing is that startups often don't have the time or capital to build a data center even though public cloud is just more expensive. If you're bootstrapping a business then it makes sense. My advice would be to always use only those features of the public cloud that you can also use on your private cloud, such as Kubernetes.

I am confused. I am not saying anything about building a datacenter
Post reply on HN