Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

cep.dev

Post reply on HN
Print view View original

191–200 of 247 posts

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#191

Post by antonyt »

bob1029 wrote: post #106

> Not using Function as a Service(FaaS) more FaaS is almost certainly a mistake. I get the appeal from an accountant's perspective, but from a debugging and development perspective it's really fucking awful compared to using a traditional VM. Getting at logs in something like azure functions is a great example of this. I pushed really hard for FaaS until I had to support it. It's the worst kind of trap. I still get s…

> Getting at logs in something like azure functions is a great example of this.

This is the least of the problems I've experienced with Azure Functions. You'd have to try very hard to NOT end up with useful logs in Application Insights if you use any of the standard Functions project templates. I'm wondering how this went wrong for you?

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#192

Post by mannyv »

econner wrote: post #3

It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.

If you know what you're doing you don't need AWS support.

We add support when we want to do something new, like MediaTailor + SSAI. At that point we're exploring and trying to get our heads around how things work. Once it works there's no real point in support.

That said, you need to ask your account manager about (1) discounts in exchange for spend commitments, and (2) technical assistance. In general we have a talk with our AM when we're doing something new, and they rope in SMEs from the various products for us.

We're not that big, and I haven't worked for large companies, and it's always been a mystery to me why people have problems dealing with AWS. I've always found them to be super responsive and easy to get ahold of. OTOH we actually know what we're doing technically.

Google Cloud, OTOH, is super fucked up. I mean seriously, I doubt anyone there has any idea WTF is happening or how anything works anymore. There's no real cohesion, or at least there wasn't the last time I was abused by GCP.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#193

Post by travisd »

h1fra wrote: post #150
econner wrote: post #3

It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.

I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.

I found two separate bugs in GCP products. One with gVisor where it would sometimes null-truncate large network packets (this was very hard to diagnose – why is my JSON full of null bytes?) and one where Cloud Run broke sudo sporadically (sudo in a FaaS is definitely niche, I had essentially containerized a very old application written by undergraduates).

Both times they were serious production bugs that took at least a week to resolve, though I only had the lowest tier of support package.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#194

Post by 0ckpuppet »

h1fra wrote: post #150
econner wrote: post #3

It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.

I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.

[dead]

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#195

Post by austinsharp »

consumer451 wrote: post #155

> Multiple applications sharing a database [0] > Regret Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side. Can anyone else share their experience with this decision? [0] https://cep.dev/posts/every-infrastructure-decision-i-endors...

Highly recommend reading Designing Data-Intensive Apps [1] and Monolith to Microservices [2]. I can't remember which (maybe both?) but I definitely took away the idea that if services share a DB, that DB's schema is now a public interface and becomes much more difficult to evolve with new requirements.

[1] https://www.amazon.com/Designing-Data-Intensive-Applications... [2] https://www.amazon.com/Monolith-Microservices-Evolutionary-P...

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#196

Post by stackskipton »

secondcoming wrote: post #137
karolist wrote: post #77

Earlier quoted context omitted.

To replace Kubernetes, you inevitably have to reinvent Kubernetes. By the time you build in canaries, blue/green deployments, and rolling updates with precise availability controls, you've just built a bespoke version of k8s. I'll take the industry standard over a homegrown orchestration tool any day.

We've used ECS back when we were on AWS, and now GCE. We didn't have to invent any homegrown orchestration tool. Our infra is hundreds of VMs across 4 regions. Can you give an example of what you needed to do?

Really? What deploys your code now? I'm SRE, walk me through high level. How do I roll back?

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#197

Post by sandorscribbles »

verst wrote: post #183
sandorscribbles wrote: post #176

Earlier quoted context omitted.

if you need to bump a quota above the predetermined range of what googlers think is "normal" usage (which is far too low to run anything at scale)you have to talk to a human to negotiate the quota bump. why? because googlers in their infinite engineering wisdom use "gcp quotas" not as a cost optimization guardrail for customers benefit, but to inform google on when and how much metal they need to buy for their datace…

I have to defend the Googlers here (I work at a different hyperscaler). Teams / services need to optimize their COGS. That means optimizing infrastructure cost. A lot of pay as you go service may not have any base cost to customers but they require some infrastructure to be provisioned. Without quotas you can have a lot of provisioned infrastructure which does not produce any revenue to even collectively break even.…

i may have not emphasized enough how important quotas are for customers. quotas are very important guardrails for orgs that ensure that newly hired engineer who wants to "test drive the cloud" by running a BigQuery tutorial they found on github, gets stopped before they burn $10k in an afternoon. however, quotas on gcp are there for the benefits of google and not geared towards the customers. first there is an ever expanding tree of potential quotas complicating production rollouts of infra and second they are all set insanely low so even the smallest POC gets blocked. requesting a small increase routes the quota through software and auto-approval, requesting a quota that allows for a production workload? 3 weeks + help from your account rep, if google has blessed you the privilege of being allowed to talk with a human googler. no account rep you say, well your production workload can just wait around for google support to potentially acknowledge your existence.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#198

Post by themafia »

nijave wrote: post #130
themafia wrote: post #61

> “This EC2 instance type running 24/7 at full load is way less expensive than a Lambda running”. For the same amount of memory they should cost _nearly_ identical. Run the numbers. They're not significantly different services. Aside from this you do NOT pay for IPv4 when using Lambda, you do on EC2, and so Lambda is almost always less expensive.

I'm curious how that plays out when you factor in other infrastructure components like DB and load balancers. On Lambda, load balancing is handled out of the box but you may need to introduce things like connection poolers for the DB you could have gotten away without on EC2 Think it also depends if you're CPU or memory constrained. Lambda seemed more expensive for CPU heavy workloads since you're stuck with certain…

I wouldn't know. I use DynamoDB almost exclusively. These simply aren't considerations that occur with that product.

It is true that it can be hard to size workloads into lambdas rather unusual CPU configuration; however, the real beauty of lambda is, you can just fork several parallel copies of your function. We can sometimes fork up to 250 instances just for a single "job."

If you're in the same boat we are where your workloads parallelize easily then Lambda has been incredibly cost effective for this use case.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#199

Post by mnahkies »

h1fra wrote: post #150
econner wrote: post #3

It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.

I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.

We've only raised a handful of support cases with GCP the past 5 years, but we happened to raise one this week and they've put us onto a preview feature that solves the problem we were facing - I'm suddenly wondering if we should be trying our luck with support more often instead of figuring it out ourselves.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#200

Post by gouggoug »

juffasdfwa wrote: post #154
gouggoug wrote: post #98

Earlier quoted context omitted.

> Is that an easy mistake to make and a hard one to recover from, in your experience? If you're alone in a codebase? Probably not. In a company with many contributors of varying degrees of competence (from your new grad to your incompetent senior staff), yes. In large repositories, without extremely diligent reviewers, it's impossible to prevent developers from creating the most convoluted anti-patterny spaghetti cod…

> but their constrained nature makes it easier to reign in the zealous junior developer who just discovered OOP and insists on trying it everywhere... I don't think this is true anymore. Junior devs of today seem to be black pilled on OOP.

Let my geriatric self rephrase this for you and make the point more obvious: "[...] who just discovered [insert latest design pattern trend of your choice] and insists on trying it everywhere"
Post reply on HN