Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

cep.dev

Post reply on HN
Print view View original

241–247 of 247 posts

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#241

Post by mettamage »

AIorNot wrote: post #121
mettamage wrote: post #105

As a non infra guy I'll say this. I'm curious about Linear. At my own company I vibecoded my own project management app against the JIRA API because I can't stand our version of JIRA. It's too many clicks, too many things to remember and it's unintuitive.

As everyone knows JIRA sucks but some perfect implementation of it exists in the ether at some company you will never work at :) Theses days AI in doc, spec and production lifecycle means we need AI first ticket tooling - haven’t used Linear but I suspect that works far better with AI then JIRA

I actually know people that like JIRA believe it or not, both software engineers

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#242

Post by __turbobrew__ »

gouggoug wrote: post #29
nine_k wrote: post #21

Earlier quoted context omitted.

Any opinion on Pulumi?

Not an opinion on Pulumi specifically, but an opinion on using imperative programming languages for infrastructure configuration: don't do it. (This includes using things like CDKTF) Infrastructure needs to be consistent, intuitive and reproducible. Imperative languages are too unconstrained. Particularly, they allow you to write code whose output is unpredictable (for example, it'd be easy to write code that creates…

> for example, it'd be easy to write code that creates a resources based on the current time of day

Some languages make that impossible to do, for example starlark.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#243

Post by __turbobrew__ »

austinsharp wrote: post #195
consumer451 wrote: post #155

> Multiple applications sharing a database [0] > Regret Thanks for this data point. I am currently trying to make this call, and I was still on the fence. This has tipped me to the separate db side. Can anyone else share their experience with this decision? [0] https://cep.dev/posts/every-infrastructure-decision-i-endors...

Highly recommend reading Designing Data-Intensive Apps [1] and Monolith to Microservices [2]. I can't remember which (maybe both?) but I definitely took away the idea that if services share a DB, that DB's schema is now a public interface and becomes much more difficult to evolve with new requirements. [1] https://www.amazon.com/Designing-Data-Intensive-Applications... [2] https://www.amazon.com/Monolith-Microservice…

The way I think of it is that a DB should only be accessed by a single replicaset in k8s. Only processes of identical code should share the DB. Everything else is through RPC interfaces.

This is how large scale systems are built, but the pattern makes less sense the smaller your footprint is.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#244

Post by sleepychu »

tech2 wrote: post #237
sleepychu wrote: post #89

Earlier quoted context omitted.

We were saved by the bell when they announced the increased account limit for S3 buckets (1M buckets, now, 1k I think before). Just before they announced that I was working on creating org accounts specifically to contain S3 buckets and then permitting the primary app to use those accounts just for their bucket allocation. AWS themselves recommend an account per developer, IIRC. It's as you say, some policy or limita…

I have to ask, because wow that's a lot of buckets, but what kind of activity requires breaching even a 1,000 bucket limit per account?

Simplistic tenant isolation and cost tracking :-)

I know there are other solutions to this particular problem but this model is extremely easy to reason about. When the application accesses tenant objects or delegates that access with pre-signed URLs it is doing so with ephemeral credentials that literally could not access the objects in another tenancy.

That and a similar DB isolation, allows most of our handlers to be very simple as far as tenant isolation goes.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#245

Post by secondcoming »

stackskipton wrote: post #196
secondcoming wrote: post #137

Earlier quoted context omitted.

We've used ECS back when we were on AWS, and now GCE. We didn't have to invent any homegrown orchestration tool. Our infra is hundreds of VMs across 4 regions. Can you give an example of what you needed to do?

Really? What deploys your code now? I'm SRE, walk me through high level. How do I roll back?

It used be Google Deployment Manager but that's dead soon so terraform.

To roll back you tell GCE to use the previous image. It does all the rolling over for you.

Our deployment process looks like this:

- Jenkins: build the code to debian packages hosted on JFrog

- Jenkins: build a machine image with ansible and packer

- Jenkins: deploy the new image either to test or prod.

Test deployments create a new Instance Group that isn't automatically attached to any load balancer. You do that manually once you've confirmed everything has started ok.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#246

Post by nothrabannosir »

Too wrote: post #236
nothrabannosir wrote: post #81

Earlier quoted context omitted.

Couldn't disagree more. I have written both TF and then CDKTF extensively (!), and I am absolutely never going back to raw TF. TF vs CDKTF isn't declarative vs imperative, it's "anemic untyped slow feedback mess" vs "strong typesystem, expressive builtins and LSP". You can build things in CDKTF that are humanly intractable in raw TF and it requires far less discipline, not more, to keep it from becoming an unmaintain…

This does not match my experience. Terraform validate will check for types and you have great IDE support with refactor, find usages, resource documentation and everything else you might expect. You can of course build a monster of untyped dictionaries but that’s true for almost any language. Do you have examples of something else that the validation step does not find?

I never had this integrated with an editor but maybe I didn’t try hard enough. With CDKTF, a stock typescript IDE automatically has tab complete for every provider you want. That + being able to use functions, variables, loops, arrays, and structs for composition made it a completely different experience.

If those things all end up coming to terraform I’m all ears. For now it seemed more like a compilation target.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#247

Post by nijave »

themafia wrote: post #198
nijave wrote: post #130

Earlier quoted context omitted.

I'm curious how that plays out when you factor in other infrastructure components like DB and load balancers. On Lambda, load balancing is handled out of the box but you may need to introduce things like connection poolers for the DB you could have gotten away without on EC2 Think it also depends if you're CPU or memory constrained. Lambda seemed more expensive for CPU heavy workloads since you're stuck with certain…

I wouldn't know. I use DynamoDB almost exclusively. These simply aren't considerations that occur with that product. It is true that it can be hard to size workloads into lambdas rather unusual CPU configuration; however, the real beauty of lambda is, you can just fork several parallel copies of your function. We can sometimes fork up to 250 instances just for a single "job." If you're in the same boat we are where y…

We're on k8s and a lot of our Lambdas are Docker images so it's fairly trivial to throw the same thing on k8s with a pod autoscaler for instances and cluster autoscaler to add nodes.
Post reply on HN