Live data from Hacker News

Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

cep.dev

221–230 of 247 posts

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#221

After listing dozens of infrastructure products/projects, "My general infrastructure advice is “less is better”. That made me laugh. Yes I get that they probably didn't use all of these at the same time.

> less is better

Goes on to use Kubernetes and entire GitOps stacks to run a process. I truly do wonder what difficulty there is in transferring a binary to the system and writing a system unit file and being done with it.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#223

Pagerduty: They haven't yet hit that point where PD doubles the prices for them. Or they don't have everyone on the platform, it will be their next Datadog (too expensive)

PagerDuty's pricing trajectory is following the exact same playbook as Datadog. Start cheap enough that teams adopt it without finance approval, then jack up per-seat pricing once it's embedded in every runbook and escalation policy. The insidious part with on-call tooling specifically is that switching costs are higher than almost any other category. Your escalation chains, schedules, integrations with monitoring, i…

Very clever. Our team is small enough right now for this to not be an issue, but I've ran into this issue previously and this feels like a far more practical design to avoid lockin.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#224
post #97
post #24

Earlier quoted context omitted.

Terraform/openTofu is more than OK. The fact that you can use to to configure your Cisco products as well as AWS is honestly great for us. It's also a bit like ansible: if you don't manage it carefully and try to separate as much as possible early, it starts bloating, so you have to curate early. Terragrunt is the only sane way to deploy terraform/openTofu in a professional environment though.

I never understood this. Why not use Ansible instead, especially if you already use it? Doubly so when you have Cisco config to manage. The experience is generally so much better it's not comparable, and it is much easier to infer running state.

It's not the same purpose. Ansible is useful to configure your IAC, terraform to deploy and handle the state, which is very useful when you have multiple teams working on the same infrastructure.

What you can do if you _really_ like ansible is to use it to generate the terraform files (typically from Jinja2 template). In practice, i think Terragrunt is easier to use if you already have terraform modules. But if i was back at my first "real" job, where we had between 50 and 80 ansible modules (very short ones, it was really good, i've never saw an infrastructure that complex handled that concisely and easily), and if we had to use terraform, i would use ansible to generate terraform files 100%.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#226
The SQLite-per-customer pattern mentioned in the database subthread is underrated. I've been running a FastAPI app with a single SQLite database (WAL mode + FTS5) and the operational simplicity is genuinely life-changing compared to managing Postgres.

The key insight: for read-heavy workloads on a single machine, SQLite eliminates the network hop entirely. Response times drop to sub-15ms for full-text search queries. The tradeoff is write concurrency, but if your write volume is low (mine is ~20/day), it's a non-issue.

The one thing I'd add to the article: the biggest infrastructure regret I see is premature complexity. Running Postgres + Redis + a message queue when your app gets 100 requests/day is solving problems you don't have while creating problems you do (operational overhead, debugging distributed state, config drift between environments).

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#228
post #225

> Multiple applications sharing a database This is a classic. I'd say that for every company, big or small, ends up taking the #1 spot on technical debt.

Martin Fowler called this (anti-)pattern an "integration database" and discouraged it since at least 2004[1]

[1]: https://martinfowler.com/bliki/IntegrationDatabase.html

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#229

> Not using Function as a Service(FaaS) more FaaS is almost certainly a mistake. I get the appeal from an accountant's perspective, but from a debugging and development perspective it's really fucking awful compared to using a traditional VM. Getting at logs in something like azure functions is a great example of this. I pushed really hard for FaaS until I had to support it. It's the worst kind of trap. I still get s…

What's the issue with logging? I would have expected stdout/stderr to get automatically transferred to the providers managed logging solution (e.g. cloudwatch). Though I never really understood the appeal of FaaS over something like Google-Cloud-Run.

Cloud Run is one service where GCP really shines. It is very flexible: handles services and long running background jobs, less annoying run time limitations like you find in Lambda.

Re: Infrastructure decisions I endorse or regret after 4 years at a startup (2024)

#230

Earlier quoted context omitted.

What's the issue with logging? I would have expected stdout/stderr to get automatically transferred to the providers managed logging solution (e.g. cloudwatch). Though I never really understood the appeal of FaaS over something like Google-Cloud-Run.

As a developer who spent a couple months developing a microservice using aws lambda functions: it SUCKS. There's no interactive debugging. Deploy for a minute or 5 depending on the changes, then trigger the lambda, wait another 5 minutes for all the logs to show up. Then proceed with printf/stack trace debugging. For reasons that I forgot, locally running the lambda code on my dev box was not applicable. Locally depl…

I've witnessed developers editing Lambda code live in the AWS console. It is extremely painful to watch.
Post reply on HN