Live data from Hacker News

Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

news.ycombinator.com

11–20 of 96 posts

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#11
Killer one that has shafted us is tying everything into popular deployment and CI tooling. Your product should be buildable, deployable and tested on a standalone workstation at any time. We have lost literally days fucking around with broken products which get their tentacles into that shit.

I'd use AWS + ECR + ECS personally. Stay away from kubernetes until you have a fairly large deployment as it requires huge administrative overhead to keep it up to date and well managed and a lot of knowledge overhead. Keep your containers entirely portable. Make sure you understand persistent storage and backups for any state.

Also stay the hell away from anything which is not portable between cloud vendors.

As for observability, carefully price up buy vs build because buy gets extremely expensive. Last thing you want is a $500k bill for logging one day.

And importantly every time you do something in any major public cloud, someone is making money out of you and it affects your bottom line. You need to develop an accounting mentality. Nothing is free. Build around that assumption. Favour simple, scalable architectures, not complex sprawling microservices.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#13
I’d argue the obvious answer is address the lack of great answers for declarative schema migration in PostgreSQL. There is Skeema https://github.com/skeema/skeema but it doesn’t support Postgres and Prisma iirc forces you into an ORM, atlas looks perfect but has a nonstandard license.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#14
A year ago I left my job at a 500-employee SaaS business working on the team that maintains the devops infrastructure, to found a startup. For me the biggest pain point is going from nothing to a sufficiently flexible devops setup.

There are a lot of great tools out there, but making them play well together is an exercise for the reader. There are also a lot of preference-based choices you need to make in how you want your setup to look, and what you chose will affect what tools make sense to you.

Do you go monorepo or polyrepo? If you go monorepo, how do you decide what to build and deploy on each merge? If you go polyrepo, how do you keep stuff in sync between any code you want to share?

Once a build is complete, how do you trigger a deployment? How does your CI system integrate with your deployment system, or is the answer "with some shell scripts you have to write"?

> How do you deploy resources?

For us, we have a monorepo setup with bazel. I wrote some fairly primitive scripts to scan git changes to decide what to build. We use Buildkite for CI, which triggers rollouts to kubernetes with ArgoCD. I had to do a non-trivial amount of work to tie all this together, but it's been fairly robust and has only needed a minimal amount of care and feeding.

> How do you define architecture?

Kubernetes charts for our services are in git, but there's some amount of extra stuff deployed (ingress controller, for example) that is documented in a text file

> How do you manage your environments

We don't need to deploy environments super often, so just do it manually and update documentation in the process if any variations are needed.

> observability

Datadog and sumologic.

Overall our setup doesn't come close to the setup I worked on at my last employer, but I have to balance time spent on devops infra with time spent on the product, and that setup took ~5 full time engineers to maintain.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#15
post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

Lambda? Digital ocean app service? Azure containers?

I guess it is proprietary tech though which might be off putting.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#16
There's a lot of noise when trying to learn the advanced features of each cloud provider's "way of doing XYZ." I think it helps to focus on the things worth protecting: secrets, credentials, code.

Who has access? How do we audit / rotate? How do we secure?

You can use this approach for each step along the way, how to secure secrets in your cloud? code? IaC? container deployments? CI/CD?

If we assume infra / app is code, the tooling matters a lot less. How do you provision certificates via IaC? How do you grant IAM to resources and how do you revoke?

There are examples like https://github.com/terraform-google-modules/terraform-exampl... of more advanced IaC architectures, but you can start as small or as complex as you want and evolve if done properly.

Personally, I love me some Kubernetes + ArgoCD (GitOps) + Google Workload Identity + Google Secret Manager, but I am 100% biased.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#17
post #11

Killer one that has shafted us is tying everything into popular deployment and CI tooling. Your product should be buildable, deployable and tested on a standalone workstation at any time. We have lost literally days fucking around with broken products which get their tentacles into that shit. I'd use AWS + ECR + ECS personally. Stay away from kubernetes until you have a fairly large deployment as it requires huge adm…

ECS is the way imo. I dislike the complexity in auth around cloud services but ECS is a really nice way to build scalable systems cheaply. I am a big fan of using the underlying Fargate spot machines within ECS.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#18
post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

There’s also Docker swarm for containerized workloads.

Personally I like VMs, and I’ve been toying around with the idea of having a LXD cluster on dedicated servers where each LXD container hosts some of my workloads.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#19
post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

There’s also Docker swarm for containerized workloads. Personally I like VMs, and I’ve been toying around with the idea of having a LXD cluster on dedicated servers where each LXD container hosts some of my workloads.

I set up Docker Swarm at my previous company, but it was a dead and stale project 5 years ago when I last used it. I honestly cannot recommend it in 2024.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#20
The basic thing when dealing with the "Cloud" is to stay away from their exclusive offering and vendor-specific stuff to avoid lock-in. Once you're locked in you're gonna get fleeced really hard as they are aware that moving away in such a situation is much harder.

In general when you have a solution that is supposed to handle any problem and scenario like with AWS you'll eventually end up with some complicated Frankenstein-y creation, there's probably no going around it if they want such a robust set of features and capabilities.

Post reply on HN