Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments? • Hacker News Board

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#61

Post by denvrede » Wed, Mar 27, 2024, 4:05 PM UTC

bjackman wrote: post #45

KaiserPro wrote: post #27

There are a lot of ways of doing things, and its really easy to make mistakes. my advice would be: Separate your build from your infra. Whilst its nice to have your cloud be spun up with CI, its really not a great use case, and means your CI has loads of power that can be abused. Gitlab with local runners is a good place to start for CI. its relatively simple and your personal runners can be shared between projects (…

> You are unlikely to change cloud providers, so choose one and stick to it. Use their managed features. I am curious about this because I see opposing views expressed by different people. I have never personally been in a position where the decision has been relevant. I work at a cloud provider an I'm told that a big slice of our revenue comes from customers who are already load-balancing across multiple clouds, so…

> I work at a cloud provider an I'm told that a big slice of our revenue comes from customers who are already load-balancing across multiple clouds, so if we degrade perf/dollar they just turn a dial to shift load to our competitors.

It's only anecdata but I'd highly question that. All companies I worked in (startup, midsize, megacorps) went with one cloud provider and stick to it. That is also not only my experience but also from friends who work in the same field. There might be a slight difference with megacorps where I saw them using multiple cloud providers but more like: Team A is using AWS, Team B is using GCP. But never: Team A is using AWS and has a copy running, ready to go, on GCP.

I tend to agree with the GP, to a degree. Chose an cloud provider and stick with it, the probability that the company changes the provider is very low (in the end all cloud provider offer the same with similar prices, no need to switch) but don't fully buy in. Like if you're on AWS, of course use RDS for DB and S3 for object storage but don't use Code Pipelines to build. So don't go "fully" proprietary.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#62

Post by datavirtue » Wed, Mar 27, 2024, 4:09 PM UTC

cebert wrote: post #22

tj1516 wrote: post #6

Earlier quoted context omitted.

Understood. But what if you have a multi-cloud setup or you want to migrate to a different cloud to efficiently use credits?

I personally think multi-cloud is over hyped and the complexity isn’t worth it for most organizations. The apps I am responsible for are one AWS region / 3 AZ and we have sufficient uptime. We could be multi-region but the added cost wasn’t worth the benefit and complexity. Multi-cloud makes zero sense.

Multicloud is just an inevitable reality--the default. The only time I have run into homogenous cloud architectures has been in large organizations and even then I was removed from the multi-cloud concerns because I was focused on a single product/service that was hosted in a single cloud infrastructure.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#63

Post by avtar » Wed, Mar 27, 2024, 4:27 PM UTC

denvrede wrote: post #55

jimangel wrote: post #16

There's a lot of noise when trying to learn the advanced features of each cloud provider's "way of doing XYZ." I think it helps to focus on the things worth protecting: secrets, credentials, code. Who has access? How do we audit / rotate? How do we secure? You can use this approach for each step along the way, how to secure secrets in your cloud? code? IaC? container deployments? CI/CD? If we assume infra / app is co…

> There are examples like https://github.com/terraform-google-modules/terraform-exampl ... of more advanced IaC architectures, but you can start as small or as complex as you want and evolve if done properly. Is there something like this for AWS?

https://github.com/terraform-aws-modules

Check out the ECS repository for more complete examples.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#64

Post by sofixa » Wed, Mar 27, 2024, 4:49 PM UTC

denvrede wrote: post #58

sph wrote: post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

> Recently I had to set up a zero-downtime system for my app, I spent a week seriously considering a move to k3s, but the entire kubernetes ecosystem of churn frustrated me so much I simply wrote a custom script based on Caddy, regular container health checks and container cloning. Easier to understand, 20 lines of code and I don't have to sell my soul to the k8s devil just yet. I'd say it depends where you're coming…

> I never get where all the "k8s is the devil" comments come from. There is nothing really complex about it. It's a well defined API with some controllers, that's it.

And Linux is just a kernel with some tools, which are all well defined, that's it!. But if you need to debug a complex interaction "that's it" and "it's well defined" isn't enough.

Kubernetes is quite complex, with a lot of interactions between different components. Upgrades are a pain because all those interactions need to be verified to be compatible with one another, and the versioned APIs, as cool as a concept they are on paper, mean that there's constantly moving targets that need constant supervision. You can't just jump a version, you need to check all your admission controllers, CNI and CSI drivers, Ingress controller, cert-manager and all other things are compatible with the new version and with each other. This is not trivial at any scale, which is why many orgs adapt the approach of just deploying a new cluster, redeploying everything to it and switching over, which is indicative of exactly how much of a pain it is.

Even Google themselves that created it admit it's complex and have 3 managed services with different levels of abstraction to make it less complicated to use and maintain.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#65

Post by sofixa » Wed, Mar 27, 2024, 5:06 PM UTC

sph wrote: post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

> That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes.

I disagree - there's cloud PaaSes like Fargate or Cloud Run, or when self-managing, a third option which is much less known but quite easier while also being more flexible - HashiCorp Nomad. Disclaimer time: I work at HashiCorp, but I've had this opinion for years before joining - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at... (it's out of date, but the principles still apply, just to a different extent, Nomad having gotten easier). All opinions are my own etc.

Not FOSS anymore, but free and source available, composable, and does a big portion of what Kubernetes does at a fraction of the complexity. I ran it in production for a few years and everything was a breeze, unlike the Kubernetes clusters I was maintaining at the same time. You get all the "basics" of HA, failover, health checks, easy ingress, advanced deployment types, basic secrets storage, etc. without having to write thousands of lines of YAML, and with simple upgrades and close to no maintenance.

> Also, I seem to be an outlier, judging from the rest of the comments, by running on dedicated servers. These days everybody is using one of the clouds and terribly afraid of managing servers. I think it's going to be hard to make DevOps better when everyone is in the "loving" Azure/AWS/GCP embrace: you're basically positioning as their competitor, as the cloud vendor itself is always trying to upsell its customers and reduce friction to as close to zero as possible.

Because running on dedicated servers means you spend time on managing them which could be more productively spent working on whatever your product is. Don't get me wrong, I love that stuff, but it's like when I started my blog - I spent weeks on CI/CD, a nice theme, a fancy static site generator, all sorts of optimisations, choosing a good hosting provider etc. etc. etc.... instead of actually doing what I was supposed to, writing content. When you manage your own servers your costs might even be higher compared to a F/C/P aaS from a cloud provider, especially with free trial year/free tier/startup credits. As long as you keep an eye on not locking yourself in, you could easily migrate most products to self-managed dedicated servers if performance/costs require it.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#66

Post by mperham » Wed, Mar 27, 2024, 5:09 PM UTC

jmstfv wrote: post #35

I run a profitable SaaS business all on my own. Keeping it very simple: I push code to Github; then Capistrano (think bash script with some bells and whistles) deploys that code to the server and restarts systemd processes, namely Puma and Sidekiq. The tech stack is fairly simple as well: Rails, SQLite, Sidekiq + Redis, and Caddy, all hosted on a single Hetzner dedicated server. The only problem is that I can't deplo…

If you can, look into the `sidekiq-iteration` gem.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#67

Post by noop_joe » Wed, Mar 27, 2024, 5:18 PM UTC

mdaniel wrote: post #59

noop_joe wrote: post #51

Full disclosure I work at a company [1] attempting to address this very problem. Without dedicated devops the challenge is allocating developer time to do the integration work required to get the various ops tools playing nicely together. In my experience that work is non-trivial and eventually leads to some form of dedicated ops. That is also part of a dynamic -- lots of tools available for solving these problems ex…

for others similarly curious, here's an example of the thing: https://github.com/noop-inc/template-java-spring-boot/blob/m... they seem to be using the excellent lima https://github.com/lima-vm/lima#readme > for booting on macOS; I run colima for its containerd and k8s support but strongly recommend both projects $(brew install lima colima)

We were using lima, but switched to directly interface with qemu. We were able to get better control for networking and lots of performance and status notification improvements.

Linked in parent is a blueprint which is how we define applications. That config can be used to stand up an application locally as well as in a cloud environment without additional configuration.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#68

Post by sph » Wed, Mar 27, 2024, 5:35 PM UTC

denvrede wrote: post #58

sph wrote: post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

> Recently I had to set up a zero-downtime system for my app, I spent a week seriously considering a move to k3s, but the entire kubernetes ecosystem of churn frustrated me so much I simply wrote a custom script based on Caddy, regular container health checks and container cloning. Easier to understand, 20 lines of code and I don't have to sell my soul to the k8s devil just yet. I'd say it depends where you're coming…

k8s is fine. It's the ecosystem around it that I dread.

I understand how it works, it makes sense, but then you're faced with Helm, kustomize, jsonnet and a lot of bullshit just to have minimal and reproducible templating around YAML. Or maybe you should use Ansible to set it up. Maybe instead try ArgoCD. Everybody and their dog is an AWS or GCP evangelists and keep trying to discourage you from running it outside of the blessed clouds. If anything breaks you're told you're stupid and that's what you get, and I should've paid someone else to manage it.

It feels like everybody is selling you something to just manage the complexity they have created. This is what keeps me away. It's insane.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#69

Post by sph » Wed, Mar 27, 2024, 5:39 PM UTC

sofixa wrote: post #65

sph wrote: post #7

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc. Recently…

> That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes. I disagree - there's cloud PaaSes like Fargate or Cloud Run, or when self-managing, a third option which is much less known but quite easier while also being more flexible - HashiCorp Nomad. Disclaimer time: I work at HashiCorp, but I've had this opinion for years before joining - https://atodorov.me/20…

This is silly but I will not use any Hashicorp product ever again, for two reasons: one is Terraform (it is a bad product with a bad DSL), the second is that I bought one service from them that advertised a clear 30-day money back guarantee, found a bug in one hour of running, a multiple-year old Github issue they had been ignoring, and to make it worse, they completely ghosted me when I went to ask for a refund. Both on Twitter and via email.

So, I don't really care to try out Nomad for my business, honestly.

Re: Ask HN: Startup Devs -What's your biggest pain while managing cloud deployments?

#70

Post by zzyzxd » Wed, Mar 27, 2024, 5:58 PM UTC

cjk2 wrote: post #11

Killer one that has shafted us is tying everything into popular deployment and CI tooling. Your product should be buildable, deployable and tested on a standalone workstation at any time. We have lost literally days fucking around with broken products which get their tentacles into that shit. I'd use AWS + ECR + ECS personally. Stay away from kubernetes until you have a fairly large deployment as it requires huge adm…

> I'd use AWS + ECR + ECS personally. Stay away from kubernetes until you have a fairly large deployment as it requires huge administrative overhead to keep it up to date and well managed and a lot of knowledge overhead.

> ...

> Also stay the hell away from anything which is not portable between cloud vendors.

But if your team happens to have Kubernetes expertise, I would say go for Kubernetes.

Anything after building container images will be somewhat cloud-specific regardless. And arguably it can be easier to hire infra/platform folks with Kubernetes skillsets later when you need to.