Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

101–110 of 516 posts

Re: Don't rent the cloud, own instead

#101

Post by muvlon »

bpavuk wrote: post #83
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible! plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.

Re: Don't rent the cloud, own instead

#102

Post by Neil44 »

dist-epoch wrote: post #82
satvikpendem wrote: post #47

I just read about Railway doing something similar, sadly their prices are still high compared to other bare metal providers and even VPS such as Hetzner with Dokploy, very similar feature set yet for the same 5 dollars you get way more CPU, storage and RAM. https://blog.railway.com/p/launch-week-02-welcome

Their pricing page is so confusing: CPU: $0.00000772 per vCPU / sec This seems to imply $40 / month for 2 vCPU which seems very high? Or maybe they mean "used" CPU versus idle?

Billing per used or not idle cpu cycle would be quite interesting. Number of cores would just effectively be your cost cap. Efficiency would be even more important. And if the provider over subscribes cores you just pay less. Actually that's probably why they don't do it...

Re: Don't rent the cloud, own instead

#103

Post by nicman23 »

account42 wrote: post #69
pageandrew wrote: post #66

Earlier quoted context omitted.

The point was about redundancy / geo spread / HA. It’s significantly more difficult to operate two physical sites than one. You can only be in one place at a time. If you want true reliability, you need redundant physical locations, power, networking. That’s extremely easy to achieve on cloud providers.

You don't need full "cloud" providers for that, colocation is a thing.

or just to be good at hiding the round trip of latency

Re: Don't rent the cloud, own instead

#105

Post by preisschild »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

Been using Hetzner Cloud for Kubernetes and generally like it, but it has its limitations. The network is highly unpredictable. You at best get 2Gbit/s, but at worst a few hundreds of Mbit/s.

https://docs.hetzner.com/cloud/technical-details/faq/#what-k...

Re: Don't rent the cloud, own instead

#106

Post by faust201 »

olavgg wrote: post #63
speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

> I would rather pay a competent cloud provider than being responsible for reliability issues. Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

> Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

It is a different skillset. SRE is also an under-valued/paid (unless one is in FAANGO).

Re: Don't rent the cloud, own instead

#107

Post by preisschild »

bpavuk wrote: post #83
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible! plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

I'm a NixOS fan, but been using Talos Linux on Hetzner nodes (using Cluster-API) to form a Kubernetes Cluster. Works great too!

Re: Don't rent the cloud, own instead

#108

Post by Symbiote »

jillesvangurp wrote: post #81
lelanthran wrote: post #52

Earlier quoted context omitted.

Your calculation assumes that an FTE is needed to maintain a few beefy servers. Once they are up and running that employee is spending at most a few hours a month on them. Maybe even a few hours every six months. OTOH you are specifically ignoring that you'll require mostly the same time from a cloud trained person if you're all-in on AWS. I expect the marginal cost of one employee over the other is zero.

> Once they are up and running You should also calculate the cost of getting it up and running. With Google Cloud (I don't actually use AWS), I mainly worry about building docker containers in CI and deploying them to vms and triggering rolling restarts as those get replaced with new ones. I don't worry about booting them. I don't worry about provisioning operating systems or configuration to them. Or security update…

You are massively overcomplicating this.

> I don't worry about booting them. I don't worry about provisioning operating systems or configuration to them. Or security updates. They come up with a lot of pre-provisioned monitoring and other stuff. No effort required on my side.

These are not difficult problems. You can use the same/similar cloud install images.

A 10 year old nerd can install Linux on a computer; if you're a professional developer I'm sure you can read the documentation and automate that.

> And for production setups. You need people on stand by to fix the server in case of hardware issues; also outside office hours.

You could use the same person who is on standby to fix the cloud system if that has some failure.

> Also, where does the hardware live?

In rented rackspace nearby, and/or in other locations if you need more redundancy.

> What's your process when it fails? Who drives to wherever the thing is and fixes it? What do you pay them to be available for that? What's the lead time for spare components? Do you actually keep those in supply? Where?

It will probably report the hardware failure to Dell/HP/etc automatically and open a case. Email or phone to confirm, the part will be sent overnight, and you can either install it yourself (very, very easy for things like failed disks) or ask a technician to do it (I only did this once with a CPU failure on a brand new server). Dell/HP/etc will provide the technician, or your rented datacentre space will have one for simpler tasks like disks.

Re: Don't rent the cloud, own instead

#109

Post by lelanthran »

jillesvangurp wrote: post #81
lelanthran wrote: post #52

Earlier quoted context omitted.

Your calculation assumes that an FTE is needed to maintain a few beefy servers. Once they are up and running that employee is spending at most a few hours a month on them. Maybe even a few hours every six months. OTOH you are specifically ignoring that you'll require mostly the same time from a cloud trained person if you're all-in on AWS. I expect the marginal cost of one employee over the other is zero.

> Once they are up and running You should also calculate the cost of getting it up and running. With Google Cloud (I don't actually use AWS), I mainly worry about building docker containers in CI and deploying them to vms and triggering rolling restarts as those get replaced with new ones. I don't worry about booting them. I don't worry about provisioning operating systems or configuration to them. Or security update…

> You should also calculate the cost of getting it up and running.

I was not doing the calculation. I was only pointing out that it was not as simple as you make it out to be.

Okay, a few other things that aren't in most calculations:

1. Looking at jobs postings in my area, the highest paid ones require experience with specific cloud vendors. The FTEs you need to "manage" the cloud are a great deal more expensive than developers.

2. You don't need to compare on-prem data center with AWS - you can rent a pretty beefy VPS or colocate for a fraction of the cost of AWS (or GCP, or Azure) services. You're comparing the most expensive alternative when avoiding cloud services, not the most typical.

3. Even if you do want to build your own on-prem rack, FTEs aren't generally paid extra for being on the standby rota. You aren't paying extra. Where you will pay extra is for hot failovers, or machine room maintenance, etc, which you don't actually need if your hot failover is a cheap beefy VPS-on-demand on Hetzner, DO, etc.

4. You are measuring the cost of absolute 0% downtime. I can't think of many businesses that have such high sensitivity to downtime. Even banks handle downtime much larger than that even while their IT systems are still up. With such strict requirements you're getting into the spot where the business itself cannot continue because of catastrophe, but the IT systems can :-/. What use is the IT systems when the business itself may be down?

The TLDR is:

1. If you have highly paid cloud-trained FTEs, and

2. Your only option other than Cloud is on-prem, and

3. Your FTEs are actually FT-contractors who get paid per hour, and

4. Your uptime requirements are moire stringent than national banks,

yeah, then cloud services are only slightly more expensive.

You know how many businesses fall into that specific narrow set of requirements?

Re: Don't rent the cloud, own instead

#110

Post by weavie »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

What is the upper limit of Hertzner? Say you have an AWS bill in the $100s of millions, could Hertzner realistically take on that scale?
Post reply on HN