Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

81–90 of 516 posts

Re: Don't rent the cloud, own instead

#81

Post by jillesvangurp »

lelanthran wrote: post #52
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

Your calculation assumes that an FTE is needed to maintain a few beefy servers. Once they are up and running that employee is spending at most a few hours a month on them. Maybe even a few hours every six months. OTOH you are specifically ignoring that you'll require mostly the same time from a cloud trained person if you're all-in on AWS. I expect the marginal cost of one employee over the other is zero.

> Once they are up and running

You should also calculate the cost of getting it up and running. With Google Cloud (I don't actually use AWS), I mainly worry about building docker containers in CI and deploying them to vms and triggering rolling restarts as those get replaced with new ones. I don't worry about booting them. I don't worry about provisioning operating systems or configuration to them. Or security updates. They come up with a lot of pre-provisioned monitoring and other stuff. No effort required on my side.

And for production setups. You need people on stand by to fix the server in case of hardware issues; also outside office hours. Also, where does the hardware live? What's your process when it fails? Who drives to wherever the thing is and fixes it? What do you pay them to be available for that? What's the lead time for spare components? Do you actually keep those in supply? Where? Do you pay for security for wherever all that happens? What about cleaning, AC, or a special server room in your building. All that stuff is cost. Some of it is upfront cost. Some of it is recurring cost.

The article is a about a company that owns its own data center. The cost they are citing (5 million) is substantial and probably a bit more complete. That's one end of the spectrum.

Re: Don't rent the cloud, own instead

#82

Post by dist-epoch »

satvikpendem wrote: post #47

I just read about Railway doing something similar, sadly their prices are still high compared to other bare metal providers and even VPS such as Hetzner with Dokploy, very similar feature set yet for the same 5 dollars you get way more CPU, storage and RAM. https://blog.railway.com/p/launch-week-02-welcome

Their pricing page is so confusing: CPU: $0.00000772 per vCPU / sec

This seems to imply $40 / month for 2 vCPU which seems very high?

Or maybe they mean "used" CPU versus idle?

Re: Don't rent the cloud, own instead

#83

Post by bpavuk »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible!

plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

Re: Don't rent the cloud, own instead

#84

Post by bigfatkitten »

speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

> Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO.

Speaking as someone who does this, it is very straightforward. You can rent space from people like Equinix or Global Switch for very reasonable prices. They then take care of power, cooling, cabling plant etc.

Re: Don't rent the cloud, own instead

#85

Post by fauigerzigerk »

simianwords wrote: post #54
fauigerzigerk wrote: post #43

Earlier quoted context omitted.

I'm starting to wonder though whether companies even have the in-house competence to compare the options and price this risk correctly. >Now a good company would concentrate risk on their differentiating factor or the specific part they have competitive advantage in. Yes, but one differentiating factor is always price and you don't want to lose all your margins to some infrastructure provider.

Software companies have higher margins so these decisions are lower stakes. Unless on premises helps the bottom line of the main product that the company provides, these decisions don't really matter in my opinion. Think of a ~5000 employee startup. Two scenarios: 1. if they win the market, they capture something like ~60% margin 2. if that doesn't happen, they just lose, VC fund runs out and then they leave In this…

I'm not disputing that there are situations where it makes sense to pay a high risk premium. What I'm disputing is that price doesn't matter. I get the impression that companies are losing the capability to make rational pricing decisions.

>Unless on premises helps the bottom line of the main product that the company provides, these decisions don't really matter in my opinion.

Well, exactly. But the degree to which the price of a specific input affects your bottom line depends on your product.

During the dot com era, some VC funded startups (such as Google) made a decision to avoid using Windows servers, Oracle databases and the whole super expensive scale-up architecture that was the risk-free, professional option at the time. If they hadn't taken this risk, they might not have survived.

[Edit] But I think it's not just about cloud vs on-premises. A more important question may be how you're using the cloud. You don't have to lock yourself into a million proprietary APIs and throw petabytes of your data into an egress jail.

Re: Don't rent the cloud, own instead

#86

Post by juvoly »

> Cloud companies generally make onboarding very easy, and offboarding very difficult. If you are not vigilant you will sleepwalk into a situation of high cloud costs and no way out. If you want to control your own destiny, you must run your own compute.

Cost and lock-in are obvious factors, but "sovereignty" has also become a key factor in the sales cycle, at least in Europe.

Handing health data, Juvoly is happy to run AI work loads on premise.

Re: Don't rent the cloud, own instead

#87

Post by arjie »

Realistically, it's the speed with which you can expand and contract. The cloud gives unbounded flexibility - not on the per-request scale or whatever, but on the per-project scale. To try things out with a bunch of EC2s or GCEs is cheap. You have it for a while and then you let it go. I say this as someone with terabytes of RAM in servers, and a cabinet I have in the Bay Area.

Re: Don't rent the cloud, own instead

#88

Post by DyslexicAtheist »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

this is what we did in the 90ies into mid 2000:

> Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills

back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a person was full stack in a sense that they covered backups, networking, firewalls, and knew how to source hardware.

the argument was that this was too expensive and the cloud was better. so hundreds of thousands of SME's embraced the cloud - most of them never needed Google-type of scale, but got sucked into the "recurring revenue" grift that is SaaS.

If you opposed this mentality you were basically saying "we as a company will never scale this much" which was at best "toxic" and at worst "career-ending".

The thing is these ancient skills still exist. And most orgs simply do not need AWS type of scale. European orgs would do well to revisit these basic ideas. And Hetzner or Lithus would be a much more natural (and honest) fit for these companies.

Re: Don't rent the cloud, own instead

#89

Post by adamcharnock »

mgaunard wrote: post #80
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

you're missing 5, what they are doing. There is a world of difference between renting some cabinets in an Equinix datacenter and operating your own.

Fair point!

5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).

There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).

Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?

If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?

I think this is one of the more interesting areas to think about, at least for me!

Re: Don't rent the cloud, own instead

#90

Post by epolanski »

speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

> but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO

I worked in a company with two server farms (a main and a a backup one essentially) in Italy located in two different regions and we had a total of 5 employees taking care of them.

We didn't hear about them, we didn't know their names, but we had almost 100% uptime and terrific performance.

There was one single person out of 40 developers who's main responsibility were deploys, and that's it.

It costed my company 800k euros per year to run both the server farms (hardware, salaries, energy), and it spared the company around 7-8M in cloud costs.

Now I work for clients that spend multiple millions in cloud for a fraction of the output and traffic, and I think employ around 15+ dev ops engineers.

Post reply on HN