Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

71–80 of 516 posts

Re: Don't rent the cloud, own instead

#71

Post by PunchyHamster »

pageandrew wrote: post #66
olavgg wrote: post #63

Earlier quoted context omitted.

> I would rather pay a competent cloud provider than being responsible for reliability issues. Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

The point was about redundancy / geo spread / HA. It’s significantly more difficult to operate two physical sites than one. You can only be in one place at a time. If you want true reliability, you need redundant physical locations, power, networking. That’s extremely easy to achieve on cloud providers.

You can just rent the rack space in datacenter and have that covered. It's still much cheaper than running that in cloud.

It doesn't make sense if you only have few servers, but if you are renting equivalent of multiple racks of servers from cloud and run them for most of the day, on-prem is staggeringly cheaper.

We have few racks and we do "move to cloud" calculation every few years and without fail they come up at least 3x the cost.

And before the "but you need to do more work" whining I hear from people that never did that - it's not much more than navigating forest of cloud APIs and dealing with random blackbox issues in cloud that you can't really debug, just go around it.

Re: Don't rent the cloud, own instead

#72

Post by bonesss »

simianwords wrote: post #17
d1sxeyes wrote: post #14

Earlier quoted context omitted.

It’s also opex vs capex, which is a battle opex wins most of the time.

I think it wins because opex is seen as stable recurring cost and capex is seen as the money you put in your primary differentiation for long term gains.

For mature Enterprises my understanding is that the financial math works out such that the cloud becomes smart for market validation, before moving to cheaper long term solution once revenue is stable.

Scale up, prove the market and establish operations on the credit card, and if it doesn’t work the money moves onto more promising opportunities. If the operation is profitable you transition away from the too expensive cloud to increase profitability, and use the operations incoming revenue to pay for it (freeing up more money to chase more promising opportunities).

Personally I can’t imagine anything outside of a hybrid approach, if only to maintain power dynamics with suppliers on both sides. Price increases and forced changes can be met with instant redeployments off their services/stack, creating room for more substantive negotiations. When investments come in the form of saving time and money, it’s not hard to get everyone aligned.

Re: Don't rent the cloud, own instead

#74

Post by adamcharnock »

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between:

1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based).

2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend.

3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time).

4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.

A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system.

Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team!

[0] https://lithus.eu, adam@

Re: Don't rent the cloud, own instead

#75

Post by PunchyHamster »

jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

> At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden.

You rent a dataspace, which is OPEX not CAPEX, and you just lease the servers, which turns big CAPEX into monthly OPEX bill

Running your own DC is "we have two dozen racks of servers" endeavour, but even just renting DC space and buying servers is much cheaper than getting same level of performance from the cloud.

> This flips when you start getting into the multiple FTEs per month in cost for just the hosting. At that point you probably have additional cost measured in 5-10 FTE in staffing anyway to babysit all of that. So now you can talk about trading off some hosting FTEs for modest amount of extra staffing FTEs and make net gains.

YOU NEED THOSE PEOPLE TO MANAGE CLOUD TOO. That's what always get ignore in calculations, people go "oh, but we really need like 2-3 ops people to cover datacenter and have shifts on the on-call", but you need same thing for cloud too, it is just dumped on programmers/devops guys in the team rather than having separate staff.

We have few racks and the part related to hardware is small part of total workload, most of it is same as we would (and do for few cloud customers) in cloud, writing manifests for automation.

Re: Don't rent the cloud, own instead

#76

Post by langarus »

ocdtrekkie wrote: post #25
langarus wrote: post #5

This is a great solution for a very specific type of team but I think most companies with consistent GPU workloads will still just rent dedicated servers and call it a day.

It's the opposite. The more consistent your workload the more practical and cost-effective it is to go on-prem. Cloud excels for bursty or unpredictable workloads where quickly scaling up and down can save you money.

Other benefits: easy access to reliable infrastructure and latest hardware which you can swap as you please. There are cases where it makes sense to navigate away from the big players (like dropbox going from aws to on-prem), but again you make this move when you want to optimize costs and are not worried about the trade-offs.

Re: Don't rent the cloud, own instead

#77

Post by bambax »

jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

> it doesn't make much sense for the majority of startup companies until they become late stage

Here's what TFA says about this:

> Cloud companies generally make onboarding very easy, and offboarding very difficult. If you are not vigilant you will sleepwalk into a situation of high cloud costs and no way out.

and I think they're right. Be careful how you start because you may be stuck in the initial situation for a long time.

Re: Don't rent the cloud, own instead

#78

Post by TonyStr »

Symbiote wrote: post #41
TonyStr wrote: post #19

Earlier quoted context omitted.

Azure provides their own "Total Cost of Ownership" calculator for this purpose [0]. Notably, this makes you estimate peripheral costs such as cost of having a server administrator, electricity, etc. [0] - https://azure-int.microsoft.com/en-us/pricing/tco/calculator...

I plugged in our own numbers (60 servers we own in a data centre we rent) and Microsoft thinks this costs us an order of magnitude more than it does. Their "assumption" for hardware purchase prices seems way off compared to what we buy from Dell or HP. It's interesting that the "IT labour" cost they estimate is $140k for DIY, and $120k for Azure. Their saving is 5 times more than what we spend...

Thank you, I've wanted to see someone use this in the real world. When doing Azure certifications (AZ900, AZ204, etc.), they force you to learn about this tool.

Re: Don't rent the cloud, own instead

#79

Post by speedgoose »

olavgg wrote: post #63
speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

> I would rather pay a competent cloud provider than being responsible for reliability issues. Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

At a previous job, the company had its critical IT infrastructure on their own data center. It was not in the IT industry, but the company was large and rich enough to justify two small data centers. It notably had batteries, diesel generators, 24/7 teams, and some advanced security (for valid reasons).

I agree that solving technical issues is very fun, and hosting services is usually easy, but having resilient infrastructure is costly and I simply don't like to be woken up at night to fix stuff while the company is bleeding money and customers.

Re: Don't rent the cloud, own instead

#80

Post by mgaunard »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

you're missing 5, what they are doing.

There is a world of difference between renting some cabinets in an Equinix datacenter and operating your own.

Post reply on HN