Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

131–140 of 516 posts

Re: Don't rent the cloud, own instead

#131

Post by direwolf20 »

pageandrew wrote: post #66
olavgg wrote: post #63

Earlier quoted context omitted.

> I would rather pay a competent cloud provider than being responsible for reliability issues. Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

The point was about redundancy / geo spread / HA. It’s significantly more difficult to operate two physical sites than one. You can only be in one place at a time. If you want true reliability, you need redundant physical locations, power, networking. That’s extremely easy to achieve on cloud providers.

How much does your single site go down?

On cloud it's out of your control when an AZ goes down. When it's your server you can do things to increase reliability. Most colos have redundant power feeds and internet. On prem that's a bit harder, but you can buy a UPS.

If your head office is hit by a meteor your business is over. Don't need to prepare for that.

Re: Don't rent the cloud, own instead

#132

Post by amluto »

belorn wrote: post #115
DyslexicAtheist wrote: post #88

Earlier quoted context omitted.

this is what we did in the 90ies into mid 2000: > Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a pe…

I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.

In the Bay Area there are little datacenters that will happily colocate a rack for you and will even provide an engineer who can swap disks. The service is called “remote hands”. It may still be faster to drive over.

Re: Don't rent the cloud, own instead

#133

Post by direwolf20 »

wobfan wrote: post #61
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

To be fair, I think people are vastly over estimating the work they would have and the power they would need. Yes, if you have to massively scale up, then it'll take some work, but most of it is one-time work. You do it, and when it runs, you only have a fraction of work over the next months to maintain it. And with fraction, I mean below 5%. And keep in mind that >99% of startups who think of "yeah we need this and…

Startups don't know how much hardware they need when they release to customers. The extreme flexibility of cloud makes a lot of sense for them.

Re: Don't rent the cloud, own instead

#134

Post by amluto »

dagi3d wrote: post #130

> San Diego power cost is over 40c/kWh, ~3x the global average. It’s a ripoff, and overpriced simply due to political dysfunction. Mind anyone elaborate? Always thought this is was a direct cause of the free market. Not sure if by dysfunction the op means lack of intervention.

Did you say “free market”? There is one provider. There is a lot of regulation, mostly incompetent. It’s a mess.

Re: Don't rent the cloud, own instead

#135

Post by Symbiote »

belorn wrote: post #115
DyslexicAtheist wrote: post #88

Earlier quoted context omitted.

this is what we did in the 90ies into mid 2000: > Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a pe…

I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.

I don't think I've ever seen a non-hot-swap disk in a normal server. The oldest I dealt with had 16 HDDs per server, and only 12 were accessible from the outside, bu the 4 internal ones were still hot-swap after taking the cover off.

Even some really old (2000s-era) junk I found in a cupboard at work was all hot-swap drives.

But more realistically in this case, you tell the data centre "remote hands" person that a new HDD will arrive next-day from Dell, and it's to go in server XYZ in rack V-U at drive position T. This may well be a free service, assuming normal failure rates.

Re: Don't rent the cloud, own instead

#136

Post by direwolf20 »

srg0 wrote: post #65
AndroTux wrote: post #50

Earlier quoted context omitted.

contingency plan: Don't build your data center out of wood.

Plastic is made from the same stuff as gasoline.

Drain cleaner and hydrochloric acid makes salt water. Water is made of highly explosive hydrogen. Salt is made of toxic chlorine and explosive sodium.

Re: Don't rent the cloud, own instead

#137

Post by mattbillenstein »

PunchyHamster wrote: post #75
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

> At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. You rent a dataspace, which is OPEX not CAPEX, and you just lease the servers, which turns big CAPE…

Honestly, the way I've seen a lot of cloud done, they need _more_ people to manage that than a sensible private cloud setup.

Re: Don't rent the cloud, own instead

#139

Post by doublerabbit »

vidarh wrote: post #42
cgsmith wrote: post #6

I used to colocate a 2U server that I purchased with a local data center. It was a great learning experience for me. Im curious why a company wouldn't colocate their own hardware? Proximity isnt an issue when you can have the datacenter perform physical tasks. Bravo to the comma team regardless. It'll be a great learning experience and make each person on their team better. Ps... bx cable instead of conduit for elect…

The main reason not to colocate is if you're somewhere with high real estate costs... E.g Hetzner managed servers competes on price w/co-location for me because I'm in London.

I colocate in London, a single server / firewall comes to around £5k a year. I also colocate two other servers in some northern UK location in some industrial estate for £2k as my backups. I've never enjoyed the cloud and dedicated server's have their own caveats too.

Budget hosts such as Hetzner/OVH have been known to suddenly pull the plug for no reason.

My kit is old, second hand old (Cisco UCS 220 M5, 2xDell somethings) and last night I just discovered I can throw in two NVIDIA T4's and turn it in to a personal LLM.

I'm quite excited having my own colocated server with basic LLM abilities. My own hardware with my own data and my own cables. Just need my own IP's now.

Re: Don't rent the cloud, own instead

#140

Post by iso1631 »

Lucasoato wrote: post #95
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.

> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …)

Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.

Post reply on HN