Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

311–320 of 516 posts

Re: Don't rent the cloud, own instead

#311

Post by sgarland »

faust201 wrote: post #106
olavgg wrote: post #63

Earlier quoted context omitted.

> I would rather pay a competent cloud provider than being responsible for reliability issues. Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have.

> Why do so many developers and sysadmins think they're not competent for hosting services. It is a lot easier than you think, and its also fun to solve technical issues you may have. It is a different skillset. SRE is also an under-valued/paid (unless one is in FAANGO).

SRE has also lost nearly all meaning at this point, and more or less is equivalent to "I run observability" (but that's a SaaS solution too).

Re: Don't rent the cloud, own instead

#312

Post by adolph »

scalemaxx wrote: post #304

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

Yeah, I think the major improvement of cloud services was the rationalization of them into services with a cost instead of "ask that person for a whatsit" and "hopefully the associate goomba will approve."

All teams will henceforth expose their data and functionality through service interfaces

https://gist.github.com/chitchcock/1281611

Re: Don't rent the cloud, own instead

#313

Post by j45 »

siliconc0w wrote: post #310

You can also buy the hardware and hire an IT vendor to rack and help manage it as smart hands so you never need to visit the datacenter. With modern beefy hardware, even large web services only need a few racks so most orgs don't even to manage a large footprint. Sure you have to schedule your own hardware repairs or updates but it also means you don't need to wrangle with the ridiculous cost-engineering, reserved in…

That’s called managed servers or managed services.

I have run bare metal and manage services you just have to be clear on what you have coverage for when disaster strikes or be willing to proactively replace hard drives before they die.

Re: Don't rent the cloud, own instead

#314

Post by coffeebeqn »

wobfan wrote: post #61
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

To be fair, I think people are vastly over estimating the work they would have and the power they would need. Yes, if you have to massively scale up, then it'll take some work, but most of it is one-time work. You do it, and when it runs, you only have a fraction of work over the next months to maintain it. And with fraction, I mean below 5%. And keep in mind that >99% of startups who think of "yeah we need this and…

One decent server would be enough to run 99.5% of startups backends.

Re: Don't rent the cloud, own instead

#315

Post by antonvs »

bojangleslover wrote: post #179
torginus wrote: post #151

Earlier quoted context omitted.

I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're us…

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

> One of AWS's favorite situations

I'll give you an alternative scenario, which IME is more realistic.

I'm a software developer, and I've worked at several companies, big and small and in-between, with poor to abysmal IT/operations. I've introduced and/or advocated cloud at all of them.

The idea that it's "more expensive" is nonsense in these situations. Calculate the cost of the IT/operations incompetence, and the cost of the slowness of getting anything done, and cloud is cheap.

Extremely cheap.

Not only that, it can increase shipping velocity, and enable all kinds of important capabilities that the business otherwise just wouldn't have, or would struggle to implement.

Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers.

Re: Don't rent the cloud, own instead

#316

Post by re-thc »

scalemaxx wrote: post #304

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

> you shouldn't own the infrastructure (capital expense) and instead just account for the cost as operational expense (opex)

That was part of the reason.

The real reason was the internal infrastructure team in many orgs got nowhere. There was a huge queue and many teams instead had to find infinite workarounds including standing up their own. The "cloud" provided a standardized way to at least deal with this mess e.g. single source of billing.

> A 1990s VP of IT would look at this post and say, what's new?

Speed. The US lives in luxury but outside of that it often takes a LONG time to get proper servers. You don't just go online. There are many places where you have to talk to a vendor with no list price and the drama continues. Being out of capacity can mean weeks to months before you get anywhere.

Re: Don't rent the cloud, own instead

#317

Post by sgarland »

jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

> But until then it's a long term cost optimization with really high upfront capital expenditure and risk.

The upfront capex does not need to be that high, unless you're running your own AI models. Other than leasing new ones, as a sibling comment stated, you can buy used. You can get a solid Dell 2U with a full service contract (3 years) for ~$5-10K depending on CPU / memory / storage configuration. Or if you don't mind going older - because honestly, most webapps aren't doing anything compute-heavy - you can drop that to < $1K/node. Replacement parts for those are cheap, so buy an extra of everything.

Re: Don't rent the cloud, own instead

#318

Post by IG_Semmelweiss »

scalemaxx wrote: post #304

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

There's a very interesting insight from your message.

The Cloud providers made a lot of sense to finance departments since aside from the promised savings, you would take that cloud expense now and lower your tax rate.

After the passing of the One Beautiful Bill ("OBB"), the law allows you to accelerate CapEx to instead expense it[1], similar to the benefit given by cloud service providers.

This puts way more wind on the sails of the on-prem movement, for sure

[1] https://www.iqxbusiness.com/big-beautiful-bill-impact-on-cap...

Re: Don't rent the cloud, own instead

#319

Post by coffeebeqn »

JKCalhoun wrote: post #248

Naive comment from a hobbyist with nothing close to $5M: I'm curious about the degree to which you build a "home lab" equivalent. I mean if "scaling" turned out to be just adding another Raspberry Pi to the rack (where is Mr. Geerling when you need him?) I could grow my mini-cloud month by month as spending money allowed. (And it would be fun too.)

You sure can. Pi are pretty underpowered you can get machines with more cores and memory and pcie lanes and networking out there and virtualize them

Re: Don't rent the cloud, own instead

#320

Post by treesknees »

sgarland wrote: post #301

Note that they're running R630/R730s for storage. Those are 12-year old servers, and yet they say each one can do 20 Gbps (2.5 GBps) of random reads. In comparison, the same generation of hardware at AWS ({c,m,r}4) instance maxes out at 50% of that for EBS throughput on m4, and 70% on r4 - and that assumes carefully tuned block sizes. Old hardware is _plenty_ powerful for a lot of tasks today.

I’m on a project at work replacing our R430s and R730s. They’ve been absolute tanks with very few hardware failures. That said, my company chooses to have OEM support for replacing failed components and keeping firmware/bios/idrac updated. You can absolutely run these if you’re OK with 3rd party replacements or parting out spare machines. Some industries are more tolerant to this than others.
Post reply on HN