Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

451–460 of 516 posts

Re: Don't rent the cloud, own instead

#451

Post by conductr »

IG_Semmelweiss wrote: post #318
scalemaxx wrote: post #304

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

There's a very interesting insight from your message. The Cloud providers made a lot of sense to finance departments since aside from the promised savings, you would take that cloud expense now and lower your tax rate. After the passing of the One Beautiful Bill ("OBB"), the law allows you to accelerate CapEx to instead expense it[1], similar to the benefit given by cloud service providers. This puts way more wind on…

CFO here and I capex everything I can, never understood why you'd want to opex this. I'm trying to make EBITDA as enticing as possible for investors and anyone else that cares. Also want to show we have control over technology cost and it grows at a step function instead of a linear. Capex spending is usually large and planned, so we monitor it more closely and need to see a good reason to approve a large new purchase. Giving AWS a credit card is giving devs a blank check.

Re: Don't rent the cloud, own instead

#452

Post by alecco »

Counterpoint: "Why I'm Selling All My GPUs" https://www.youtube.com/watch?v=C6mu2QRVNSE

TL;DW: GPU rental arbitrage is dead. Regulation hell. GPU prices. Rental price erosion. Building costs rising. Complexity of things like backup power. Delays of connection to energy grid. Staffing costs.

Re: Don't rent the cloud, own instead

#453

Post by dh2022 »

asdfman123 wrote: post #444
dh2022 wrote: post #440

LOL’ed IRL at “ In a future blog post I hope I can tell you about how we produce our own power and you should too.” Producing own power as a pre-requisite for running on-prem is a non-starter for many.

"It's really not hard to create your own coal power. Our engineers have built a small coal power generator and simply get coal from our mines (which I'll describe in a future blog post)."

LOL'ed again IRL :).

Re: Don't rent the cloud, own instead

#454

Post by SomaticPirate »

This makes sense for HPC and ML workloads. Big batch jobs where you are pushing the hardware and having everything local is a clear advantage. Also this company sells hardware so it makes sense for them to have hardware experience. Still think that for the majority on here, needing to make a physical phone call to their data center team (!!) to rack a server is a nutty proposition. You think the AWS api is slow? Trying calling Steve. If you have fixed compute costs after a year, sure, look at pulling some stuff on prem.

Re: Don't rent the cloud, own instead

#455

Post by iso1631 »

mrweasel wrote: post #173
Lucasoato wrote: post #95

Earlier quoted context omitted.

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.

> I’m a bit scared of managing the services on my own I see it from the other direction, when if something fails, I have complete access to everything, meaning that I have a chance of fixing it. That's down to hardware even. Things get abstracted away, hidden behind APIs and data lives beyond my reach, when I run stuff in the cloud. Security and regular mistakes are much the same in the cloud, but I then have to laye…

Do you want the power to fix or do you want the paper to wave so you aren't held accountable.

The main benefit of outsourcing to aws etc is that the CEO isn't yelling at you when it breaks, because their golf buddies are in the same situation.

Re: Don't rent the cloud, own instead

#456

Post by themafia »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

> while largely maximising operational costs

The core services are cheap. S3 is cheap. Dynamo is cheap. Lambda is exceedingly cheap. Not understanding these services on their own terms and failing to read the documentation can lead one to use them in highly inefficient ways.

The "cloud" isn't just "another type of server." It's another type of /service/. Every costly stack I've seen fails to accept this truth.

Re: Don't rent the cloud, own instead

#457

Post by themafia »

bojangleslover wrote: post #179
torginus wrote: post #151

Earlier quoted context omitted.

I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're us…

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

> 7) Paired with some "certified AWS partner"

What do you think RedHat support contracts are? This situation exists in every technology stack in existence.

Re: Don't rent the cloud, own instead

#458

Post by slashdave »

Torq_boi wrote: post #397
tgtweak wrote: post #385

>San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48” intake fans and dual 48” exhaust fans to keep the air cool. To ensure low humidity ( Oh man, this is bad advice. Airborn humidity and contaminants will KILL your servers on a very short horizon in most places - even San Diego. I highly s…

Yes, it's easy to destroy the servers with a lot of dust and/or high humidity. But with filtering and ensuring humidity never exceeds 45% we've had pretty good results.

You can also kill the servers with too low humidity.

Re: Don't rent the cloud, own instead

#459

Post by ragall »

bluGill wrote: post #281
4nonverbal-alit wrote: post #273

Earlier quoted context omitted.

From what I've seen, if you're depending on AWS, if something fails you too need someone 24x7 so that you can take action as well. Sometimes magic happens and systems recover after aws restarts their DNS, but usually the combination of event causes the application to get into an unrecoverable state that you need manual action. It doesn't always happen but you need someone to be there if it ever happens. Or bare minim…

How many problems is AWS able to handle for you that you are never aware of though?

Distributed systems can partly fail in many subtly different ways, and you almost never notice it because there are people on-call taking care of them.

Re: Don't rent the cloud, own instead

#460

Post by ragall »

mrweasel wrote: post #171
torginus wrote: post #151

Earlier quoted context omitted.

I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're us…

Just this week a friend of mine was spinning up some AWS managed service, complaining about the complexity, and how any reconfiguration took 45 minutes to reload. It's a service you can just install with apt, the default configuration is fine. Not only is many service no longer cheaper in the cloud, the management overhead also exceed that of on-prem.

Cloud was never cheaper. It was as convenient.
Post reply on HN