Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

511–516 of 516 posts

Re: Don't rent the cloud, own instead

#511

Post by acdha »

Symbiote wrote: post #507
acdha wrote: post #368

Earlier quoted context omitted.

> Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers This has been my experience as well. There are legitimate points of criticism but every time I’ve seen someone try to make that argument it’s been comparing significantly different levels of service (e.g. a storage compariso…

You are welcome to criticise my DB cluster comparison: https://news.ycombinator.com/item?id=46910521

That leaves out staffing, backups, development and testing of a multi-location failover mechanism as robust as the RDS one, and a bunch of security compliance work if that’s relevant.

It’s totally possible to beat AWS and volume is the way to do it–your admin’s salary doesn’t scale be linearly with storage–but every time I’ve tried to account for all of the costs it’s been close enough that it’s made sense to put people on things which can’t be outsourced.

Re: Don't rent the cloud, own instead

#512

Post by octoberfranklin »

tgtweak wrote: post #510
octoberfranklin wrote: post #470

Earlier quoted context omitted.

Cryptocurrency mines have been doing outside-air cooling in desert climates, at ungodly scales, for a long time now. At least a decade of that "long time" involved ordinary servers stuffed with GPUs (not ASICs) -- first for Bitcoin, then for Ethereum (until ~3 years ago).

Yes I've run many here in Canada - but you still need to keep intake air very clean and they will absolutely rust out or short from dirt ingestion if filters and humidity are out of check. They're also designed with higher tolerance to temperature than servers. The amount of dead antminers I've seen from snow ingestion is honestly concerning.

Yeah sure, you need good air filters. But bag filters are cheap. Even at the small scale of a $5m datacenter the cost of good air filters is a rounding error.

Re: Don't rent the cloud, own instead

#513

Post by bpavuk »

lstodd wrote: post #435
muvlon wrote: post #101

Earlier quoted context omitted.

Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.

Why this fixation on Nix? You don't need Nix to run bare metal.

Nix makes sure that everything is exactly as you declared, and that in case of [INSERT APOCALYPTIC EVENT], you'll be able to recover much faster

Re: Don't rent the cloud, own instead

#514

Post by Symbiote »

acdha wrote: post #511
Symbiote wrote: post #507

Earlier quoted context omitted.

You are welcome to criticise my DB cluster comparison: https://news.ycombinator.com/item?id=46910521

That leaves out staffing, backups, development and testing of a multi-location failover mechanism as robust as the RDS one, and a bunch of security compliance work if that’s relevant. It’s totally possible to beat AWS and volume is the way to do it–your admin’s salary doesn’t scale be linearly with storage–but every time I’ve tried to account for all of the costs it’s been close enough that it’s made sense to put peo…

If this database is a large portion of the infrastructure required then the fixed-ish costs don't scale so well, but a smaller cloud/hosting company should be considered.

But I have over 60 servers. Using the pricing calculator for the two AWS SaaS services that closely align with our primary service (40+ of those servers), we'd face a cost of over $1.2M/year if reserved for 3 years and paid upfront — that's for the service alone, I haven't added any bandwidth costs, or getting the data into those systems, and I've picked the minimum values for storage and throughput as I don't know what these should be. (Probably not the minimum.)

Add the remaining compute (~20 decent servers), a petabyte-scale storage pool, and all the rest, and the bill would likely exceed our entire IT budget including hardware, hosting, cloud services we do use, and all the salaries.

My rough estimate is our infrastructure costs would increase 8-10 times using AWS, our staff costs wouldn't reduce, and the risk to the budget would increase with variable usage.

This is tax money being spent, so I am asked every few years to justify why we aren't using cloud. (That's why I'm putting this much effort into a HN reply, the question was asked again recently.)

I know someone working in another country on essentially the same system for that country. They went all-in on AWS and pay every 1-2 months what we spend in a year, but have a fraction of our population/data.

Re: Don't rent the cloud, own instead

#515

Post by coredog64 »

torginus wrote: post #288
coredog64 wrote: post #259

Earlier quoted context omitted.

> If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours. If ECS is faster, then you're more satisfied with AWS and less likely to migrate. You're also open to additional services that might bring up the spend (e.g. ECS Container Insights or X-Ray) Source: Former Amazon employee

We did some benchmarks and ECS was definitely quite a bit more expensive for a given capacity than just running docker on our own EC2 instances. It also bears pointing out that a lot of applications (either in-house or off-the-shelf) expect a persistent mutable config directory or sqlite database. We used EFS to solve that issue, but it was very awkward, expensive and slow, its certainly not meant for that.

Wait until you run the smallest Fargate capacities and then Container Insights costs more in CloudWatch than your infrastructure costs :)

Re: Don't rent the cloud, own instead

#516

Post by masterlee_fn »

macmac_mac wrote: post #168

Chatgpt: # don’t own the cloud, rent instead the “build your own datacenter” story is fun (and comma’s setup is undeniably cool), but for most companies it’s a seductive trap: you’ll spend your rarest resource (engineer attention) on watts, humidity, failed disks, supply chains, and “why is this rack hot,” instead of on the product. comma can justify it because their workload is huge and steady, they’re willing to ru…

[dead]
Post reply on HN