Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

471–480 of 516 posts

Re: Don't rent the cloud, own instead

#471

Earlier quoted context omitted.

I also thinks it’s risk model too. Every time I see these kind of posts I think it misses the point there is a balance not only on cost like you describe but risk as well. You are paying to offload some of the risk from yourself.

The issue is that they have already paid off their datacenter 5x over compared to cloud. For offline, batch training, I don't ses how any amount of risk could offset the savings.

Model training, perhaps. Web hosting? Not so much.

Re: Don't rent the cloud, own instead

#473

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

One of the biggest problems with the self-hosted situations I’ve seen is when the senior engineers who set it up leave and the next generation has to figure out how to run it all. In theory with perfect documentation they’d have a good head start to learn it, but there is always a lot of unwritten knowledge involved in managing an inherited setup. With AWS the knowledge is at least transferable and you can find peopl…

In my experience, a back end on the cloud isn't necessarily any less complex than something self hosted.

Re: Don't rent the cloud, own instead

#474

Earlier quoted context omitted.

That's not a factual statement over reality, but more of a normative judgement to justify resignation. Yes, professionals that know how to actually do these things are not abundantly available, but available enough to achieve the transition. The talent exists and is absolutely passionate about software freedom and hence highly intrinsically motivated to work on it. The only thing that is lacking so far is the demand…

> and the talent available will skyrocket, when the market starts demanding it. Part of what clouds are selling is experience. A "cloud admin" bootcamp graduate can be a useful "cloud engineer", but it takes some serious years of experience to become a talented on prem sre. So it becomes an ouroboros: moving towards clouds makes it easier to move to the clouds.

Moving towards the brothel makes it easier to get away from the brothel.

Re: Don't rent the cloud, own instead

#475

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

It’s all anecdotal but in my experiences it’s usually opposite. Bored senior engineer wants to use something new and picks a AWS bespoke service for a new project. I am sure it happens a multitude of ways but I have never seen the case you are describing.

Since when is "bored" a synonym for "dishonest"?

Re: Don't rent the cloud, own instead

#476

Earlier quoted context omitted.

its quite simple. if you are a profitable company paying taxes, you 100% want to defer taxes (part of EBITDA) thus trade earnings for market share. This is exactly what TCI did [1] with cable [1] https://www.colinkeeley.com/blog/john-malone-operating-manua...

Believe you're talking about conserving cash through reduced taxes since this guy was against paying taxes. However, spending a premium on cloud services over what you could with an on-prem capital investment does not help your cash position. His tenant of frugality would have conflicted, especially since the cloud premium can easily exceed the tax rate - that is to say, paying taxes would have been cheaper Section i…

Probably depends on where your gross margins would be with cloud and if you're higher or lower growth. If cloud will let you grow faster (HA/DR on-prem is hard) and you'll still have 75-80%+ gross margins, why slow top-line growth to do on-prem?

Re: Don't rent the cloud, own instead

#477

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

The end result of all this is that the percentage of people who know how to implement systems without AWS/Azure will be a single digit. From that point on, this will be the only "economic" way, it doesn't matter what the prices are.

It already is like that, but not because of the cloud. Those of us who begun with computers in the era of the command line were forced to learn the internals of operating systems, and many ended up turning this hobby into a job.

Youngsters nowadays start with very polished interfaces and smartphones, so even if the cloud wasn't there it would take them a decade to learn systems design on-the-job, which means it wouldn't happen anyway for most. The cloud nowadays mostly exists because of that dearth of system internals knowledge.

While there still are around people who are able to design from scratch and operate outside a cloud, these people tend to be quite expensive and many (most?) tend to work for the cloud companies themselves or SaaS businesses, which means there's a great mismatch between demand and supply of experienced system engineers, at least for the salaries that lower tier companies are willing to pay. And this is only going to get worse. Every year, many more experienced engineers are retiring than the noobs starting on the path of systems engineering.

Re: Don't rent the cloud, own instead

#478

Earlier quoted context omitted.

Your numbers don't line up, if you are spending 5k in cloud costs, and on prem is 1/3 of cloud. At 48 month replacement cycle, 1/3 of 5k * 48 months is 80k. So it is 80k vs 5k a month for 48 months. I think the primary reason that people over fixate on the cloud is that they can't do math. So renting is a hedge.

You hit the nail on the head regarding the math. Most teams treat cloud costs as an inevitable tax rather than an engineering variable. As someone with an accounting background turned Cloud Architect, I see this 'math gap' daily. Usually, it's not a cloud vs. on-prem issue, but a lack of infrastructure discipline—idle resources and unoptimized NATs burn through that 48-month budget faster than hardware depreciation e…

The whole discussion and the article are just an instance of an optimization problem, for a crowd that claims to be technical, the fact that the discussion has so much heat is revealing.

Would love to see people read, write and do more math.

Re: Don't rent the cloud, own instead

#479
post #373

Earlier quoted context omitted.

> and the talent available will skyrocket, when the market starts demanding it. Part of what clouds are selling is experience. A "cloud admin" bootcamp graduate can be a useful "cloud engineer", but it takes some serious years of experience to become a talented on prem sre. So it becomes an ouroboros: moving towards clouds makes it easier to move to the clouds.

> A "cloud admin" bootcamp graduate can be a useful "cloud engineer", If by useful you mean "useful at generating revenue for AWS or GCP" then sure, I agree. These certificates and bootcamps are roughly equivalent to the Cisco CCNA certificate and training courses back in the 90's. That certificate existed to sell more Cisco gear - and Cisco outright admitted this at the time.

In part - yes. Useful as in capable of spinning up services without opening glaring security holes or bringing half of the infra down. Like with any tech, it takes experience and guardrails to use it efficiently and effectively.

Re: Don't rent the cloud, own instead

#480

Note that they're running R630/R730s for storage. Those are 12-year old servers, and yet they say each one can do 20 Gbps (2.5 GBps) of random reads. In comparison, the same generation of hardware at AWS ({c,m,r}4) instance maxes out at 50% of that for EBS throughput on m4, and 70% on r4 - and that assumes carefully tuned block sizes. Old hardware is _plenty_ powerful for a lot of tasks today.

On a related-different axis, I've consistently seen on-prem GPUs running identical workloads ~35% faster than the same workloads on the same cloud hardware, regardless of intermediate infra stack layering/versioning choices. Weird but I'm not complaining!

They might be undervolting and underclocking the GPUs to improve longevity?
Post reply on HN