Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Switching to AWS Graviton slashed our infrastructure bill

squeaky.ai

Post reply on HN
Print view View original

71–80 of 140 posts

Re: Switching to AWS Graviton slashed our infrastructure bill

#71

Post by MuffinFlavored »

Agingcoder wrote: post #68

I'm not sure I understand the point of this article : in theory they don't depend on x86 only code, so they've switched to arm and it worked, as expected, and things are cheaper. I'm happy that they've shrunk their bill, but I somehow expected some kind of 'unfortunately, things went wrong because of bizarre memory model issues causing difficult concurrency bugs'. What am I missing?

> in theory they don't depend on x86 only code, so they've switched to arm and it worked, as expected, and things are cheaper.

What's Intel's response to this as a company? I know that isn't mentioned in the article but... just curious

Does Intel have any ARM offering whatsoever?

Does AMD have any ARM offering?

Re: Switching to AWS Graviton slashed our infrastructure bill

#72

Post by jiggawatts »

gryf wrote: post #49
tester756 wrote: post #48

Earlier quoted context omitted.

>We moved from multiple DCs to AWS and the costs are approximately 8x what the pre-migration costs are. 8x?? that's crazy, what where you doing wrong then?

Everything, all at once. SMEs can't reliably manage that transition with any skillset and still deliver a product at the same time.

I’m very curious to hear more details!

Did you use reservations to reduce costs?

Was it a lift and shift with VM configs staying as-is? (I’ve seen a lot of empty 1TB “app” drives burning money in the cloud!)

You complain about PaaS services, but I can’t imagine 8 data centres worth of stuff being converted to PaaS in hurry!

Re: Switching to AWS Graviton slashed our infrastructure bill

#73

Post by sofixa »

arecurrence wrote: post #40
c7DJTLrn wrote: post #2

Cloud cost optimisation is underrated. In the companies I've worked in nobody has really given a shit (at least not under normal economic circumstances). In the industry there's a strong avoidance of ARM compute instances for no good reason. If I were building from scratch today I would definitely go with Graviton.

Completely agree... the only exception I've run into is that for small operations build tooling often doesn't work well with arm64. EG: GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64... likewise aws-cdk literally could not run an arm64 fargate ecs deployment for months after support was added (They simply did not support the required attribute in the container definition). I would…

> GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64

What type of container, and on what runner? That has not been my experience at all, a cross-compiling buildx build with Python and a bunch of libraries takes only slightly longer for arm64 than it did for x86.

Re: Switching to AWS Graviton slashed our infrastructure bill

#74

Post by scarface74 »

gryf wrote: post #30
jacobwg wrote: post #18

Earlier quoted context omitted.

Anecdotally, this is starting to shift with M1 MacBooks, Graviton is looking more attractive for precisely that architecture parity reason for teams using majority M1 devices.

Yeah if only. Our ops people are too uneducated to be able to deploy anything Apple. Literally there are armies of factory pressed Windows monkeys but nothing in the Apple space. Note to apple: please start concentrating on the enterprise sector. We're dying over here. My Dell weighs 3x my personal M1 MBP, has a shitty keyboard with keys designed for Borrowers, the battery lasts 8 minutes and it reduces my sperm coun…

> Note to apple: please start concentrating on the enterprise sector. We're dying over here.

Enterprise sales are where the customer is not the user. Apple does best when the user is the purchaser.

Also, I know for a fact that Macs are well supported at scale by many large tech companies including my own.

Re: Switching to AWS Graviton slashed our infrastructure bill

#75

Post by _joel »

fred_is_fred wrote: post #59
_joel wrote: post #16

Earlier quoted context omitted.

It is unless you're the first poor soul to embark on the journey with lots of x86_64 buildup. Having said that, it's been fun though so far. Managed to migrate our dev local k8s toolchain and been using buildx to make multiarch images and manifests for our internal stuff.

We've moved everything to graviton except EKS (MSK, RDS, etc). Did you have any major issues? Would you do it again?

We've only just started! Hence me hitting all the fun issues with our 'organically' grown dev toolchain :)

It's been fairly simple so far with a few niggles. We won't be able to go full arm due to having to support 'stuff' but for the management planes and prometheus etc it's all hunky dory.

Re: Switching to AWS Graviton slashed our infrastructure bill

#76

Post by scarface74 »

gryf wrote: post #65
lostmsu wrote: post #61

Earlier quoted context omitted.

Are your developers forced to use laptops?

Unfortunately yes. I would rather a desktop but they don't know how to pay half as much for the same specification. The desks in the office are all equipped with docks and expensive WiFi mesh driven by COVID mentality so that is the status quo. Just send me a fucking workstation. Nope too hard.

Who wants to be in an office in 2022 unless necessary (specialized hardware, etc.)

Re: Switching to AWS Graviton slashed our infrastructure bill

#77

Post by moloch-hai »

Agingcoder wrote: post #68

I'm not sure I understand the point of this article : in theory they don't depend on x86 only code, so they've switched to arm and it worked, as expected, and things are cheaper. I'm happy that they've shrunk their bill, but I somehow expected some kind of 'unfortunately, things went wrong because of bizarre memory model issues causing difficult concurrency bugs'. What am I missing?

You can get those bugs when you are doing your own atomics, and your code relies on x86's relaxed memory semantics. It looks like their code is JS and Go, which buries that stuff. Services they use were already proven out on ARM. (Or, maybe, are not on ARM?)

Relaxed memory bus semantics imposes a pretty substantial performance cost. Depending on how they are billed, this might account for a big chunk of their lower cost. But probably not.

Their real problem is that they are firmly entrenched in proprietary Amazon services, so switching to another cloud would be very difficult. Amazon can raise prices 35% anytime, and what can they do?

Re: Switching to AWS Graviton slashed our infrastructure bill

#78

Post by waych »

sofixa wrote: post #73
arecurrence wrote: post #40

Earlier quoted context omitted.

Completely agree... the only exception I've run into is that for small operations build tooling often doesn't work well with arm64. EG: GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64... likewise aws-cdk literally could not run an arm64 fargate ecs deployment for months after support was added (They simply did not support the required attribute in the container definition). I would…

> GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64 What type of container, and on what runner? That has not been my experience at all, a cross-compiling buildx build with Python and a bunch of libraries takes only slightly longer for arm64 than it did for x86.

My favorite way to watch this slow down is to introduce some node workloads into the build workflow.

Re: Switching to AWS Graviton slashed our infrastructure bill

#79

Post by lostmsu »

scarface74 wrote: post #62
lostmsu wrote: post #61

Earlier quoted context omitted.

Are your developers forced to use laptops?

Do you propose they carry desktops back and forth from their office and to conference rooms?

Every conference room could have a device to facilitate presentation.

Re: Switching to AWS Graviton slashed our infrastructure bill

#80

Post by yazaddaruvala »

jiggawatts wrote: post #69
c7DJTLrn wrote: post #2

Cloud cost optimisation is underrated. In the companies I've worked in nobody has really given a shit (at least not under normal economic circumstances). In the industry there's a strong avoidance of ARM compute instances for no good reason. If I were building from scratch today I would definitely go with Graviton.

At $dayjob I found an unused box in the cloud running an expensive database engine. It was idle for months, created to be used by a consultant on a project that had wound up. The consultant had quit his consultancy on top of this. I was told under no uncertain terms not to even think of touching this VM because “the budget has been approved”. I was shocked at the flagrant waste of money and assumed it was a one-off a…

FWIW, in these situations you're better off proposing:

"I'm going to reuse this VM, to help our ... fleet scale better."

That way your management continues to use their allocated budget, and your real prod systems work slightly better (also will eventually require less additional $ to scale up - helping the company i.e. shareholders).

The thing to remember:

You would assume all middle management really manages are a top line and a bottom line. Numbers related to their KPIs/OKRs are roughly a top line, and numbers related to their resources (humans and cloud infra budget) are roughly their bottom line.

The reality: Middle management's resources (humans and cloud infra budget) are not their bottom line. Middle management gets rewarded (promoted) when they have "enough scope", scope has roughly always been defined by number of people (it now also includes things like cloud cost budget). As such middle management has to say "we need to do more with less", but they are promoted based on these numbers going up!

Is this reward structure in the best interest of companies (i.e. customers and/or shareholders)? No, neither. Is there a better system? Not yet. Is the reward structure created by middle management for middle management? Likely.

So in the meanwhile, if you don't want to become unmotivated, might as well work within the current reward structures.

Post reply on HN