Live data from Hacker News

Google Kubernetes Engine adds support for Arm nodes

cloud.google.com

31–40 of 51 posts

Re: Google Kubernetes Engine adds support for Arm nodes

#31

Earlier quoted context omitted.

I've been passively curious about this for a homelab--does linux virtualize on apple silicon today? I was of the impression this didn't work when they announced m1 in 2020, but I could be misremembering.

Absolutely. It worked from day one. There are issues with emulating x86 Linux, but that’s a different story. For ARM Linux — just ordinary qemu works fine.

Oh, interesting. That's great to hear!

Re: Google Kubernetes Engine adds support for Arm nodes

#32

Great. Hopefully Azure is next, so we can get ARM Github Actions runners on which I can build ARM binaries. I can already do it with qemu-user but it's slow.

You can already self-host GH Runners on ARM.

What kind of ARM machine should I buy if I want to self-host an ARM server? My not-that-recent look at the market shows everyone kind of doing their own thing; Amazon makes Amazon's ARM servers, Apple makes Apple's ARM chips, etc. As some random guy who wants to test on ARM, it's annoying. Maybe things have improved recently, though?

(I'm guessing self-host realistically means "get a VM on AWS", which is probably fine for CI if you already use AWS. A little annoying to have another monthly bill to pay if you don't, though.)

Re: Google Kubernetes Engine adds support for Arm nodes

#33
post #10

People would probably be more interested in the underlying Google Cloud ARM announcement: https://news.ycombinator.com/item?id=32084887

Yup, my team did a little copy-paste lab if folks want to kick the tires: https://github.com/sadasystems/gke-multiarch-guide

Also, the PM's are going to do an AMA for this on reddit tomorrow:

https://www.reddit.com/r/googlecloud/comments/vy8hx3/ama_wit...

Come join us! Also, hi Moles :)

Re: Google Kubernetes Engine adds support for Arm nodes

#34
post #2

Pricing is in line with the cloud: expensive. That's it, they made it: now Arm servers are mainstream.

I always found the CPU pricing of cloud to be relatively reasonable. It's everything else that's expensive, and egress bandwidth just ridiculous.

I've done some cost analyses between our AWS and DC infrastructure.

To come up with our on-prem compute costs, we baked in the cost of power, real estate, staff, taxes, network infrastructure, servers (both in-use and in reserve), etc. On the AWS side, we used 3 year RIs and Savings Plan. After all that, there was around a 30% cost advantage on-prem. That's non-trivial, but not as big as one might think.

Outbound networking, however, is ludicrously cheaper on-prem. It's about 85% cheaper on-prem than in AWS. Bandwidth is not expensive outside the public cloud.

In fact, egress volume is the #1 cost driver for us moving a service on-prem or building it there to begin with. Some of the AWS managed services are also very pricey, but nowhere near the egregious markup of egress bandwidth.

Re: Google Kubernetes Engine adds support for Arm nodes

#35

Earlier quoted context omitted.

You can already self-host GH Runners on ARM.

The point is to not have to self-host. It costs money, is a pain because maintenance is required, and there are security issues with running CI for public pull requests.

Azure already has ARM VMs (as well as AKS)?

The equivalent would be cloud build, not GKE GH actions, right? does cloud build support arm runners?

Re: Google Kubernetes Engine adds support for Arm nodes

#37

Earlier quoted context omitted.

You can already self-host GH Runners on ARM.

What kind of ARM machine should I buy if I want to self-host an ARM server? My not-that-recent look at the market shows everyone kind of doing their own thing; Amazon makes Amazon's ARM servers, Apple makes Apple's ARM chips, etc. As some random guy who wants to test on ARM, it's annoying. Maybe things have improved recently, though? (I'm guessing self-host realistically means "get a VM on AWS", which is probably fin…

Buy a Mac mini and install Linux on it? That feels like it is (or will soon be) the best option.

Re: Google Kubernetes Engine adds support for Arm nodes

#38

Earlier quoted context omitted.

You can already self-host GH Runners on ARM.

What kind of ARM machine should I buy if I want to self-host an ARM server? My not-that-recent look at the market shows everyone kind of doing their own thing; Amazon makes Amazon's ARM servers, Apple makes Apple's ARM chips, etc. As some random guy who wants to test on ARM, it's annoying. Maybe things have improved recently, though? (I'm guessing self-host realistically means "get a VM on AWS", which is probably fin…

Your best options IMO are either 1) a Mac Mini with 16GB of RAM running a Linux VM or Asahi Linux connected OR 2) an Nvidia Jetson Xavier AGX; you can get the 32GB version for like $700 USD I think. (The newer Orin is a fair bit more expensive.)

You realistically want something with ARMv8.2 or better, these are relatively easy to acquire with the current supply chains, they're beefy, can be equipped with fast storage, and they are both small units you can put on your desk. Note that the Xavier will require you to fiddle with the usual Nvidia bullshit through their SDK but it's otherwise a standard Ubuntu machine. It should be possible to get another distro on there too. The M1 will almost certainly perform better overall watt-for-watt though.

Re: Google Kubernetes Engine adds support for Arm nodes

#39

Earlier quoted context omitted.

I always found the CPU pricing of cloud to be relatively reasonable. It's everything else that's expensive, and egress bandwidth just ridiculous.

I've done some cost analyses between our AWS and DC infrastructure. To come up with our on-prem compute costs, we baked in the cost of power, real estate, staff, taxes, network infrastructure, servers (both in-use and in reserve), etc. On the AWS side, we used 3 year RIs and Savings Plan. After all that, there was around a 30% cost advantage on-prem. That's non-trivial, but not as big as one might think. Outbound net…

85% cheaper seems little. In my case (collocation) bandwidth is 95% cheaper (i.e AWS is 20x as expensive) than AWS.

Re: Google Kubernetes Engine adds support for Arm nodes

#40
post #2

Pricing is in line with the cloud: expensive. That's it, they made it: now Arm servers are mainstream.

Dell and HP are still nowhere to be seen in the on-prem Arm server market. Is there not enough of a cost advantage for there to be demand?

https://www.hpe.com/us/en/newsroom/press-release/2022/06/hew...
Post reply on HN