Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AmpereOne: Cores Are the New MHz

jeffgeerling.com

Post reply on HN
Print view View original

51–60 of 132 posts

Re: AmpereOne: Cores Are the New MHz

#51

Post by johnklos »

revnode wrote: post #28
loudmax wrote: post #20

I found this part particularly interesting: > Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second. If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. T…

It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

You know, there's nothing wrong with running a slow LLM.

For some people, they lack the resources to run an LLM on a GPU. For others, they want to try certain models without buying thousands of dollars of equipment just to try things out.

Either way, I see too many people putting the proverbial horse before the cart: they buy a video card, then try to fit LLMs in to the limited VRAM they have, instead of playing around, even if at 1/10th the speed, and figuring out which models they want to run before deciding where they want to invest their money.

One token a second is worlds better than running nothing at all because someone told you that you shouldn't or can't because you don't have a fancy, expensive GPU.

Re: AmpereOne: Cores Are the New MHz

#52

Post by MrDrMcCoy »

revnode wrote: post #28
loudmax wrote: post #20

I found this part particularly interesting: > Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second. If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. T…

It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

Bummer that they have no stats for AMD, Intel, Qualcomm, etc (C|G|N|X)PUs.

Re: AmpereOne: Cores Are the New MHz

#53

Post by zozbot234 »

johnklos wrote: post #51
revnode wrote: post #28

Earlier quoted context omitted.

It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

You know, there's nothing wrong with running a slow LLM. For some people, they lack the resources to run an LLM on a GPU. For others, they want to try certain models without buying thousands of dollars of equipment just to try things out. Either way, I see too many people putting the proverbial horse before the cart: they buy a video card, then try to fit LLMs in to the limited VRAM they have, instead of playing arou…

> For some people, they lack the resources to run an LLM on a GPU.

Most people have a usable iGPU, that's going to run most models significantly slower (because less available memory throughput, and/or more of it being wasted on padding, compared to CPU) but a lot cooler than the CPU. NPU's will likely be a similar story.

It would be nice if there was an easy way to only run the initial prompt+context processing (which is generally compute bound) on iGPU+NPU, but move to CPU for the token generation stage.

Re: AmpereOne: Cores Are the New MHz

#54

Post by SahAssar »

wmf wrote: post #36
cbmuser wrote: post #35

Earlier quoted context omitted.

Sounds like x86-64 is starting to lose some of it’s market share very soon.

Half of all CPUs in AWS are ARM already. Update: "Over the last two years, more than 50 percent of all the CPU capacity landed in our datacenters was on AWS Graviton." So not half of all but getting there. https://www.nextplatform.com/2024/12/03/aws-reaps-the-benefi...

That quote says that half of all new CPUs are Graviton. Very different.

Re: AmpereOne: Cores Are the New MHz

#55

Post by Twirrim »

droideqa wrote: post #39
wmf wrote: post #36

Earlier quoted context omitted.

Half of all CPUs in AWS are ARM already. Update: "Over the last two years, more than 50 percent of all the CPU capacity landed in our datacenters was on AWS Graviton." So not half of all but getting there. https://www.nextplatform.com/2024/12/03/aws-reaps-the-benefi...

I just looked it up - that is a mistaken statistic. 50% of their CPUs are not Arm, but AWS has 50% of all server-side Arm CPUs. "But that total is beaten by just one company – Amazon – which has slightly above 50 percent of all Arm server CPUs in the world deployed in its Amazon Web Services (AWS) datacenters, said the analyst."[0] [0]: https://www.theregister.com/2023/08/08/amazon_arm_servers/

In part that'll be because they mandated every service team migrate to ARM. Service teams had to have extensive justification to avoid it. With good reason, too, the reason for the effort was the significant cost savings.

Re: AmpereOne: Cores Are the New MHz

#56

Post by Twirrim »

jsheard wrote: post #6
znpy wrote: post #2

Weird, but this makes me thing X86-64 might actually be better? It isn't mentioned anywhere in __this__ article, but the power draw of that chip is 276W. I got this from Phoronix [1]: > The AmpereOne A192-32X boasts 192 AmpereOne cores, a 3.2GHz clock frequency, a rated 276 Watt usage power Which is interesting because it's almost half of AMD's 192-core offering [2]. Why is this interesting? the AMD offering draws a…

Hyperthreading doesn't even get close to doubling actual performance, it depends on the workload but AMDs Zen5 gains about 15% from HT on average according to Phoronix's benchmarks. https://www.phoronix.com/review/amd-ryzen-zen5-smt/8

There's also the added fun that using hyperthreading/SMT cores impacts your workload on the corresponding primary cores as well. Depending on your workload SMT can end up being a throughput/latency trade off you might not have been aware of.

Ampere doesn't do SMT, so you get comparable performance across the board.

Re: AmpereOne: Cores Are the New MHz

#57

Post by spott »

com2kid wrote: post #31
stonemetal12 wrote: post #21

Earlier quoted context omitted.

When was the last time you saw a server with 10GB Ram no matter the number of cores\threads?

I've run plenty of microservices with 256 or 512 GB of RAM, and they were handling large loads. So long as each request is short lived, and using a runtime with low per request overhead (e.g. Node), memory is not really a problem for many types of workloads.

Just to be clear, you meant MB of RAM, right?

Re: AmpereOne: Cores Are the New MHz

#58

Post by geerlingguy »

znpy wrote: post #2

Weird, but this makes me thing X86-64 might actually be better? It isn't mentioned anywhere in __this__ article, but the power draw of that chip is 276W. I got this from Phoronix [1]: > The AmpereOne A192-32X boasts 192 AmpereOne cores, a 3.2GHz clock frequency, a rated 276 Watt usage power Which is interesting because it's almost half of AMD's 192-core offering [2]. Why is this interesting? the AMD offering draws a…

I mentioned it a couple times in the article, but here's a bullet point right in the conclusion:

> "Arm is more efficient": that's not always true—AMD just built the most efficient 192 core server this year, beating Ampere

I also talk about idle power draw being very high in comparison to AMD.

Since I don't have a Turin system to test I didn't do direct wattage comparisons but recommended the linked Phoronix comparison.

Re: AmpereOne: Cores Are the New MHz

#60

Post by incrudible »

voxadam wrote: post #32
ThinkBeat wrote: post #17

How much does one of these servers' cost? Is the estimate that it is cheaper based on comparable server with the same or similar core count?

From the article: > the AmpereOne A192-32X is $5,555, while the EPYC 9965 is almost $15,000!

That is 192 EPYC cores though. The EPYC 9654 costs half as much as the A192-32x, has half as many cores, but still beats the Ampere in geometric mean of the phoronix suite.

https://www.phoronix.com/review/ampereone-a192-32x/12

Post reply on HN