Traditionally, one of the salient features of ARM systems was that it didn't have fans. Here we are now.
True in more ways than one…
AmpereOne: Cores Are the New MHz
111–120 of 132 posts
Re: AmpereOne: Cores Are the New MHz
#112Earlier quoted context omitted.
They actually distribute 240V in USAmerica, one neutral wire and two opposite-phase 120V wires. Outlets in homes are 120V but wiring 240V is not difficult. https://en.wikipedia.org/wiki/Split-phase_electric_power This is different from Europe where we usually distribute 3 phases 130V.
> but wiring 240V is not difficult. Most homes don't have more than a few 240v circuits, and they are usually always dedicated to a single device. My house has 4: range, oven, drier, water heater, all with only a single outlet. The concept of wiring split-phase 240V is not difficult, but getting 4-conductor 12-gauge wiring run to a new location in an existing building is not exactly easy. So, the power delivery type/…
Sure, but wiring 240v is still not difficult -- even in the States. The physical act of wiring 240v is on par with that of wiring 120v.
Re: AmpereOne: Cores Are the New MHz
#113> I'm also happy to see systems like this that run without exotic water cooling and 240V power. Always funny to see 240V power described as exotic (or at least exotic-adjacent). It's the standard across practically the entire world except North and Central America!
Every home in the US with an electric stove or water heater is using 240v electricity. It isn't really very exotic. 3 phase power to homes IS very exotic though and I'm not sure why it would be useful in homes.
In Norway, I understand that electric ranges are commonly cabled up with 3-phase wire that's about the size of a not-special extension cord here in the States. They accomplish this by using 400v.
Meanwhile, also here in the States: The wiring for my very normal (240v) electric range is the size of a baby's arm and might be even stiffer. (That last claim may be a bit hyperbolic: I've bent at least my share of 6/3 wire, but I don't think I've ever bent a baby's arm.)
Re: AmpereOne: Cores Are the New MHz
#114Earlier quoted context omitted.
It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
This resource looks very bad to me as they don't check batched inference at all. This might make sense now when most people a just running single query at once, but pretty soon almost everything will be running queries in parallel to take advantage of the compute.
How do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
Re: AmpereOne: Cores Are the New MHz
#115Earlier quoted context omitted.
From the article: > the AmpereOne A192-32X is $5,555, while the EPYC 9965 is almost $15,000!
That is 192 EPYC cores though. The EPYC 9654 costs half as much as the A192-32x, has half as many cores, but still beats the Ampere in geometric mean of the phoronix suite. https://www.phoronix.com/review/ampereone-a192-32x/12
> The EPYC 9654 costs half as much as the A192-32x
Where do you see EPYC 9654 being sold for 3k$? Only price I can find online is about the same as A192-32x, around 5k.
Re: AmpereOne: Cores Are the New MHz
#116Earlier quoted context omitted.
This resource looks very bad to me as they don't check batched inference at all. This might make sense now when most people a just running single query at once, but pretty soon almost everything will be running queries in parallel to take advantage of the compute.
How do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
It depends on the framework. Here's a LlamaSharp example: https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Exa...
Re: AmpereOne: Cores Are the New MHz
#117Earlier quoted context omitted.
How do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
It depends on the framework. Here's a LlamaSharp example: https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Exa...
My question wasn't about how to run multiple queries against the LLM but rather how is it even possible from transformer architecture PoV to have a single LLM hosting multiple and different end clients. I'm probably missing something but can't figure that out yet.
Re: AmpereOne: Cores Are the New MHz
#118Earlier quoted context omitted.
It depends on the framework. Here's a LlamaSharp example: https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Exa...
My question wasn't about how to run multiple queries against the LLM but rather how is it even possible from transformer architecture PoV to have a single LLM hosting multiple and different end clients. I'm probably missing something but can't figure that out yet.
If you have a branchless program, you can execute the same step of the program on multiple different inputs. https://en.wikipedia.org/wiki/SIMD
Re: AmpereOne: Cores Are the New MHz
#119Earlier quoted context omitted.
3 phases 230v you mean?
> In Europe, three-phase 230/400 V is most commonly used. However, 130/225 V, three-wire, two-phase electric power discontinued systems called B1 are used to run old installations in small groups of houses when only two of the three-phase high-voltage conductors are used.
I know how to wikipedia too! They said usually, not "in some really rare cases"