Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

tomshardware.com

Post reply on HN
Print view View original

191–200 of 331 posts

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#191

Post by Workaccount2 »

segmondy wrote: post #122
Workaccount2 wrote: post #103

Earlier quoted context omitted.

But they aren't keeping up They are lauded for the ability to cost ratio, or their ability to parameter ratio, but virtually everyone using LLMs for productive work are using ChatGPT/Gemini/Claude. They are kind of like Huffy bicycles. Good value, work well, but if you go to any serious event, no one will be riding one.

they are keeping up. i have been using just chinese models for the last 2 years. chatgpt/gemini/claude have marketing. there's nothing that you can do with those models that can't be done with deepseek, glm or kimi. if there is, do let us know.

They can't attract a large contingent of users. Because despite being able to do everything the big name models can do, they cannot do it as well.

This aligns with the benchmarks as well; they benchmark great for what they are, but still bottom of the barrel when competing for "state of the art."

And yes, it's great you daily Chinese models, but the vast majority of people try them, say "impressive", then go back to the most performant models.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#192

Post by svachalek »

bee_rider wrote: post #154
kilotaras wrote: post #47

Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found Instead of 1192 GPUs they now use 213 for serving those requests.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#193

Post by Workaccount2 »

MSFT_Edging wrote: post #149
Workaccount2 wrote: post #103

Earlier quoted context omitted.

But they aren't keeping up They are lauded for the ability to cost ratio, or their ability to parameter ratio, but virtually everyone using LLMs for productive work are using ChatGPT/Gemini/Claude. They are kind of like Huffy bicycles. Good value, work well, but if you go to any serious event, no one will be riding one.

The downside of their efficiency and cost-ratio is that they undermine the circular economy of massive data centers, GPU sales, and VC money that is constructing an extremely wasteful bubble.

The bubble is there in China too, it's just on the governments books instead of private investors books.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#194

Post by svachalek »

make3 wrote: post #159
bee_rider wrote: post #154

Earlier quoted context omitted.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

the models are huge, so not a single (latest gen) one can fit on a single GPU. It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.

Per the very short article, the solution was to pack multiple models per GPU.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#195

Post by corimaith »

delfinom wrote: post #130
ahmeneeroe-v2 wrote: post #124

Earlier quoted context omitted.

Is that the best way? China's rise had already happened by the 2010s Preventing that could have been prevented in the 70s, 80s, 90s by stopping offshoring, blocking student visas, and prosecuting IP theft.

Eh? They are a country with a culture that values education like a few other asian countries. They were set back by the colonial bullshit of Europe, devastation of WW2 and communist revolutions and so. They would have gotten to the same point as they are today, just longer if anything if any attempt was made to hamper it.

Traditional Confucian Exams were more of elite dispute resolution mechanisms. The idea of practical education is largely introduced by the West.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#196

Post by notepad0x90 »

csomar wrote: post #38
notepad0x90 wrote: post #12

Earlier quoted context omitted.

I think anti-immigrant rhetoric will have the most impact against the US. A lot of the people innovating on this stuff are being maligned and leaving in droves. Aside from geography, attracting talent from all over the world is the one edge the US has a nation over countries like China. But now the US is trying to be xenophobic like China, restrict tech import/export like China but compete against 10x population and…

> But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india. How can they have international hegemony before they clear their regional order? China is more interested in aligning Taiwan than invading; though it’ll probably invade if it can’t align it diplomatically. China is probably not interested in contin…

They've dominated their region for a long time. Vietnam and NK are their sattellites basically. Russia is their close ally. The only regional opposition they have is India. Taiwan is a small bug to them, the only reason they haven't invaded it is because of TSMC and ASML, mainland China hasn't caught up with them and Nvidia yet.

They're all over Africa and south Asia. But unlike the US/West they don't exert political influence. When they build infrastructure for example, they set up worker camps, isolated from the local population. they only employ their own imported people and clean up and leave quietly afterwards.

They're acting like good business partners, instead of a superpower wielding it's might and extending its influence. it's good for business for all involved parties for sure, and smart too. But not having strong influence means for example, the US can come in, outbid them, bail out african loans to China and they lose that source of commerce.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#197

Post by marknutter »

notepad0x90 wrote: post #12
hunglee2 wrote: post #2

The US attempt to slow down China's technological development succeeds on the basis of preventing China from directly following the same path, but may backfire in the sense it forces innovation by China in a different direction. The overall outcome for us all may be increase efficiency as a result of this forced innovation, especially if Chinese companies continue to open source their advances, so we may in the end h…

I think anti-immigrant rhetoric will have the most impact against the US. A lot of the people innovating on this stuff are being maligned and leaving in droves. Aside from geography, attracting talent from all over the world is the one edge the US has a nation over countries like China. But now the US is trying to be xenophobic like China, restrict tech import/export like China but compete against 10x population and…

Nobody is anti-immigrant outside of a small pocket of anti-H1B folks in the tech community. People are, however, anti-illegal-immigrant, which is completely different.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#198

Post by garblegarble »

ambicapter wrote: post #95
wood_spirit wrote: post #88

Earlier quoted context omitted.

But at the same time they are fielding multiple new stealth aircraft and their jets and missiles outperformed western aircraft in the recent Pakistan India flare-up.

So you'd think they'd be able to build a commercial jet liner, no?

China currently can't make the high-performance, efficient, long-life jet engines that US & Europe make. The commercial market is heavily cost-sensitive, so they can't compete there currently as a result.

This doesn't matter so much for military purposes: they can easily eat the cost of a higher maintenance and replacement schedule on a smaller number of military jets with fewer hours on them.

This gives them more iteration cycles, speeding their building up of experience. They're catching up. Industrial espionage will help them along too, but not as much as the experience from engineering their own designs.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#199

Post by slaw »

tw04 wrote: post #146
dlisboa wrote: post #11

Earlier quoted context omitted.

History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years. In many senses there's hubris in the western* view of China accomplishments: most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the name…

> History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years. Really? How long has China been attempting to build their own jet engines? How long have they been attempting to build competitive CPUs? History has shown withholding tech successfully keeps them at least a generation behind the west. In some fields like CPUs they…

Jet engine since at least 2011. CPU since 2000

https://en.wikipedia.org/wiki/ACAE_CJ-1000A

https://en.wikipedia.org/wiki/Semiconductor_Manufacturing_In...

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#200

Post by gessha »

corimaith wrote: post #195
delfinom wrote: post #130

Earlier quoted context omitted.

Eh? They are a country with a culture that values education like a few other asian countries. They were set back by the colonial bullshit of Europe, devastation of WW2 and communist revolutions and so. They would have gotten to the same point as they are today, just longer if anything if any attempt was made to hamper it.

Traditional Confucian Exams were more of elite dispute resolution mechanisms. The idea of practical education is largely introduced by the West.

Which reminds me of this article about the origin of the research university by the Germans:

https://asteriskmag.com/issues/10/the-origin-of-the-research...

Post reply on HN