Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

tomshardware.com

Post reply on HN
Print view View original

281–290 of 331 posts

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#281

Post by DiogenesKynikos »

Alupis wrote: post #244
lmz wrote: post #240

Earlier quoted context omitted.

Much higher standards than European ones? Because the cars do sell there as well.

Yes. The EU focuses on safety of pedestrians, and the US focuses on safety of occupants. That's not to say a vehicle cannot do both well (see the many European vehicles sold in the US), but that is to say Chinese vehicles may not meet the US standards. The US has a lot of vehicle regulations that significantly differ from the EU market.

The EU focuses on both aspects of safety.

You can look at the Euro NCAP ratings for the 2023 BYD Seal, for example: https://www.euroncap.com/en/results/byd/seal/50012. They break down the rating based on safety for adult occupants, child occupants and pedestrians. These ratings are based on many different crash tests.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#282

Post by redserk »

ahmeneeroe-v2 wrote: post #215
Yoric wrote: post #175

Earlier quoted context omitted.

Frankly, this sounds like you're repeating propaganda. When the US called its allies to its wars, NATO responded. Now that the rest of NATO is being threatened, the US is playing neutral, trying to see which side will bid highest for their help.

Which NATO ally is being threatened?

Poland[1, 3], Latvia [2], Lithuania [3]

1. https://apnews.com/article/russia-nato-members-borders-airsp...

2. https://www.reuters.com/world/europe/russian-drone-that-cras...

3. https://www.reuters.com/world/europe/lithuania-says-russian-...

Among other events, like drones being spotted near commercial airports.

Are you suggesting repeated airspace intrusions and acts against civilians are merely acts of innocence?

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#283

Post by jgalt212 »

godelski wrote: post #258
jychang wrote: post #235

Earlier quoted context omitted.

They definitely won't idle out- if they idle out, it'll take on the order of up to 60 seconds to load the model back into VRAM, depending on the model. That's an eternity for a request. I highly doubt they will timeout any model they serve.

> That's an eternity for a request. I highly doubt they will timeout any model they serve. That's what easing functions are for. Let's say 10 GPUs are in use. You keep another 3 with the model loaded. If demand increases slowly you slowly increase your headroom. If demand increases rapidly, you also increase rapidly. The correct way to do this is more complicated and you should model based on your usage history, but…

The thundering herd breaks this scheme.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#285

Post by DiogenesKynikos »

Alupis wrote: post #238
fooker wrote: post #176

Earlier quoted context omitted.

> China may never be able to export these chips While you type this, the rest of the world is already using Chinese cars, something that was unthinkable a year or two ago. The US has closed the market off from this for its auto industry to survive.

This statement doesn't make a lot of sense. 40-50% of vehicles are foreign-made already[1]. I would strongly wager it's vastly more likely that these Chinese vehicles do not meet US safety standards - which are quite high. [1] https://www.whitehouse.gov/fact-sheets/2025/03/fact-sheet-pr...

Chinese companies make cars with 5-star European crash ratings. They are perfectly capable of meeting US safety standards.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#286

Post by DiogenesKynikos »

myth_drannon wrote: post #42
hunglee2 wrote: post #2

The US attempt to slow down China's technological development succeeds on the basis of preventing China from directly following the same path, but may backfire in the sense it forces innovation by China in a different direction. The overall outcome for us all may be increase efficiency as a result of this forced innovation, especially if Chinese companies continue to open source their advances, so we may in the end h…

China's innovation relies on the stolen western IP, without it, China is nothing. Also USSR/Russia is no longer a scientific powerhouse that can supply China with some military innovation. A dictatorship combined with cheap labour it 100% guarantees that the country's innovation is stunted, no matter what the Chinese propaganda claims.

Nowadays, China produces about the same amount of high-quality R&D as the US does.

Your view of China is several decades out of date. Chinese labor isn't even that cheap any more. China is moving up the value chain and outsourcing production that needs cheap labor to poorer countries (or replacing workers with robots altogether).

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#287

Post by cnr »

miki123211 wrote: post #271
bee_rider wrote: post #154

Earlier quoted context omitted.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a du…

Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#288

Post by behnamoh »

miki123211 wrote: post #271
bee_rider wrote: post #154

Earlier quoted context omitted.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a du…

> This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.

Local models are never a dumb idea. The only time it's dumb to use them in an enterprise is if the infra is Mac Studio with M3 Ultra because pp time is terrible.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#289

Post by carderne »

cnr wrote: post #287
miki123211 wrote: post #271

Earlier quoted context omitted.

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a du…

Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?

Layman understanding:

Because as a function of hardware and electricity costs, a “cloud” GPU will be many times more efficient per output token. You aren’t loading/offloading models and don’t have any parts of the GPU waiting for input. Everything is fully saturated always.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#290

Post by throwaway2037 »

audunw wrote: post #273
notepad0x90 wrote: post #143

Earlier quoted context omitted.

it's bad for the US, because China has 10x population. the US can't make up in quality, what it lacks in quantity without immigration and attracting foreigners.

I don’t think you realise how many people in China still live in poverty, with not much prospects of improvement. I don’t see how having 100 million poor small scale farmers is a benefit in this equation. You still can’t become a Chinese citizen. You can come to USA or Europe and build a life for yourself. While some people go to China to make some money for a few years you can’t really build a life. So I think US an…

    > You still can’t become a Chinese citizen.
This is untrue.

This Wiki page says: https://en.wikipedia.org/wiki/Chinese_nationality_law

    > Foreign nationals may naturalize if they are permanent residents in any part of China
More specifically: I recall living in Hongkong and learning about non-ethnic Chinese people (usually South Asians) who became Chinese citizens to acquire a Hongkong passport. The process required them to denounce all existing citizenships. In the eyes of HK and mainland gov'ts, those people are Chinese citizens with HK PR and carry HK passport. The candidates needed to demonstrate sufficient language skills in either Cantonese or Mandarin. (I'm unsure if other regional languages were allowed.)

    > You can come to USA or Europe and build a life for yourself.
There is a tiny minority of foreigners who do this in mainland China, as well as Japan, Korea, Taiwan, Thailand, Cambodia, and Vietnam. Usually, they come to teach English, then marry a local and "build a life". Some also come as skilled migrants.

    > Yeah they can exploit cheap labor from the poor interior. But the US is doing something similar in some ways with central/southern America.
I don't follow the part about the US exploiting LATAM labour. Can you explain more?
Post reply on HN