Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

tomshardware.com

Post reply on HN
Print view View original

291–300 of 331 posts

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#291

Post by throwaway2037 »

reliabilityguy wrote: post #5
hunglee2 wrote: post #2

The US attempt to slow down China's technological development succeeds on the basis of preventing China from directly following the same path, but may backfire in the sense it forces innovation by China in a different direction. The overall outcome for us all may be increase efficiency as a result of this forced innovation, especially if Chinese companies continue to open source their advances, so we may in the end h…

Tbh this whole situation reminds of how Japan excelled in making a lot more with a lot less after WW2, e.g., fuel-efficient engines, light cars, etc. these constraints were not present in the US (and to some extent in Europe), and resulted in US cars being completely not competitive in non-US markets.

    > US cars being completely not competitive in non-US markets
This is definitely untrue in Canada and large parts of LATAM. American cars are all over those places, along with Japanese cars.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#292

Post by sofixa »

huntertwo wrote: post #137
sofixa wrote: post #26

Earlier quoted context omitted.

> China has been pushing to build its own aircraft for >23 years. It took 14 years for COMAC to get its first regional jet flying commercial flights on a Chinese airline, and 21 years to get a narrow-body plane flying a commercial flight on a Chinese airline And both those planes have a strong dependency on "western" components that won't be overcome before the 2030s, and even then, they're around a generation behind…

5 years behind becomes 3 years behind. China is expanding their manufacturing abilities faster than the US. Soon they will surpass the US. Look no further than their generic consumer electronics manufacturing.

It's not 5 years behind, engine tech is more on a ~15 year cycle, or even more.

CFM LEAP, latest short-to-medium-haul airliner engine from CFM (GE+Safran) is from 2013 (first run). Its predecessor, CFM56, is from 1974 (first run) and saw a few evolutions, including as late as 2009.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#293

Post by tobyhinloopen »

jychang wrote: post #235
svachalek wrote: post #192

Earlier quoted context omitted.

Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.

They definitely won't idle out- if they idle out, it'll take on the order of up to 60 seconds to load the model back into VRAM, depending on the model. That's an eternity for a request. I highly doubt they will timeout any model they serve.

Why does it take 60 seconds to load data from RAM to VRAM? Shouldn't the PCIE bandwidth allow it to fully load it in a few seconds?

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#294

Post by nickysielicki »

> Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds.

I mean, it really shouldn't take tens of seconds for those initialization(s) to occur. There's no good fundamental reason that it should take that long. It's just bloat.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#295

Post by andsoitis »

MSFT_Edging wrote: post #144
dlisboa wrote: post #11

Earlier quoted context omitted.

History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years. In many senses there's hubris in the western* view of China accomplishments: most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the name…

The whole "western" or "the west" always makes me laugh. Half the time it's a dog whistle for "white". Like many right-wing commentators love saying "Western Values" to avoid saying "white, Euro-centric, Christian values". Mexico is a modern country, an industrialized country, a country that is exactly as "western" as the US or Canada. They have the same religious beliefs, speak a dialect of a European language. They…

> Yet they're not white enough to be part of "The West".

In many contexts, Mexico and other LatAm countries are included in the Western Civilization grouping. For instance: https://worldpopulationreview.com/country-rankings/western-c...

Earlier in your comment you say “half the time” while you end with “90% of the time” the phrase “western” is a racist expression, undermining your argument that is already flawed, emotional, and anti-constructive.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#296

Post by cnr »

miki123211 wrote: post #271
bee_rider wrote: post #154

Earlier quoted context omitted.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a du…

Let's say, then, that it's not so much "dumb and wasteful" as "energy inefficient". In fact, this can be quite wise in a modern world full of surveillance-as-a-business and "us-east-1 disasters"

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#297

Post by philippejara »

myth_drannon wrote: post #201
g8oz wrote: post #64

Earlier quoted context omitted.

You are mistaken about American intellectual property theft. They engaged in extensive IP theft from Britain in the 18th and 19th century with the encouragement of the government. See https://interestingengineering.com/innovation/the-spies-who-... Also during World War I the American government seized German chemical patents thereby launching the American chemical industry. So that is an example of theft by the state…

I'm talking about modern times, after WW2. Not when US had slavery and lynching, blacks was a form of entertainment.

After it took over and solidified itself as the world leader at it of course there was no need to steal IP anymore, most of it is registered there already and to the benefit of their nation. I'm sure people a hundred years from now will be having similar conversations if china or someone else takes the us's place and naturally starts caring about IP(and the runner ups stop caring).

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#298

Post by OJFord »

cnr wrote: post #287
miki123211 wrote: post #271

Earlier quoted context omitted.

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors. If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet. If that kind of latency isn't acceptable to you, you have to keep the models loaded. This (along with batching) is why large local models are a du…

Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?

I believe GP means it still to be connected to 'if this kind of latency is unacceptable to you' - i.e. you can't load/use/unload, you have to keep it in RAM all the time.

In that case it's massively increasing your memory requirement not just to the peak the model needs, but to + whatever the other biggest use might be that'll be inherently concurrent with it.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#299

Post by wagwang »

dontlaugh wrote: post #276
wagwang wrote: post #206

Earlier quoted context omitted.

This world view is just wrong from top to bottom. Between 1945-1970, China was in a much worser state that any of those 3rd world countries in many aspects. Are you saying post 1970, the reason why these countries did not develop as fast as china is because of colonialism and intervention? You can apply the same arguments to Japan in 45 and Korea in 50. But but but aid? Ok and we send billions to countries all around…

Yes. Imperialism (as a system of extracting wealth from poor countries) continues to exist, but China has a working countermeasure. You can see similar with Vietnam.

Sure if you water down the term of imperialism then yes. It is then incumbent on you to prove "imperialism" among the worst performers economically. What imperialism plagues liberia and ethiopia and honduras and papua new guinea

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#300

Post by wagwang »

Cyph0n wrote: post #207
wagwang wrote: post #206

Earlier quoted context omitted.

This world view is just wrong from top to bottom. Between 1945-1970, China was in a much worser state that any of those 3rd world countries in many aspects. Are you saying post 1970, the reason why these countries did not develop as fast as china is because of colonialism and intervention? You can apply the same arguments to Japan in 45 and Korea in 50. But but but aid? Ok and we send billions to countries all around…

But I just pointed out that China is an exception? Also, China had access to human capital at a scale that no African country had. Japan pre-45 was a world power, and had industrialized by the early 1900s. WW2 was a mere setback. Korea is more of a “miracle” than Japan was, but they also did well to industrialize ASAP. They also didn’t face the brunt of European colonialism.

Lots of miracles and exceptions in a working theory
Post reply on HN