Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

tomshardware.com

Post reply on HN
Print view View original

271–280 of 331 posts

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#271

Post by miki123211 »

bee_rider wrote: post #154
kilotaras wrote: post #47

Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found Instead of 1192 GPUs they now use 213 for serving those requests.

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking…

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors.

If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet.

If that kind of latency isn't acceptable to you, you have to keep the models loaded.

This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#272

Post by wasmitnetzen »

Alupis wrote: post #244
lmz wrote: post #240

Earlier quoted context omitted.

Much higher standards than European ones? Because the cars do sell there as well.

Yes. The EU focuses on safety of pedestrians, and the US focuses on safety of occupants. That's not to say a vehicle cannot do both well (see the many European vehicles sold in the US), but that is to say Chinese vehicles may not meet the US standards. The US has a lot of vehicle regulations that significantly differ from the EU market.

That's not true, it's just that the Chinese NCAP is modeled after the EU one: https://en.wikipedia.org/wiki/C-NCAP

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#273

Post by audunw »

notepad0x90 wrote: post #143
rayiner wrote: post #65

Earlier quoted context omitted.

> But now the US is trying to … compete against 10x population and lack of similar levels of internal strife and fissures. I can’t tell whether you think the anti-immigration stance is a good thing or bad thing.

it's bad for the US, because China has 10x population. the US can't make up in quality, what it lacks in quantity without immigration and attracting foreigners.

I don’t think you realise how many people in China still live in poverty, with not much prospects of improvement. I don’t see how having 100 million poor small scale farmers is a benefit in this equation.

You still can’t become a Chinese citizen. You can come to USA or Europe and build a life for yourself. While some people go to China to make some money for a few years you can’t really build a life. So I think US and Europe will still attract talent long term, and I don’t think you can discount that. China used to have the benefit of low cost labor, but that’s going away. What do they have to offer when that’s gone?

Chinas population isn’t 10x. It’s 4x. If you believe the numbers (the idea that local governments over report is not a fringe theory).

But it’s really only the wealthy coastal regions that matters in this comparison, and in that regard the population sizes are much closer. Yeah they can exploit cheap labor from the poor interior. But the US is doing something similar in some ways with central/southern America. The Hukou system means that China does act like a bunch of separate states in many regards, rather than one truly unified country.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#275

Post by shakow »

Cyph0n wrote: post #207
wagwang wrote: post #206

Earlier quoted context omitted.

This world view is just wrong from top to bottom. Between 1945-1970, China was in a much worser state that any of those 3rd world countries in many aspects. Are you saying post 1970, the reason why these countries did not develop as fast as china is because of colonialism and intervention? You can apply the same arguments to Japan in 45 and Korea in 50. But but but aid? Ok and we send billions to countries all around…

But I just pointed out that China is an exception? Also, China had access to human capital at a scale that no African country had. Japan pre-45 was a world power, and had industrialized by the early 1900s. WW2 was a mere setback. Korea is more of a “miracle” than Japan was, but they also did well to industrialize ASAP. They also didn’t face the brunt of European colonialism.

> They also didn’t face the brunt of European colonialism.

But they faced the brunt of Chinese & Japanese colonialism, and a full-blown civil war after which their GDP per capita was in the same ballpark as Kenya's.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#276

Post by dontlaugh »

wagwang wrote: post #206
Cyph0n wrote: post #202

Earlier quoted context omitted.

> Otherwise Nigeria or Indonesia would be scientific powerhouses, and Iran would have modern fighter jets. The reason why these statements are not true is because of colonization, delayed industrialization, and Western intervention post-independence. Getting out of this “quicksand” is exceedingly difficult. China did well to industrialize quickly and keep intervention at bay - in fact, you could argue that it making…

This world view is just wrong from top to bottom. Between 1945-1970, China was in a much worser state that any of those 3rd world countries in many aspects. Are you saying post 1970, the reason why these countries did not develop as fast as china is because of colonialism and intervention? You can apply the same arguments to Japan in 45 and Korea in 50. But but but aid? Ok and we send billions to countries all around…

Yes.

Imperialism (as a system of extracting wealth from poor countries) continues to exist, but China has a working countermeasure. You can see similar with Vietnam.

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#277

Post by dontlaugh »

markdown wrote: post #220
wagwang wrote: post #206

Earlier quoted context omitted.

This world view is just wrong from top to bottom. Between 1945-1970, China was in a much worser state that any of those 3rd world countries in many aspects. Are you saying post 1970, the reason why these countries did not develop as fast as china is because of colonialism and intervention? You can apply the same arguments to Japan in 45 and Korea in 50. But but but aid? Ok and we send billions to countries all around…

Aid is the single largest barrier to development and progress over the long term.

“Those who come with wheat, millet, corn or milk, they are not helping us. Those who really want to help us can give us ploughs, tractors, fertilizers, insecticides, watering cans, drills and dams. That is how we would define food aid.” ― Thomas Sankara

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#278

Post by HPsquared »

wobfan wrote: post #259
Alupis wrote: post #244

Earlier quoted context omitted.

Yes. The EU focuses on safety of pedestrians, and the US focuses on safety of occupants. That's not to say a vehicle cannot do both well (see the many European vehicles sold in the US), but that is to say Chinese vehicles may not meet the US standards. The US has a lot of vehicle regulations that significantly differ from the EU market.

Do you mind providing sources for that? Because, as a EU citizen, I have never in my life seen any tests that carmarkers are advertising with that focus on pedestrians. I am regularly seeing tests that focus on occupants though, e.g. the Euro NCAP. But I am by no means an expert. It would be hard to focus on pedestrian safety from a carmaker standpoint except for adding software features that recognize people in fron…

Pedestrian impact design has been a thing in the European market for a while. One example: the pop-up hood that detects a pedestrian impact, it's a pyrotechnic device that makes the hood pop up to give more of a cushion for the pedestrian to land on. Also I think pedestrian safety is one reason modern BMWs and Audis have the timing chains (very annoyingly for maintenance) fitted at the back of the engine, to give more space for pedestrian impact. Stuff like that. Also the automated emergency braking systems, more recently.

https://www.euroncap.com/en/car-safety/the-ratings-explained...

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#279

Post by aleph_minus_one »

deadbabe wrote: post #152
hshdhdhj4444 wrote: post #110

Earlier quoted context omitted.

Ironically, the best way America could have prevented China’s rise in tech was by stapling green cards to diplomas of Chinese citizens who completed their higher education in the U.S. like the plan in the early 2010s.

But they didn’t do it, because the current administration can’t get it through their thick skulls that the key advantage the US can have in this world is a monopoly on all the really smart people.

> [T]he key advantage the US can have in this world is a monopoly on all the really smart people.

Dream on ...

Re: Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

#280

Post by antonvs »

ahmeneeroe-v2 wrote: post #160
brookst wrote: post #131

Earlier quoted context omitted.

The US has become unreliable and erratic. Countries aren’t cutting ties or anything, but certainly investing to reduce exposure to capricious US leadership. Much of Europe is increasing domestic military production rather than just buying more from the US precisely because the US has publicly discussed leaving NATO and/or not honoring its guarantees.

Increasing domestic military production is actually a great outcome. Obviously Wall Street would have preferred purchasing from US-listed/owned arms companies, but from the perspective of a military alliance, having well-armed allies is the main point. It's really hard to argue with Trump's methods if they led to Europe finally spending on their own defense.

> It's really hard to argue with Trump's methods

On the contrary, his methods are ridiculous. He could have achieved similar ends without kowtowing to Russia, without squandering the opportunity to further weaken Russia’s military capacity, and so on. Something similar applies to pretty much everything else he’s done. He incurs collateral damage on everything even when it’s completely unnecessary to do so. It’s a definitional example of egregious incompetence.

Post reply on HN