Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

Would this make cloud providers running low volume fine-tuned models more economically viable?

You're taking things too one-to-one.

Skilled immigrants may not contribute to birth rate much, but immigrants as a whole contribute to the workforce across the spectrum. More working age people means less demand for unskilled labor, more demand for skilled labor and more competition for higher achievements to qualify for skilled work.

There are millions of phd's and super-talented engineers, but it is a small percentage of those that actually innovate and invent new things. And for them to do that, you need a corporate/commercial sector funding it. Even someone flipping burgers at mcdonalds is a consumer contributing to economic activity, which in turn contributes to funding competitive R&D and risk taking.

Simply having lots of people and free schools won't do much on its own. You need R&D funded, you need companies and the government itself to invest in risky scientific endeavors. Highly skilled jobs need to pay well. For example, there is a metric crapton of talent in Europe that flocks to the US for the pay alone, even though most of them hate it here. Even Candian pay across the border is dismal. That's why Europe doesn't have Nvidias, Intels, Googles,etc..

This very site alone belongs to US venture capitalists which are a product of capital available, a pipeline of educated labor domestically as well as immigrants. The products and services companies sell is mostly funded by consumers buying things, they can buy those things because they have jobs that pay well. The guy who flips burgers at mcdonalds buys a nintendo switch, the help desk worker nvidia gpus,etc... if your population is too old, those things don't happen, old people conserve money and their economic activity doesn't go as far.

Have you heard of the vitality curve? It's how in virtually everything involving human contribution, 10-20% carry the "thing" 10-20% are detrimental to it and everyone in between is needed to keep it from crumbling. I believe that's why performance reviews are always in quintiles. Either way, I don't know if the top 10% that give the US an advantage are immigrants, but some of them for sure. and a lot of the papers I'm seeing from the US in recent years have not been from US sounding names. But the middle 60% or so, it doesn't matter where they're from, you need enough people that are skilled and competent to keep the ship afloat.

If all the variables are the same, China has more people so it wins by default. The US however can attract talent from all over the world for the top 10% talent and have them compete. I don't know the stats but let's say 95% of educated people are native born. That still doesn't mean the competition for top jobs is adequate. To compete with China, the US's top 10% talent must have more quality to make up for the lack of quantity. Quality isn't measured by numbers and it isn't a product of random lack you can improve by increasing quantity. it's a product of competition and the incentives and rewards at the end, which includes compensation but more than that - the quality of life money affords.

In other words, whether immigrants are smarter or not, they can either contribute to the economy by being good and reliable consumers and laborers that create more economic activity and drive the demand and opportunities for skilled work, or, they can drive up the compeition for skilled work, driving up quality.

What you have in the US, is a lot of educated people are into things like health care these days, because that's where the demand is. Even immigrants. But in east asia, it's much worse, they do needs lots more health care workers and care givers for the elderly, which even there, they're using more and more immigrants.

The bulk importing of immigrants only serves to stabilize the economy. The importing of educated immigrants and workers (Most of YC would collapse without H-1B lol) drives competition and increases quality (innovation,inventiveness,etc..).

You can have more americans, even have more americans attend more college. But you can't kick out americans that refuse to pursue education or are content with mediocrity. You can filter out immigrants by telling them they don't have enough education or money (we've been doing this for a long time in the US), but you can't do that with natural born americans.

If you work in tech, this should be of no surprise to you.

By perception of the population at least, yes. I mean, the US is literally on the verge of a civil war lol.

  > That's an eternity for a request. I highly doubt they will timeout any model they serve.

That's what easing functions are for.

Let's say 10 GPUs are in use. You keep another 3 with the model loaded. If demand increases slowly you slowly increase your headroom. If demand increases rapidly, you also increase rapidly.

The correct way to do this is more complicated and you should model based on your usage history, but if you have sufficient headroom then very few should be left idle. Remember that these models do requests in batches.

If they don't timeout models, they're throwing money down the drain. Though that wouldn't be uncommon.

I went to a school that was heavy on immigrants and had lots of 1st gen citizens as students and all they did was advocate against people like me for admissions and for preferential admissions for their own group. So in my opinion, skilled immigration is not a transfer of talent but an expansion of the upper classes who go to war with each other over a small number of seats. Ironically this zero sum game keeps overall skill levels the same. For every immigrant, say, one citizen loses a seat somewhere.