51–60 of 98 posts
Earlier quoted context omitted.
First crypto then AI, I wish GPUs were left alone for gaming.
Didn’t nvidia try to block this in software by slowing down mining? Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.
Earlier quoted context omitted.
I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.
The universe used to have hierarchies. Fast memory close, slow memory far. Registers. L1. L2. L3. RAM. Swap. The same thing would make a lot of sense here. Super-fast memory close, with overflow into classic DDR slots. As a footnote, going parallel also helps. 8 sticks of RAM at 1/8 the bandwidth each is the same as one stick of RAM at 8x the bandwidth, if you don't multiplex onto the same traces.
The issue is that GPUs are massively parallel. A 4090 has 128 streaming multiprocessors, each executing 128 "threads" or "lanes" in parallel. If each "thread" works on a different part of memory that leaves you with 1kB of L1 cache per thread, and 4.5kB of L2 cache each. For each clock cycle you might be issuing thousands of request to your memory controller for cache misses and prefetching. That's why you want insanely fast RAM.
You can write CUDA code that directly accesses your host memory as a layer beyond that, but usually you want to transfer that data in bigger chunks. You probably could make a card that adds DDR4 slots as an additional level of hierarchy. It's the kind of weird stuff Intel might do (the Phi had some interesting memory layout ideas).
Earlier quoted context omitted.
No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…
Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB. Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.
Earlier quoted context omitted.
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inferen…
>24GB is fairly limiting
Can I take a moment to suggest that maybe we're very spoiled?
24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"?
To think Bill once said 640KB would be enough.
The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
Can you elaborate on this? Intel's reputation for software support hasn't been stellar, what's changed?
Earlier quoted context omitted.
Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it w…
How comes you don't care about full bandwidth?
Earlier quoted context omitted.
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
I tend to agree that it would be niche. The machine learning enthusiast market is far smaller than the gamer market. But selling to machine learning enthusiasts is not a bad place to be. A lot of these enthusiasts are going to go on to work at places that are deploying enterprise AI at scale. Right now, almost all of their experience is CUDA and they're likely to recommend hardware they're familiar with. By making co…
Earlier quoted context omitted.
No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
It doesn't even matter if that's your primary goal or not.
Earlier quoted context omitted.
I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inferen…
>models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.
The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run.
The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market.
I mean inequality is definitely increasing at a worrying rate these days, but let's keep the discussion on topic...
Earlier quoted context omitted.
No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge…
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
If Intel sells a stackable kit with a lot of RAM and a reasonable interconnect a lot of corporate customers will buy. It doesn't even have to be that good, just half way between PCIe 5.0 and NVLink.
But it seems they are still too stuck in their old ways. I wouldn't count on them waking up. Nor AMD. It's sad.