"I am not sure how many people will run AI models locally. It still seems like a niche application to me. However, it will make decent machines to play video games." I don't know who will be the winner but with some of the recent releases from gemma it seems more probable that you may run some models locally if only from a cost perspective, not even considering business security. Not sure how this type of architectur…
I also don't get why this twitter user is linked here, versus all the news articles about this new hardware that have been everywhere over the past number of days.
Performances of local models are pretty bad compared to what AI vendors offer, token generation is just too slow to be that useful. And you need to allocate GBs of memories, something that will stay very expensive to buy for a long time. Running local models will stay niche for a while, unless we see breakthroughs
Dumb idea --- how about if we limit local models to specific domains --- medicine for example. Most doctors don't care much about engineering or accounting or software development or 10000 other things that big vendor models address. This area is yet to be really explored. Nvidia aims to provide the hardware to do so.
That's a fairly obvious idea, not dumb at all, but unfortunately it doesn't seem to pan out. Trying to specialize an LLM in one area harms its 'cognition' in all areas. For instance, if you train a coding model without all the Shakespeare and soap operas and Wikipedia and pirated Stephen King books and ancient Roman history and whatever, you end up with a worse coding model.
The Unified Memory pool is what will continue to be the “game changer” in systems architecture, especially outside of data centers. The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory. Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. A unifie…
yeah, you only see double digits in performance degradation from going from pcie 5 to 3 with a 5090 (at x16 speed), with everything else its like in the single digits area.
The Qualcomm Snapdragon X2 Elite Extreme trounces Nvidia's chip in single core CPU performance. It beats Intel and AMD's best, too. It has unified memory. It's the only CPU in the same league as Apple's M-series in both CPU performance and power efficiency. And it's available in laptops today, not later this year. People are sleeping on Qualcomm.
Microsoft is sleeping on Qualcomm with their lousy port of Windows to Arm processors…
I'm not sure they are sleeping. I have an older version and it can run games and other things just fine, its just over priced and not properly cooled. The driver/firmware support from Lenovo / Qualcomm is purely garbage. You're lucky to get a driver update to fix anything. For months it just overheated and video would start corrupting but that got fixed finally. You cant just go to Qualcomm's website and download new drivers even though it looks like you can - they really dont get how modern GPU's work on Windows - a driver updates to optimize for games is really something important because of how Windows is but the experience is pulling teeth. If the systems were Neo priced (500-700 USD) and had a cooling fan I'd be all on board with these systems. Right now, AMD with unified memory is just the better deal for the $1200 (2025) systems to run Windows and an average workload.
The Unified Memory pool is what will continue to be the “game changer” in systems architecture, especially outside of data centers. The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory. Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. A unifie…
Unified memory is only a feature because NVidia so aggressively uses VRAM for market segmentation. The 5090 ($2k MSRP but realistically $3-3.5k) is almost the same as the RTX 6000 Pro (~$10k). Same memory bandwidth (1800GB/s). Slightly different CUDA cores (21k vs 24k). Big difference? VRAM (32GB vs 96GB). NVidia ultimately doesn't want to upset this segmentation so the RTX Spark will never undermine their other offe…
> 5090 ($2k MSRP but realistically $3-3.5k)
These days, more like >$4.1K (at least in the US).
> you may run some models locally if only from a cost perspective I have a hard time believing running a model on a laptop will be cheaper than running it in a datacenter. Why wouldn't economies of scale apply here as with every other computation?
Because economy of scale isn't really the right metric here. A machine you were you were going to buy anyway essentially has a TCO of $0.
AI models will pretty undeniably affect your electricity bill; yes you already own the computer, but it will cost more to run it if it's doing inference!
Seems like not? Judging based on https://github.com/qualcomm-linux something is happening, although I can't say how much. They definitively seem awake at least.
one of the biggest issue i see is the devicetree nonsense. It makes every single laptop and bios version very unique and requires a lot of housekeeping. There are also big chunks of work (as i understand it) to be done around hibernate and decent suspend support. My experience (wanted to use x13s as daily sriver) is that there was good progress for about a year, until jhovold was leading the charge, but something exp…
It feels deeply unfortunate that even with Windows on AArch64 requiring ACPI that it still doesn’t suffice for Linux, unlike on x86.
And I know a lot of that lies on the vendors, but it does feel unfortunate (from a standardisation/conformance/certification point of view) that Windows requiring it doesn’t make it easy to boot other OSes!
The Unified Memory pool is what will continue to be the “game changer” in systems architecture, especially outside of data centers. The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory. Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. A unifie…
Unified memory is only a feature because NVidia so aggressively uses VRAM for market segmentation. The 5090 ($2k MSRP but realistically $3-3.5k) is almost the same as the RTX 6000 Pro (~$10k). Same memory bandwidth (1800GB/s). Slightly different CUDA cores (21k vs 24k). Big difference? VRAM (32GB vs 96GB). NVidia ultimately doesn't want to upset this segmentation so the RTX Spark will never undermine their other offe…
I have so many questions… Since Apple already sells unified memory systems, what is the market opportunity you envision? Do you see Nvidia and Apple as competitors, and how? (And I’m not suggesting they’re not, necessarily, but I want to hear where you’re coming from, and they do have very different markets.) Hasn’t Apple used storage size (RAM & disk) for market segmentation for decades? And how does a machine with 128GB unified mem not potentially cut into some people’s reasons for wanting a 96GB GPU?
The Qualcomm Snapdragon X2 Elite Extreme trounces Nvidia's chip in single core CPU performance. It beats Intel and AMD's best, too. It has unified memory. It's the only CPU in the same league as Apple's M-series in both CPU performance and power efficiency. And it's available in laptops today, not later this year. People are sleeping on Qualcomm.
> X2 Elite Extreme
I'll wait for the 365 AI Ultimate Professional Enterprise Edition: Origins version