Intel's basic architecture keeps accelerators away from main system memory, unlike, for example, IBM's POWER architecture where the CPU and GPU are equal 'users' of memory. It's not a great breakthrough to suggest something different. The problem is - it's different, and not compatible with a lot, or most, or all, existing hardware. Also, there are some security concerns, as @stego-tech noted.
Nvidia is proposing a beast of a CPU system for Windows PCs
381–390 of 581 posts
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#382Earlier quoted context omitted.
To this day I do not get why Intel doesn't just offer massive memory options for their cards. Just charge what it costs to add the extra memory, no upcharge, and they will never be able to keep up with demand. Cheap VRAM is enough to justify a lot of open source investment into challenging CUDA.
> To this day I do not get why Intel doesn't just offer massive memory options for their cards. They seem to? Intel Arc is the cheapest option by far for a discrete card with 32GB VRAM.
Missed a zero here.
Needs 320 GB Vram
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#383The Unified Memory pool is what will continue to be the “game changer” in systems architecture, especially outside of data centers. The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory. Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. A unifie…
And here I am with 128GB Strix Halo longingly eyeing the Blackwell cards that spit tokens 10-20x the speed. The question is ultimate shape of knowledge compression and bandwidth optimization at which we arrive I suppose.
If you haven't already, check/increase the GPU memory carve-out on your UEFI.
More details: https://rocm.docs.amd.com/en/docs-7.2.0/how-to/system-optimi...
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#384Earlier quoted context omitted.
because nas hardware and software suck and everything else was a poorly executed subscription product...i think one was called helm, another was by early twitter alumni. imagine a home device that manages and maintains itself and is a joy to interact with.
And why would the hypothetical "OwnAI" product be any different?
not automatically, but a meaningful step up in ease of use (managing photo/video backup from all family devices) without a subscription would be a solid foundation
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#385Earlier quoted context omitted.
Unified memory is only a feature because NVidia so aggressively uses VRAM for market segmentation. The 5090 ($2k MSRP but realistically $3-3.5k) is almost the same as the RTX 6000 Pro (~$10k). Same memory bandwidth (1800GB/s). Slightly different CUDA cores (21k vs 24k). Big difference? VRAM (32GB vs 96GB). NVidia ultimately doesn't want to upset this segmentation so the RTX Spark will never undermine their other offe…
To this day I do not get why Intel doesn't just offer massive memory options for their cards. Just charge what it costs to add the extra memory, no upcharge, and they will never be able to keep up with demand. Cheap VRAM is enough to justify a lot of open source investment into challenging CUDA.
Memory is just one part. AMD has had offerings competitive to NVIDIA for quite some time, but nobody uses AMD cards.
The biggest advantage with NVIDIA is CUDA.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#386Earlier quoted context omitted.
Deepseek V4 Flash still has 13B active params though? That is about half as many as Qwen3.6-27B (and much more than Qwen3.6-35B-A3B). Given that RAM (even on a base M4 or 'regular' Intel/AMD system) is like an order of magnitude faster than an SSD, even Qwen 27B running from RAM will be much faster than any Deepseek V4 model with SSD offloading. And the MoE will be much faster still. Qwen 27B is also small enough to…
> even Qwen 27B running from RAM will be much faster than any Deepseek V4 model with SSD offloading. If you have reasonable amounts of RAM to cache the most likely experts, that's not true at all. Qwen 27B is marginally faster on a nearly empty context, then falls behind as context length increases due to the different attention mechanisms. Prefill for Qwen is much faster, but you're still comparing vastly different…
> the most likely experts
Is that how MoEs work? I though that an important constraint for MoEs is that experts need to be uniformly used to make sure they can be used effectively. If there is a 'common subset' that, if anything, sounds like a symptom of undertraining (i.e. the same trick will not work as well for Deepseek V4.1).
Also, even if your MoE hitrate is 90%, you still spend half your time waiting for the SSD, giving similar total speed to a 27B model!
Finally, it looks like Deepseek V4 is pretty much only runnable with antirez's ds4, and SSD streaming only works with Metal; but I would like to try what you say with llama.cpp which uses mmap to also potentially do SSD streaming. (I can maybe try the large Qwen3.5 MoEs?)
> as context length increases
What kind of context length do you consider reasonable, though? From what I know, all models (even frontier ones) start degrading once you pass a few hundred thousand tokens. So realistically, limiting context size might even improve quality, especially if you use token-efficient harnesses.
> Or you could fit the dense portion of a much more capable model and still take advantage of that hardware.
Your point about consumer hardware was that it would be "borderline unusable" when running Qwen 3.6 27B. However, you need much less hardware to run a 27B than DSv4 Flash. In addition, you can do the same 'trick' with low-end GPUs and small MoEs: my desktop with 32 GB DDR4-3200 and an RTX 2070 8GB can run the ~30B class MoEs at 20-30 tokens per second and similar speeds to my laptop.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#387Earlier quoted context omitted.
> The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory Game dev here. For anyone reading this - it’s not because we’re lazy, it’s because _it’s really hard to do_. One of the biggest differences between the current generation consoles and the current gen PCs is unified memory.
How much of that difficulty comes from the chosen game engine? I assume the engine is the primary factor in how resources are allocated.
One related question that you need to follow that with is the associated costs of switching the whole studio to another engine that's technically better, or if proposing teach studio tailor-make their own engine the costs of that engineering, if presumably they have or learn the expertise to surpass whatever they're using currently.
I'm not a game developer, but it would also seem to be a link between resource usage by the engine, and whatever content the production side are making. For all the commentary about how brilliant the id software engines are, if you examine the levels you pass through they're also very efficient with what they demand out of the engine - it's like an orchestra playing well together, not one instrument that means you can do anything.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#388Earlier quoted context omitted.
> even Qwen 27B running from RAM will be much faster than any Deepseek V4 model with SSD offloading. If you have reasonable amounts of RAM to cache the most likely experts, that's not true at all. Qwen 27B is marginally faster on a nearly empty context, then falls behind as context length increases due to the different attention mechanisms. Prefill for Qwen is much faster, but you're still comparing vastly different…
> the most likely experts Is that how MoEs work? I though that an important constraint for MoEs is that experts need to be uniformly used to make sure they can be used effectively. If there is a 'common subset' that, if anything, sounds like a symptom of undertraining (i.e. the same trick will not work as well for Deepseek V4.1). Also, even if your MoE hitrate is 90%, you still spend half your time waiting for the SS…
> Is that how MoEs work?
For any given workload/session? Empirically, yes, that's what has been found across different models. There's quite a bit of predictability that makes caching helpful.
> Also, even if your MoE hitrate is 90%, you still spend half your time waiting for the SSD, giving similar total speed to a 27B model!
There are ways of masking some of that latency, though it requires some architecture-specific cleverness which is less directly applicable to a generic engine like llama.cpp.
> Finally, it looks like Deepseek V4 is pretty much only runnable with antirez's ds4, and SSD streaming only works with Metal
The llama.cpp folks are working on adding support, and the ds4 project is working on CUDA support for streaming inference, targeting the DGX Spark.
> From what I know, all models (even frontier ones) start degrading once you pass a few hundred thousand tokens.
DeepSeek V4 seems to do quite well on recall tasks even with large context. That's one plausible benefit of its compressed attention mechanism, compared to earlier models. Some degradation will likely still be there, but it's not necessarily obvious.
As for why people are calling Qwen 27B "borderline unusable" that may have to do with it being a dense model which makes for an increased compute intensity and pushes users towards discrete GPU platforms, since those tend to have the most compute overall as far as consumer hardware is concerned. I might agree that Qwen 27B is quite ideally tailored towards these platforms, but that does come with some limitations.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#389Earlier quoted context omitted.
yes, here is 2013 AMD presentation of the topic as example: https://events.csdn.net/AMD/GPUSat%20-%20hUMA_june-public.pd... see slide 14 especially
Ah. Well, what kind of consumer hardware/software combo could I purchase to use this? outside of perhaps the... PS4?
Everything that doesn't have a discrete GPU has unified memory these days. If you're asking for something closer to the RTX Spark or Apple Silicon then look at AMD's Strix Halo systems.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#390The Unified Memory pool is what will continue to be the “game changer” in systems architecture, especially outside of data centers. The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory. Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. A unifie…
Intel was doing UMA with their i740 graphics in the late 90s. Codename TIMNA was cancelled, but they pioneered it and used it on their you/cpu chips as well as their breakthrough 810 chipset that dominated graphics market for a decade. It was despised because it wa ubiquitous and a low performing graphics engine but games had to accommodate it. Funny that it is getting credit only now.
SGI O2 was the famous "unified memory architecture" graphics system, two years before i740 that didn't really do UMA.
O2 was popular in systems where large textures or textures generated dynamically (like mapping external video input to texture) was important