>The AMD 395+ uses unified memory, like Apple, so nearly all of it is addressable to be used by the GPU. This is why they went with the “laptop” cpu. While it’s slightly slower than dedicated memory, it allows you to run the big models, at decent token speeds.
Geerling benchmarked LLM performance on the Framework Desktop and the results look pretty lackluster to me. First, the software seems really immature. He couldn't get ROCm or the NPU working. When he finally got the iGPU working with Vulkan, he could only generate 5 tok/sec with Llama 3.1 70b (40 GB model). That's intolerably slow for anything interactive like coding or chatting imo, but I suppose that's a matter of…
The Framework Desktop is a beast
441–450 of 464 posts
Re: The Framework Desktop is a beast
#442Earlier quoted context omitted.
Apple was just the highest bidder for getting the latest TSMC process. They wouldn't have had a problem getting other customers to buy up that capacity. And Intel's missteps counted for a substantial part of the process dominance you refer to. So I'd argue that Apple isn't that special here either.
Until Apple forced other chip makers to respond, nobody else was making high end phone processors. And their A series processors are competitive with and have transistor counts comparable to most mobile and desktop PC processors (and have for years). So the alternate reality where Apple isn't a TSMC customer means that TSMC is no longer manufacturing several hundred million high transistor count processors per year.…
Re: The Framework Desktop is a beast
#443Earlier quoted context omitted.
Apple base models tend to be fairly competitive but they have some of the most extreme margins on RAM and SSDs in the industry. They charge $600 CAD to go from 16GB -> 32 GB. They charge $900 CAD for 512 GB -> 2 TB SSD.
> They charge $900 CAD for 512 GB -> 2 TB SSD. The SSD is user replaceable, so you can replace it with a cheaper third party option.
Re: The Framework Desktop is a beast
#444Earlier quoted context omitted.
Yes. Signal integrity is so finicky at frequencies DRAM operates that whether you drill the plated holes on boards that complete the circuit to go completely through the board or stop it halfway starts to matter due to signals permeating into the stubs of the holes and reflecting back into the trace causing interference. Adding a connector between RAM and CPU is like extending that long pole in the tent in the middle…
In the past, at least, RAM upgrades weren't just about filling in the slots you couldn't afford to fill on day one. RAM modules also got denser and faster over time too. This meant you could add more and better RAM to your system after waiting a couple years than it was even physically possible to install upfront. Part of the reason I have doubts about the physical necessity here is because PCI Express (x16) is rough…
But in general yes, PCIe vs RAM bandwidth is like comparing apples to watermelons. One’s bigger than the other and they’re both fruits, but they’re not the same thing.
Generally people don’t talk about random-access PCIe latency because it generally doesn’t matter. You’re looking at a best-case 3x latency penalty for PCIe vs RAM, usually more like an order of magnitude or more. PCIe is really designed for maximum throughput, not minimum latency. If you make the same tradeoffs with RAM you can start tipping the scale the other way - but people really care about random access latency in RAM (almost like it’s in the name) so that generally doesn’t happen outside of specific scenarios. 500ns 16000MT/s RAM won’t sell (and would be a massive pain - you’d probably need to 1.5x bus width to achieve that, which means more pins on the CPU, which means larger packages, which means more motherboard real estate taken and more trace length/signal integrity concerns, and you’d need to somehow convince everyone to use your new larger DIMM...).
You can also add more memory channels to effective double/quadruple/sextuple memory bandwidth, but again, package constraints + signal integrity increases costs substantially. My threadripper pro system does ~340GB/s and ~65ns latency (real world) with 8 memory channels - but the die is huge, CPUs are expensive as hell, and motherboards are also expensive as hell. And for the first ~9 months after release the motherboards all struggled heavily with various RAM configurations.
Re: The Framework Desktop is a beast
#445Okay so why is the ram non upgradable again? Because this was the main reason I was holding to PC architecture (I'm considering non upgradable ram as a method of upselling for a ridiculous price of higher amounts)
I assume it's related to the following: > The AMD 395+ uses unified memory, like Apple, so nearly all of it is addressable to be used by the GPU. There's probably some fairly strict timing requirements etc on that. Still it'd be nice if they figured out a way to have swappable RAM.
Re: The Framework Desktop is a beast
#446Earlier quoted context omitted.
Isn't this the desktop architecture that Torvalds suggested years ago?
Faster and bigger SRAM cache is as complicated of a solution as adding moar boosters to your rocket. It works, but expensive. RP2040 uses ~8x more die space as its dual CPU just for the RAM.
Re: The Framework Desktop is a beast
#447Earlier quoted context omitted.
I'm not sure, but you could always just buy the Minisforum BD790i X3D [1]. Then, you'd be able to choose your own fan and case, and you can make it arbitrarily quiet by selecting a good fan. Early BD790i boards had a bug where losing power causes it to reset all BIOS settings, but they fixed that in later batches. I wonder when they will come out with a 9955HX version. Another good thing about this board is that it h…
I did exactly this, put it in a 2U case. Fantastic performance, but even with the best Noctua I could put on it the CPU fan sounds like a Hawker Harrier doing VTOL when it's under load. Don't regret the board, but wish my rack was in another room now.
Re: The Framework Desktop is a beast
#448Earlier quoted context omitted.
from the link: > Apple integrates the storage controllers into the Apple Silicon SoC, which means that normal M.2 SSDs are incompatible.
Nevertheless, third parties have created compatible replacements that are much cheaper. They just don't need to include a SSD controller, since that is already built into the SOC.
iMacs and macbook pros on the other hand are significantly more difficult. Apple could easily rectify this but choose not to, so that you are more likely to buy their overpriced upsells on the initial purchase.
Re: The Framework Desktop is a beast
#449Earlier quoted context omitted.
Except they keep making it harder and harder to install your own drives into apple machines.
You don’t install it but run over Thunderbolt which is plenty fast.
Re: The Framework Desktop is a beast
#450Earlier quoted context omitted.
Except they keep making it harder and harder to install your own drives into apple machines.
Thunderbolt 5 is plenty fast for an external SSD. In fact you'd be hard-pressed to find an external SSD that even goes that fast
External drives are not a replacement for internal drives. I have a laptop for a reason, and on crowded trains and planes needing some desk space to plug in dongles, huibs, and external drives just doesnt cut it.