71–80 of 117 posts
Woof. What's that going to cost? And will we actually be able to buy one, because the 128GB model has been sold out pretty much since the LLM craze started...
It's competition is M5 Ultra, not cheap.
Non-ECC memory. And into the trash it goes. Refer to https://news.ycombinator.com/item?id=48969530
What would you say is the likelihood of an LLM hallucination vs. a correctable RAM error?
Woof. What's that going to cost? And will we actually be able to buy one, because the 128GB model has been sold out pretty much since the LLM craze started...
It's competition is M5 Ultra, not cheap.
Tl;dr: think hard about what you’d use this much RAM for in a desktop setting and do some research about how people like the 395 for your use case. Not all use cases work well. Anyone who thinks they are going to serve some 100+ GB LLM locally, remember that memory bandwidth becomes a key limitation for large models. While you might be able to load a model, token generation can be very slow. MoE models like Qwen3.5-1…
Lemonade server updated yesterday for an Arena model / ai routing.
---
The Lemonade local AI server allows easily deploying LLM workloads across Ryzen AI NPUs, Radeon GPUs, and CPUs with ease under both Windows and Linux. The most significant change with Lemonade 11.5 is the completion of the Lemonade Router that can be used for automatically routing queries to relevant models based on defined policies -- or even LLM-as-a-router for using a small LLM to in turn determine which model to route a particular request.
The Lemonade Router allows steering requests based on rule, classifier, semantic similarity, or LLM-as-router policies. The policies are defined in JSON files or can also be authored via the Lemonade GUI. More details on these router capabilities can be found via this commit.
Lemonade 11.5 also introduces a server-side job engine to let clients post multi-step recipes and managing them via /jobs endpoints. From these endpoints the muilti-step recipes can be paused / interrupted / resumed / deleted. And another one is that the lemond daemon can now act as an MCP client host to connect external stdio MCP servers.
I understand their reasoning but it’s still a shame Framework opted for a proprietary motherboard instead of mini-ITX with a socket CPU and GPU. I value repairability over a little bit of extra performance.
Based on the product spec sheet[1] it appears that the Framework Desktop motherboard is mini-ITX, and that the chassis is[2] as well. It’s fully possible that I’ve missed something, but it doesn’t appear to be proprietary to me. [1]: https://frame.work/desktop?tab=specs [2]: https://frame.work/products/desktop-case
I understand their reasoning but it’s still a shame Framework opted for a proprietary motherboard instead of mini-ITX with a socket CPU and GPU. I value repairability over a little bit of extra performance.
IIRC, Framework did ask AMD to explore a more repairable approach with socketable CPU and RAM for these chips and they did but came back and said it wouldn't perform well enough that way.
Non-ECC memory. And into the trash it goes. Refer to https://news.ycombinator.com/item?id=48969530
the thing about ECC memory is that it doesn't support reporting of ECC errors, so it ends up just hiding the fact that your RAM has started to go bad. this is good if you replace RAM every n years, but not if you intend to use the RAM until it starts degrading.
Wait that. It does, and that's the point. Although the CPU and motherboard do of course also have to support it, but with AMD it is possible. I do it on a 9800x3d.
One bit errors are always both detected and corrected. Multibit errors are mostly at least detected.
I understand their reasoning but it’s still a shame Framework opted for a proprietary motherboard instead of mini-ITX with a socket CPU and GPU. I value repairability over a little bit of extra performance.
What do you mean? The whole point is that these APUs present a pretty unique value proposition (lowish performance GPUs with massive amounts of RAM attached), and framework is afaik the only vendor shipping them on standard mini-ITX boards
Non-ECC memory. And into the trash it goes. Refer to https://news.ycombinator.com/item?id=48969530
Hear me out... What would you say is the likelihood of an LLM hallucination vs. a correctable RAM error?
This is about the latter, and not just the correctable ones; Detecting them also has worth.
I understand their reasoning but it’s still a shame Framework opted for a proprietary motherboard instead of mini-ITX with a socket CPU and GPU. I value repairability over a little bit of extra performance.
There is no socketable Unified Memory platform. The entire point of this SoC is that it has Unified Memory. The Framework Desktop has a Mini-ITX board, but you MUST use a solderable SoC with HBM stacked memory to get a Unified Memory outcome. For local AI workloads, that's the only part that matters, which is the entire point of wanting 192GB of RAM available on an APU. I like the repairability and modularity of my F…