51–60 of 62 posts
"integrated graphics processor, using system memory" had its name dragged through the mud for decades. So we had to rebadge it to "unified memory". Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...
Earlier quoted context omitted.
This. I don't particularly like the LLM writing style, but we've read a ton of very poorly written texts over the year with no complaints. LLMs are average writers with an annoying style, but not bad writers. If the content is good, I don't care if a LLM wrote it.
It’s a flag that no care went into it. Web surfing involves lots of little decisions following cues of “is this worth my time”
Let's also ensure the SSD doesn't age prematurely.
Earlier quoted context omitted.
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
You'd need your OS to support, and be configured to use, a disk mounting option that disables file access timestamps, otherwise reads ARE writes.
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
I might be misremembering though. Perhaps I scavenged some basic pci card, but that should still have been worse than the ProSavage.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
To be fair it's "only" half the throughput of a 4090 and a third of an RTX 6000. Significant but not an order of magnitude.
Uhh the 5090 alone is double the cost of their quoted PC prices.
Earlier quoted context omitted.
You'd need your OS to support, and be configured to use, a disk mounting option that disables file access timestamps, otherwise reads ARE writes.
Do any operating systems update file access time after every read operation instead of just at fopen?
"Can't" is not really correct. Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
This is a very understandable misconception that I wouldn't blame anyone for having but MoE is actually terrible for inference in most any local LLM / home lab scenario. MoE is popular because it's cheap to train, but because most modern routing needs the previous layer's activations (except at the very beginning) it winds up being just this side of impossible to pipeline / prefetch without all the experts resident.…