You've basically build an LRU page cache with readahead for streaming 21, 504 experts off disk.
Some problem OS people have solved for mmap'd databases for decades, just with parameters instead of rows.
Show HN: Getting GLM 5.2 running on my slow computer
201–210 of 269 posts
Re: Show HN: Getting GLM 5.2 running on my slow computer
#202Would love to collaborate
Re: Show HN: Getting GLM 5.2 running on my slow computer
#203"Slow computer" + running a 200B model = "I'll just wait a week."
Re: Show HN: Getting GLM 5.2 running on my slow computer
#204Earlier quoted context omitted.
Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
Re: Show HN: Getting GLM 5.2 running on my slow computer
#205"Slow computer" + running a 200B model = "I'll just wait a week."
Assuming steady 1 tok/second generation (which seems to be the case for M5 Max macbook), wait 1 day for a 86400 token response. In some configurations it can be as slow as 0.1 tok/s, so be prepared to wait for 10 days.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#206Earlier quoted context omitted.
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it. The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload. The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math…
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1]. Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs. [1]: https://www.hetzner.com/dedicated-rootserver/g…
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#207After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…
I've been seeing a lot of usage of the word `real`
from recent fable sessions:
- That gap is the real story: 3,873 flows
- the real conversion filter types are:
- I'll update the breakdown query to include a column for each real type
Re: Show HN: Getting GLM 5.2 running on my slow computer
#208After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…
I've been seeing a lot of usage of the word `real` from recent fable sessions: - That gap is the real story: 3,873 flows - the real conversion filter types are: - I'll update the breakdown query to include a column for each real type
I have almost completely stopped writing real in my own output as a direct consequence of this. Maybe next year they'll catch up with my alternative wordings and I'll end up switching back
Re: Show HN: Getting GLM 5.2 running on my slow computer
#209After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…
I've been seeing a lot of usage of the word `real` from recent fable sessions: - That gap is the real story: 3,873 flows - the real conversion filter types are: - I'll update the breakdown query to include a column for each real type
Unfortunately we're much too late for realnetworks and realplayer to be relevant anymore.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#210Earlier quoted context omitted.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training ... can we tru…
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.