311–320 of 328 posts
Earlier quoted context omitted.
Surely for most the desire is just an LLM provider that doesnt store or sell their queries (including by national actors). As long as that is allowed to happen surely its the answer for the vast majority.
> LLM provider that doesnt store or sell their queries > As long as that is allowed to happen It won't be. Only we can provide that, and only for ourselves.
Earlier quoted context omitted.
Yes, I meant that 192GB of RAM even with the worst quantization would result in such a large model going deep into disk swap when it entirely runs out of RAM. At least 100GB worth, if MacOS will even allow that without freezing or crashing or OOM killing the process.
You still have a core misunderstanding. Only one layer of weights is required in memory at a time. A forward pass can be over-simplified as a matrix multiplication of each layer, one at a time. There is no swapping of working RAM. We're just talking about loading the weights read-only data into RAM on-demand for each layer. It is only as slow as your storage interface.
Yes doing that would prevent destruction of an SSD through using disk space as swap RAM, but it will not be a good experience or usable at all. Note that the original post I was replying to referenced "swapping" which is generally meant to mean using system swap space as RAM.
The standard term for loading only portions of a model from disk as needed is memory mapping, not "swapping". https://www.google.com/search?client=firefox-b-d&q=llama-ser... , or same thing if you google "safetensors file memory mapping"
With a model of this large of a size, not being able to hold it in RAM? Even at worst quantization you'd be looking at 1tps or worse.
Earlier quoted context omitted.
> We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted…
tinfoil doesn't seem to distinguish between cached and input tokens. I think that would make it untenably expensive
Correct.
Privatemode correctly distinguish[1][2].
But with Tinfoil it appears to be a fundamental architectural limitation that they cannot.[3]
[1] https://www.privatemode.ai/pricing [2] https://www.privatemode.ai/blog/secure-prompt-caching [3] https://docs.tinfoil.sh/resources/caching
Earlier quoted context omitted.
While my colleagues are running 6 parallel agents at 50-100t/s each, with an actual SOTA model? Don’t you think I‘d get fired after a few weeks of that?
Here's a thought experiment for you. Let's say you can run 1000 agents at 10,000 tokens a second. Do you think you are going to be more productive than someone running at 6tk/sec with the same model? Incase it's not clear, you will be generating 10,000,000 a second. Good luck verifying it. Token generation is not the bottleneck for creative work. If you are doing a predictable work and have a good workflow and massiv…
Earlier quoted context omitted.
While my colleagues are running 6 parallel agents at 50-100t/s each, with an actual SOTA model? Don’t you think I‘d get fired after a few weeks of that?
Why would you use this when your company has access to actual SOTA? I don't get it.
So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970
I’m kinda lost here… do y’all really have machines in your houses with hundreds of gigs of RAM?? Am I just behind the times? The page advertises the 8-bit quant as taking ~800GB, which seems like it would require at least 3 consumer motherboards fully stacked w/ 4x64GB cards each. Maybe “locally” has slowly come to imply “…on your homelab”?
Earlier quoted context omitted.
Why would you use this when your company has access to actual SOTA? I don't get it.
Why would I ever use a local model by that logic? A usable model means my computer was very expensive so I‘d have the funds for a Pro plan as well.
An other reason would be because your company does not allow any source code leaks and thus every developer either has local models or none.
6 T/second local model while your colleagues have 200 EUR/month claude does not make much sense. At least I can't see a use case.
Earlier quoted context omitted.
The ram/gpu shortage won't last forever though. Moreover we can be pretty confident that long-term the prices will obey wrights law and come down in cost significantly (from the pre-shortage prices) as we learn to produce them more efficiently. LLM companies are valued as if they're going to have some enduring monopoly that they can extract money from... GLM-5.2 and similar models make that valuation very very questi…
> The ram/gpu shortage won't last forever though Don't underestimate the markets ability to remain irrational
DwarfStar work in progress numbers: I see 14 tokens/sec generation, that slopes to 10 t/s with longer 10k or more context size. Consider that the indexed attention requires evaluating 2048 selected rows, 2x DeepSeek and with less compression, so the performances with larger contexts here to south faster. Prefill can be 180 t/s on small contexts to 150 t/s and less with larger contexts. I used DeepSeek v4 PRO in this…
Thank you for your work on DwarfStar! It is truly helping democratize access to frontier tech.