331–340 of 372 posts
Earlier quoted context omitted.
Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.
Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.
Earlier quoted context omitted.
Was that cheaper than a Blackwell 6000? But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
edit: Found your comment about /r/localllama, but if you have anything more to add I'm still very interested.
Earlier quoted context omitted.
I found it useful to preface with * this section written by me typing on keyboard * * this section produced by AI * And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.
> attach 10x length worth of AI appendix that would be helpful indexing and references. Are references helpful when they're generated? The reader could've generated them themselves. References would be helpful if they were personal references of stuff you actually read and curated. The value then would be getting your taste. References from an AI may well be good-looking nonsense.
Yes, sometimes this is true, but not always.
Note, it's not one prompt (there aren't really "one prompt" any more, prompt engineering is such a 2023-2024 thing), or purely unreviewed output. It's curated output that was created by AI but iterated with me since it goes with and has to match my intention. And most of the time I don't directly prompt the agent any more. I go through a layer of agent management that inject more context into the agent that actually work on it.
Earlier quoted context omitted.
You're almost certainly (definitely, in fact) confusing the 120b and 20b models.
I'm most certainly not doing so. seg@seg-epyc:~/models$ du -sh * /llmzoo/models/* | sort -n 4.0K metrics.txt 4.0K opus 4.0K start_llama 8.2G nvidia_Orchestrator-8B-Q8_0.gguf 12K config.ini 34G Qwen3.5-27B 47G Qwen3.5-35B 51G Qwen3.5-27B-BF16 61G gpt-oss-120b-F16.gguf 65G Qwen3.5-35B-BF16 106G Qwen3.5-122B-Q6 117G GLM4.6V 175G MiniMax-M2.5 232G /llmzoo/models/small_models 240G Ernie4.5-300B 377G DeepSeekv3.2-nolight 3…
There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…
Earlier quoted context omitted.
Sort of? It’s 120V RMS to ground.
yes, this is accurate for US and “works” but it’s against code here. you’ll get mildly shocked by metallic cabinets and fixtures especially if you’re barefoot and become the new shortest path to ground. old construction in the US sometimes did this intentionally (so old, the house didn’t have grounds. Or to “pass” an inspection and sell a place) but if a licensed electrician sees this they have to fix it. I’m dealing…
I can’t tell what practice you’re referring to. Are you perhaps referring to older wiring that connects large appliances to a neutral and two hots but no ground, e.g. NEMA 10-30R receptacles? Those indeed suck and are rather dangerous. Extra dangerous if the neutral wiring is failing or undersized anywhere.
But even NEMA 10-30R receptacles are still 120V RMS phase-to-ground. (And, bizarrely, there’s an entire generation of buildings where you might find proper 4-conductor wiring to the dryer outlet and a 10-30R installed — you can test the wiring and switch to 14-30R without any rewiring.)
The exception for residential wiring is when the neutral feed from the utility transformer fails, in which case you may have 240V phase-to-phase with the actual Earth floating somewhere in the middle (via the service’s ground connection), which can result in phase-to-neutral and phase-to-ground measured anywhere in the house varying from 0 to 240V RMS.
> wet bathroom floor
A GFCI receptacle adds a considerable degree of safety and can be installed with arbitrarily old wiring. It’s even permitted by code to install one with no ground connection as long as you label it appropriately — look it up in your local code.
Earlier quoted context omitted.
It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU. The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System R…
Could you share what you are using for inference and how you are running it? I have a 64G VRAM/128G system RAM setup.
Earlier quoted context omitted.
Meh. DGX is Arm and CUDA. Strix is X86 and ROCm. Cuda has better support than ROCm . And x86 has better support than Arm. Nowadays I find most things work fine on Arm. Sometimes something needs to be built from source which is genuinely annoying. But moving from CUDA to ROCm is often more like a rewrite than a recompile.
> But moving from CUDA to ROCm is often more like a rewrite than a recompile. Isn't everyone* in this segment just using PyTorch for training, or wrappers like Ollama/vllm/llama.cpp for inference? None have a strict dependency on Cuda. PyTorch's AMD backend is solid (for supported platforms, and Strix Halo is supported). * enthusiasts whose budget is in the $5k range. If you're vendor-locked to CUDA, Mac Mini and Str…
There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…