Are we at the point where 2x 9070XT's are a viable LLM platform? (I know this has 4, just wondering for myself).
I'd go with strix halo if you're looking at that old of tech.
the latest AMD GPUs are RX 9070 XT w/32GB each
Tinybox – A powerful computer for deep learning
121–130 of 372 posts
Re: Tinybox – A powerful computer for deep learning
#122I always wonder about these expensive products: Does the company make them once its ordered or do they just make them beforehand?
In this case, they're taking wire transfers, so they're definitely building them once they get the cash.
Re: Tinybox – A powerful computer for deep learning
#123There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing. Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model. One thought: The ability to conveniently get this onto a 240v circu…
I find that the most respected writing about AI has very few signs of being written by AI. I'm guessing that's because people in the space are very sensitive to the signs and signal vs. noise.
I found it useful to preface with
* this section written by me typing on keyboard *
* this section produced by AI *
And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.
Re: Tinybox – A powerful computer for deep learning
#124There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…
> I'm also confused why this is 12U. My whole rig is 4u.
I imagine that's because they are buying a single SKU for the shell/case. I imagine their answer to your question would be: In order to keep prices low and quality high, we don't offer any customization to the server dimensions
Re: Tinybox – A powerful computer for deep learning
#125Earlier quoted context omitted.
A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.
It's more accurate to say that the typical 120V circuit is just a 240V source with the neutral tapped into the midpoint of the transformer winding.
This. It definitely comes in at a higher voltage.
Re: Tinybox – A powerful computer for deep learning
#126Quite expensive little bastard. I wonder how much does it make sense to invest in a such device, if you can get $0.40/mtok from hyperbolic for example
Re: Tinybox – A powerful computer for deep learning
#127Earlier quoted context omitted.
> And there's no room for kv, so you'll OOM around 4k of context. Can't you offload KV to system RAM, or even storage? It would make it possible to run with longer contexts, even with some overhead. AIUI, local AI frameworks include support for caching some of the KV in VRAM, using a LRU policy, so the overhead would be tolerable.
Not worth it. It is a very significant performance hit. With that said, people are trying to extend VRAM into system RAM or even NVMe storage, but as soon as you hit the PCI bus with the high bandwidth layers like KV cache, you eliminate a lot of the performance benefit that you get from having fast memory near the GPU die.
> With that said, people are trying to extend VRAM into system RAM or even NVMe storage
Only useful for prefill (given the usual discrete-GPU setup; iGPU/APU/unified memory is different and can basically be treated as VRAM-only, though a bit slower) since the PCIe bus becomes a severe bottleneck otherwise as soon as you offload more than a tiny fraction of the memory workload to system memory/NVMe. For decode, you're better off running entire layers (including expert layers) on the CPU, which local AI frameworks support out of the box. (CPU-run layers can in turn offload to storage for model parameters/KV cache as a last resort. But if you offload too much to storage (insufficient RAM cache) that then dominates the overhead and basically everything else becomes irrelevant.)"
Re: Tinybox – A powerful computer for deep learning
#128[flagged]
Re: Tinybox – A powerful computer for deep learning
#129Earlier quoted context omitted.
If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem
The $12,000 one also requires it.
Easier to get two circuits than rewire a breaker in an office you might be renting, no?
(I work for an electrical contractor so my sense of ease might be overcorrecting)
Re: Tinybox – A powerful computer for deep learning
#130Is this like the new equivalent of crypto mining? I remember the early days when they would sell hardware for farming crypto, now it’s AI?