21–30 of 331 posts
Can anyone comment on the TinyBox they are taking preorders for? The tinybox 738 FP16 TFLOPS 144 GB GPU RAM 5.76 TB/s RAM bandwidth 30 GB/s model load bandwidth (big llama loads in around 4 seconds) AMD EPYC CPU 1600W (one 120V outlet) Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks) $15,000
Can anyone comment on the TinyBox they are taking preorders for? The tinybox 738 FP16 TFLOPS 144 GB GPU RAM 5.76 TB/s RAM bandwidth 30 GB/s model load bandwidth (big llama loads in around 4 seconds) AMD EPYC CPU 1600W (one 120V outlet) Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks) $15,000
It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing.
My guess is they'll end up creating some premium extensions to the software and selling that to make money. Or maybe they can sell an enterprise cluster manager type thing that comes with support. He's good at software so it makes sense for him to sell software.
And maybe the box will sell well initially just as a "dev kit" type thing.
Can anyone comment on the TinyBox they are taking preorders for? The tinybox 738 FP16 TFLOPS 144 GB GPU RAM 5.76 TB/s RAM bandwidth 30 GB/s model load bandwidth (big llama loads in around 4 seconds) AMD EPYC CPU 1600W (one 120V outlet) Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks) $15,000
There's a reason no one uses ATI GPUs in datacenters. Their dev support is shit. Don't waste your money. Buy 6 RTX 4090's and a decent ECC-memory server, and call it a day.
My prediction is AMD is already working on this internally, except more oriented around PyTorch not Hotz's Tinygrad, which I doubt will get much traction.
If they can achieve something competitive with CUDA for $5m, why hasn't AMD done it yet?
The way they make money is for AMD to buy them out if they’re successful
If they can achieve something competitive with CUDA for $5m, why hasn't AMD done it yet?
AMD has been tackling exactly the wrong problems. They poured their money into a porting solution for developers to take CUDA code and run it on their GPUs. I guess they didn't find it worth it to really compete. I doubt it's about being able to tackle this problem with $5m, but rather convincing the company they can win.
They should've been adding tensor cores and neural acceleration to their CPUs. The need for headed graphics cards is moot and wasteful. NVIDIA solved this with the A100.
NVIDIA may spin into a mainstream enterprise CPU and systems vendor as a sales channel for converged CPU-GPU solutions beyond what they're already doing.
Can anyone comment on the TinyBox they are taking preorders for? The tinybox 738 FP16 TFLOPS 144 GB GPU RAM 5.76 TB/s RAM bandwidth 30 GB/s model load bandwidth (big llama loads in around 4 seconds) AMD EPYC CPU 1600W (one 120V outlet) Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks) $15,000
I like George's style and wish him well. But I'm not optimistic about their chances of selling $15k servers that are $10k in parts (or whatever the exact numbers are). It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing. My guess is they'll end up creating some premium extensions to the software and selling that to make money. Or maybe…
Shows you what is possible in 2.5 years. Keeps me motivated to learn.