61–70 of 84 posts
Also, it doesn't seem to say anything about the input model's format? pytorch weights? onnx?
Earlier quoted context omitted.
The apple lisa was 25k when adjusted for inflation…prices are for early adopters
Yes, and the Lisa failed, because there just weren’t enough adopters at that price. I struggle to see any sizable market for the tinybox, but I wish them good luck.
I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?
I was thinking that powerful used phones will be extremely valuable in the next years, since they are fairly cheap and more powerful than these devices.
Earlier quoted context omitted.
See tinybox https://tinygrad.org/
15,000$…
Earlier quoted context omitted.
15,000$…
$15k for a box with 144 GB GPU RAM is not bad, but I'm not clear on how they're going to run that from a single 1600W PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd need 2x 1600W PSUs and two separate 15amp circuits to run such a thing at home (in the US).
Earlier quoted context omitted.
Yes, and the Lisa failed, because there just weren’t enough adopters at that price. I struggle to see any sizable market for the tinybox, but I wish them good luck.
Lessons learned on the Lisa were applied to the Mac for less money to sell to more people. I suspect the same thing will happen with their VR/AR/XR headset.
The Vision is a whole different universe… with their cash and position they could (and may) take 10 cracks at it.
I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?
Jetson is basically like Apple. Theoretically its good, but the models with enough RAM are too expensive. Smartphones aside, little Ryzen 6000 boxes would be OK. Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.
You could maybe run something on the 4GB Jetson Nano?
But very slow: https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_poi...
Earlier quoted context omitted.
Jetson is basically like Apple. Theoretically its good, but the models with enough RAM are too expensive. Smartphones aside, little Ryzen 6000 boxes would be OK. Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.
You can get an Orin with 8GB. Or the 32/64GB AGX module for 1.3/2.3k (3k for the dev kit). Not cheap but plenty of ram and 60W power target maximum. You could maybe run something on the 4GB Jetson Nano? But very slow: https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_poi...
A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC, just barely) do llama 70B for far less money and a similar TDP.
Earlier quoted context omitted.
$15k for a box with 144 GB GPU RAM is not bad, but I'm not clear on how they're going to run that from a single 1600W PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd need 2x 1600W PSUs and two separate 15amp circuits to run such a thing at home (in the US).
You can undervolt gpus without loosing that much performance.
Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.
That's a tricky question. You're going to have to multiplex the use of the device, but since these are mostly 'ping-pong' style uses you can use something called a 'utilization factor' to figure out what a reasonable upper bound is where you still get an answer to your query in acceptable time. The typical mechanism is an input queue with a single worker to use the device. The cut-off is when the queue becomes unacce…