61–70 of 113 posts
why not one of these? https://www.amazon.sg/NVIDIA-Jetson-Orin-64GB-Developer/dp/B...
In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
> TFA doesn't say where they are "the 1,440W limit on wall outlets in California" is a pretty good hint.
> You pay a lot upfront for the hardware, but if your usage of the GPU is heavy, then you save a lot of money in the long run. Last I saw data on this wasn’t true. A like for like comparison (same model and quant) API is cheaper than elec so you never make back hardware cost. That was a year ago and api costs have plummeted so I’d imagine it’s even worse now. Datacenters have cheaper elec, can do batch inference at s…
Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
I used a similar budget and build something like this: 7x RTX 3060 - 12 GB which results in 84GB Vram AMD Ryzen 5 - 5500GT with 32GB Ram All in a 19-inch rack with a nice cooling solution and a beefy power supply. My costs? 1300 Euro, but yeah, I sourced my parts on ebay / second hand. (Added some 3d printed parts into the mix: https://www.printables.com/model/1142963-inter-tech-and-gene... https://www.printables.com…
Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
My home AI machine does image classification.
Earlier quoted context omitted.
During inference, each token passes through each parameter of the model as a matrix vector products. And then as context grows, each new token passes through all current context tokens as matrix vector products. This means bandwidth requirements grow as context sizes grow. For datacenter workloads batching can be used to efficiently use this memory bandwidth and make things compute bound instead
[I'm still not understanding] It seems to me that even if you pass in a long context on every prompt, that context is still tiny compared to the execution time on the processor/GPU/tensorcore/etc. Lets say I load up a model of 12GB on my 12GB VRAM GPU. I pass in a prompt with 1MB of context which causes a response of 500kb after 1s. That's still only 1.5MB of IO transferred in 1s, which kept the GPU busy for 1s. Incr…
Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
My home AI machine does image classification.