21–30 of 113 posts
In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
No need for that.
If the author is reading this I'll point out that the cuda toolkit you find in the repositories is generally older. You can find the latest versions straight from Nvidia: https://developer.nvidia.com/cuda-downloads?target_os=Linux&... The caveat is that sometimes a library might be expecting an older version of cuda. The vram on the GPU does make a difference, so it would at some point be worth looking at another GPU…
Whenever I get to a section that was clearly autogenerated by an LLM I lose interest in the entire article. Suddenly the entire thing is suspect and I feel like I’m wasting my time, since I’m lo lingering encountering the mind of another person, just interacting with a system.
I followed in those footsteps to create my own [1] (photo [2]).
I picked up a 24GB M40 for around $300 off eBay. I 3D printed a "cowl" for the GPU that I found online and picked up two small fans from Amazon that got int he cowl. Attached the cowl + fans keep the GPU cool. (These TESLA server GPUs have no fan since they're expected to live in one of those wind-tunnels called a server rack).
I bought the same cheap DELL server PS that the original person had used and I also had to get a break-out board (and power-supply cables and adapters) for the GPU.
Thanks to LLMs, I was able to successfully install Rocky Linux as well as CUDA and NVIDIA drivers. I SSH into it and run ollama commands.
My own hurdle at this point is: I have a 2nd 24 GB M40 TESLA but when installed on the motherboard, Linux will not boot. LLMs are helping me try to set up BIOS correctly or otherwise determine what the issue is. (We'll see.) I would love to get to 48 GB.
[0] https://www.aliexpress.us/item/3256806580127486.html
[1] https://bsky.app/profile/engineersneedart.com/post/3lmg4kiz4...
[2] https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxjqlam...
With system builds like this I always feel the VRAM is the limiting factor when it comes to what models you can run, and consumer grade stuff tends to max out at 16GB or (somemtimes) 24GB for more expensive models. It does make me wonder whether we'll start to see more and more computers with unified memory architecture (like the Mac) - I know nvidia have the Digits thing which has been renamed to something else
I thought prevailing wisdom was that a used 3090 with it's larger vram was the best budget gpu choice? And in general, if on a budget then why not buy used and not new? And more so as the author himself talks about the resale value for when he sells it on.
> I thought prevailing wisdom was that a used 3090 with it's larger vram was the best budget gpu choice? The trick is memory bandwidth - not just the amount of VRAM - is important for LLM inference. For example, the B50 specs list a memory bandwidth of 224 GB/s [1], whereas the Nvidia RTX 3090 has over 900GB/s [2]. The 4070's bandwidth is "just" 500GB/s [3]. More VRAM helps run larger models but with lower bandwidth…
I'm not really knowledgeable about this space, so maybe I'm missing something:
Why does the bus performance affect token generation? I would expect it to cause a slow startup when loading the model, but once the model is loaded, just how much bandwidth can the token generation possibly use?
Token generation is completely on the card using the memory on the card, without any bus IO at all, no?
IOW, I'm trying to think of what IO the card is going to need for token generation, and I can't think of any other than returning the tokens (which, even on a slow 100MB/s transfer is still going to be about 100x the rate at which tokens are being generated.