81–90 of 113 posts
Earlier quoted context omitted.
Using just an Ollama VL Model (gemma3/mistral-small3.1/qwen2.5vl) or a specific library?
My home server detects NSFW images in user generated content on my side project. source code: https://github.com/KevinColemanInc/NSFW-FLASK
>DECISION: Nvidia RTX 4070 I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1]. [1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI: https://news.ycombinator.com/item?id=44196991
>DECISION: Nvidia RTX 4070 I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1]. [1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI: https://news.ycombinator.com/item?id=44196991
I've been dreaming on pcpartpicker. I think Radeon RX 7900 XT - 20 GB has been the best bang for your buck. Enables full gpu 32B? Looking at what other people have been doing lately, they arent doing this. They are getting 64+ core cpus and 512GB of ram. Keeping it on cpu and enabling massive models. This setup lets you do deepseek 671B. It makes me wonder, how much better is 671B vs 32B?
32B has improved leaps and bounds in the past year. But Deepseek 671B is still a night and day comparison. 671B just knows so much more stuff.
The main issue with RAM-only builds is that prompt ingestion is incredibly slow. If you're going to be feeding in any context at all, it's horrendous. Most people quote their tokens/s with basically non-existent context (a few hundred tokens). Figure out if you're going to be using context, and how much patience you have. Research the speed you'll be getting for prompt processing / token generation at your desired context length in each instance, and make your decision based on that.
I used a similar budget and build something like this: 7x RTX 3060 - 12 GB which results in 84GB Vram AMD Ryzen 5 - 5500GT with 32GB Ram All in a 19-inch rack with a nice cooling solution and a beefy power supply. My costs? 1300 Euro, but yeah, I sourced my parts on ebay / second hand. (Added some 3d printed parts into the mix: https://www.printables.com/model/1142963-inter-tech-and-gene... https://www.printables.com…
Earlier quoted context omitted.
During inference, each token passes through each parameter of the model as a matrix vector products. And then as context grows, each new token passes through all current context tokens as matrix vector products. This means bandwidth requirements grow as context sizes grow. For datacenter workloads batching can be used to efficiently use this memory bandwidth and make things compute bound instead
[I'm still not understanding] It seems to me that even if you pass in a long context on every prompt, that context is still tiny compared to the execution time on the processor/GPU/tensorcore/etc. Lets say I load up a model of 12GB on my 12GB VRAM GPU. I pass in a prompt with 1MB of context which causes a response of 500kb after 1s. That's still only 1.5MB of IO transferred in 1s, which kept the GPU busy for 1s. Incr…
PCIe bus performance is basically irrelevant.
> Token generation is completely on the card using the memory on the card, without any bus IO at all, no?
Right. But the GPU can't instantaneously access data in VRAM. It has to be copied from VRAM to GPU registers first. For every token, the entire contents of VRAM has to be copied to the GPU to be computed. It's a memory-bound process.
Right now there's about an 8x difference in memory bandwidth between low-end and high-end consumer cards (e.g., 4060 Ti vs 5090). Moving up to a B200 more than doubles that performance again.
why not one of these? https://www.amazon.sg/NVIDIA-Jetson-Orin-64GB-Developer/dp/B...
Those DGX machines are still at right around the corner state.