How much GPU RAM would be needed to run this with just one GPU?
Nvidia releases NVLM 1.0 72B open weight model
41–50 of 57 posts
Re: Nvidia releases NVLM 1.0 72B open weight model
#42Earlier quoted context omitted.
Nvidia created CUDA and seeded the ML industry for a decade before chatgpt. They aren't given enough credit for their foresight and strategy. Most companies would have choked the community to death with greed before it ever took off. There is a reason why CUDA works on every NV gpu but ROCm support is spotty at best and only guaranteed on data center GPUs.
My analogy still holds. NVIDIA just created good shovels that are useful in both the garden and in a gold mine. AMD and Intel insisted on selling only flimsy garden shovels.
AMD and intels shovels (hardware) are fine. The ecosystem is the problem. The fundamental difference is AMD/intel see it as an upsell whereas nvidia is willing to invest in long term organic growth. The problem is the C suite and the difference between companies run by founders and bean counters.
Re: Nvidia releases NVLM 1.0 72B open weight model
#43Earlier quoted context omitted.
Its for sorting by color/print. Some things you remember instantly by touch, others not so much.
This sounds really cool - so you point it at individual items of clothing and it reads out the type of clothing and colour? Do you have any more info or links about the setup?
https://www.bemyeyes.com/ you can scroll down to the new AI version.
Re: Nvidia releases NVLM 1.0 72B open weight model
#44How much GPU RAM would be needed to run this with just one GPU?
I haven't tested it, but likely around 170GB, regardless of if you're using only one GPU or spreading it out over several ones.
Re: Nvidia releases NVLM 1.0 72B open weight model
#45Earlier quoted context omitted.
My analogy still holds. NVIDIA just created good shovels that are useful in both the garden and in a gold mine. AMD and Intel insisted on selling only flimsy garden shovels.
AMD and intels shovels (hardware) are fine. The ecosystem is the problem. The fundamental difference is AMD/intel see it as an upsell whereas nvidia is willing to invest in long term organic growth. The problem is the C suite and the difference between companies run by founders and bean counters.
We're actually in agreement, it's just that analogies are a blunt instrument.
I'm saying that Intel and AMD made single-purpose GPUs useful only for graphics. Whether that's because of the software or hardware is immaterial. Effectively, it's one product in the same sense that an iPhone is one product to a consumer, but technically it's the iPhone device + iOS the software + Apple services such as iCloud, music, etc...
Re: Nvidia releases NVLM 1.0 72B open weight model
#46Earlier quoted context omitted.
AMD and intels shovels (hardware) are fine. The ecosystem is the problem. The fundamental difference is AMD/intel see it as an upsell whereas nvidia is willing to invest in long term organic growth. The problem is the C suite and the difference between companies run by founders and bean counters.
We're actually in agreement, it's just that analogies are a blunt instrument. I'm saying that Intel and AMD made single-purpose GPUs useful only for graphics. Whether that's because of the software or hardware is immaterial. Effectively, it's one product in the same sense that an iPhone is one product to a consumer, but technically it's the iPhone device + iOS the software + Apple services such as iCloud, music, etc.…
It's not single purpose hardware or software. If you crawl over enough broken glass you can get anything to work on AMD/intel.
The distinction is one of business strategy not technology.
Re: Nvidia releases NVLM 1.0 72B open weight model
#47Earlier quoted context omitted.
Its for sorting by color/print. Some things you remember instantly by touch, others not so much.
This sounds really cool - so you point it at individual items of clothing and it reads out the type of clothing and colour? Do you have any more info or links about the setup?
Its basically a gpt4o in disguise. The feature is called BeMyAI, and it is being released via BeMyEyes.
I would have answered earlier, but the silly HN rate limiter prevented me from passing the link to you.
I dont want to look it up yet agan.
And I dont want to use HN anymore,, this rate limit time-waster really just killed my sympathy for this site.
Re: Nvidia releases NVLM 1.0 72B open weight model
#48It has a non-commercial cc-by-nc-4.0 license, I would guess the only way to use this in production is to use Nvidias data centers to host it? Or are there other ways?
Not a lawyer, not legal advice, but... the legal status quo is that neural network outputs are not copyrightable. They are currently considered not made by humans nor considered a derivative work from the training material / network weights (assuming it's not regurgitating copyrighted material verbatim). The cc-by-nc-4.0 license applies to the network weights. The only thing non-commercial about the license is that i…
> the legal status quo is that neural network outputs are not copyrightable.
Can't this flip on a dime and a billion dollar company lose billions?
Re: Nvidia releases NVLM 1.0 72B open weight model
#49How much GPU RAM would be needed to run this with just one GPU?
144GB VRAM to load the weights at FP16, 72GB quantized to FP8. To figure out the KV cache size you'll need for an LLM, you can use the following formula: https://x.com/AlpinDale/status/1841305040545329535
Simplified for posterity:
kv_bytes = kv_bits / 8
hidden_per_head = hidden_size // num_attention_heads
total_heads = hidden_per_head * num_key_value_heads
kv_bytes_per_token = 2 * kv_bytes * num_hidden_layers * total_heads
You can cut this down lower by using grouped query attention as described here: https://medium.com/@plienhar/llm-inference-series-4-kv-cachi... This allows you to divide that number by the number of grouped heads, although it trades off accuracy for VRAM usage.
But TLDR, a minimum of around 164GB of VRAM at full accuracy. To me that seems fairly low, and I think vLLM would OOM without significantly more than that, but that's about as low as you could go in theory if you're running everything at FP16. Half that, of course, for FP8.
You'll typically need to have a copy of the KV cache per GPU, if you're using multiple GPUs, so multiply the KV cache overhead by the number of GPUs you're using. This will depend on what the specs for the GPUs you're using are; for example, you'll need 3 H100s (really four, since vLLM wants the number of heads to be evenly divisible by the number of GPUs); if you're using L40Ses, you'll need eight of them; but most likely only a single AMD MI300x.
Re: Nvidia releases NVLM 1.0 72B open weight model
#50I think the only relevant part to note here is that this model showed improved text-only performance after multimodal training. Wonder if this translates to Llama models also ? Is it possible to extend Llama 3.1 405b with multi-modal training to create another SOTA large model ?
I think the answer here is "it depends." The Llama-3.2 series is an extended version of the Llama-3.1 series with multimodal (image) training, but they kept the language model weights frozen and only updated the new image weights. So in the end, the 3.2 series benchmarks identically to 3.1 on text-only tasks; the image weights provided no value to the language model weights.
Allowing the language model weights to be updated during training could potentially result in better performance on both tasks, though, if Nvidia's result replicates. I could believe that it might: after all, more diverse data is more diverse data, and the model will be forced during training to generalize more.