New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super • Hacker News Board

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#31

Post by michaelt » Mon, Jan 08, 2024, 5:32 PM UTC

baobabKoodaa wrote: post #18

loloquwowndueo wrote: post #15

Earlier quoted context omitted.

Why?

With the current generation of text and image generators, 24GB is the sweet spot

My impression is a lot of the open source action is around the just-about-runs-in-12GB region - lots of models coming out with 7B/13B and 4-bit quantisation, a few 70B models (which won't fit in 24GB anyway) and only limited stuff in between.

I suppose I could be getting a biased impression though, as of course many more people are in a position to recommend the more accessible models.

What sort of things are you running that take full advantage of that 24GB?

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#32

Post by eurekin » Mon, Jan 08, 2024, 5:32 PM UTC

ShamelessC wrote: post #20

eurekin wrote: post #13

Earlier quoted context omitted.

Thanks! That rules them out completely EDIT: For training

Are you aware that cards containing “LLM” (40-80GB) levels of VRAM cost substantially more and the status quo for consumer cards hovers around 4-12GB, only going to 24GB for top end cards?

Of course

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#33

Post by anonym29 » Mon, Jan 08, 2024, 5:34 PM UTC

Salgat wrote: post #22

baobabKoodaa wrote: post #18

Earlier quoted context omitted.

With the current generation of text and image generators, 24GB is the sweet spot

I wouldn't be shocked if the 5090 is also the only one with 24GB. Seems like NVidia is trying their hardest to suppress memory increases.

When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k, selling consumer cards with 24GB for $2k is almost a form of charity, by comparison.

AMD and Intel GPUs do not have the software ecosystem for AI workloads that Nvidia does, though AMD is rapidly improving. Nvidia has had an effective monopoly on the AI hardware space for the last year or so, and continues to have an effective near-monopoly, but that won't last forever as AMD and Intel catch up.

The VRAM is one of the largest differentiators of their cards. Sufficient VRAM allows you to run huge LLMs like 65B in-memory, which is orders of magnitudes faster than system RAM + CPU. Smaller amounts of VRAM require swapping between VRAM and system RAM and incur a major performance penalty.

Businesses are fighting to fork over $50k+/card for 40/80GB cards with the same processor as the 24GB consumer cards - it doesn't make economic sense for Nvidia to offer more on the consumer cards, lest they start cannibalizing demand for the enterprise cards.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#34

Post by baobabKoodaa » Mon, Jan 08, 2024, 5:35 PM UTC

garyfirestorm wrote: post #28

baobabKoodaa wrote: post #18

Earlier quoted context omitted.

With the current generation of text and image generators, 24GB is the sweet spot

many text generation models run on my 11G 1080Ti you can run quantized versions of these models if you aren't running it quantized, I'd say even 24 gig is not enough

If you want to get the most bang for your buck, you definitely need to run quantized versions. Yes, there are models that run in 11G, just like there are models that run in 8G, and for any other amount of VRAM - my point is that 24G is the sweet spot.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#35

Post by tagrun » Mon, Jan 08, 2024, 5:36 PM UTC

ziml77 wrote: post #26

alyandon wrote: post #19

Earlier quoted context omitted.

Considering the poor generational performance uplift the original 4000 series cards had vs the 3000 series - it almost seems like these super variants are what nVidia should have originally launched. :-/

Poor performance uplift? The 4080 is significantly faster than the 3090 with much lower power consumption!

Raw performance per dollar (after including inflation adjustment) has stagnated in 40 Series. A similar thing happened in 20 Series.

SUPER series has been a response to rival products offering better raw performance/price released afterwards.

Power consumption is a separate issue which may or may not be a concern depending on where you live.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#36

Post by eurekin » Mon, Jan 08, 2024, 5:37 PM UTC

nightski wrote: post #24

baobabKoodaa wrote: post #18

Earlier quoted context omitted.

With the current generation of text and image generators, 24GB is the sweet spot

Agreed but there is a lot more ML out there than just LLMs. You can't solve everything with prose.

Attention mechanism, the core of LLM, is universal enough to be brought back to standard vision models. Which is kind of ironic, since vision models were dominated by convolutions, and, the transformer is dubbed "convolution for text".

The real reason is that it doesn't deteriorate with regards to the input length in case of text, or far neighbourhood in case of vision. It's just a universal, new, building block that allows for shallower neural networks to perform more like their bigger versions

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#37

Post by throwup238 » Mon, Jan 08, 2024, 5:37 PM UTC

notJim wrote: post #17

selectodude wrote: post #10

Earlier quoted context omitted.

They’re trying to sell to me, the owner of a fully orphaned 3070 with a measly 8GB of vram. Not a current 40 series owner.

What does orphaned mean in this context?

The two parent 2080s were lost in a tragic cryptocurrency mining accident when the bitcoin ceiling collapsed.

Please send donations to the Aboveground Miners fund in your choice of shitcoin.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#38

Post by anonym29 » Mon, Jan 08, 2024, 5:37 PM UTC

bryanlarsen wrote: post #5

AMD also released the 7600XT today. Not particularly interesting, but if you need/want a cheapish 16GB card you now have an option beside Intel. https://ir.amd.com/news-events/press-releases/detail/1176/am...

Keep in mind that for local LLM inference, Nvidia's software support is qualitatively superior to AMD's, for the time being. That said, it's worth noting that AMD is catching up quickly.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#39

Post by baobabKoodaa » Mon, Jan 08, 2024, 5:39 PM UTC

michaelt wrote: post #31

baobabKoodaa wrote: post #18

Earlier quoted context omitted.

With the current generation of text and image generators, 24GB is the sweet spot

My impression is a lot of the open source action is around the just-about-runs-in-12GB region - lots of models coming out with 7B/13B and 4-bit quantisation, a few 70B models (which won't fit in 24GB anyway) and only limited stuff in between. I suppose I could be getting a biased impression though, as of course many more people are in a position to recommend the more accessible models. What sort of things are you run…

I actually have a 12GB card, which I purchased specifically for AI (24GB cards are too expensive for me). You're correct that 12GB is also a sweet spot in terms of what you get per dollar spent.

Re: New GeForce RTX 4080 Super, RTX 4070 Ti Super, RTX 4070 Super

#40

Post by eurekin » Mon, Jan 08, 2024, 5:39 PM UTC

michaelt wrote: post #31

baobabKoodaa wrote: post #18

Earlier quoted context omitted.

With the current generation of text and image generators, 24GB is the sweet spot

My impression is a lot of the open source action is around the just-about-runs-in-12GB region - lots of models coming out with 7B/13B and 4-bit quantisation, a few 70B models (which won't fit in 24GB anyway) and only limited stuff in between. I suppose I could be getting a biased impression though, as of course many more people are in a position to recommend the more accessible models. What sort of things are you run…

As the ancestor commenter mentioned:

> If you’re interested in ML training

Training - at least the one I tried - requires to be run in fp16 mode. So a 7b net needs 14 GB for the model weights alone, plus some extra for the context and the stuff I don't really understand (some gradient values, oh that makes sense now that I've written it)