Untitled topic • Hacker News Board

#1

Post by aurareturn » Thu, Dec 14, 2023, 3:21 AM UTC

No post body was provided.

Re: undefined

#2

Post by lostmsu » Thu, Dec 14, 2023, 4:03 AM UTC

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster.

Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM.

Re: undefined

#3

Post by aurareturn » Thu, Dec 14, 2023, 4:31 AM UTC

lostmsu wrote: post #2

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you w…

  It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster.

It really is. The LLM is benefiting from the high amount of high bandwidth RAM, which is a benefit of UMA in Macs. To get 128GB or more RAM from Nvidia, you'd have to combine two H100 GPUs, which retail for $30k+ each.

  Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM.

The CPU would have less compute power. I'd be curious to see the test results of an Epyc 128-core CPU with 12 memory channels though.

Re: undefined

#4

Post by jauntywundrkind » Thu, Dec 14, 2023, 4:35 AM UTC

lostmsu wrote: post #2

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you w…

AMD has been touting how the previous GPU the M250 has 128GB of HBM2e running 3.2TB/s.

Re: undefined

#5

Post by alanfranz » Thu, Dec 14, 2023, 6:15 AM UTC

lostmsu wrote: post #2

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you w…

Is there any real world benchmark about using 7945wx for inference?

Re: undefined

#6

Post by throwaway1249 » Thu, Dec 14, 2023, 7:13 AM UTC

This comparison is not fair, since the VRAM in the RTX4090 is not enough to hold the whole model in VRAM.

I have tested llama.cpp both on an M2 and in a 4090:

- The prompt ingestion time in M2 is pretty slow. - The extra memory of the M2 allows one to try more interesting models (Mixtral) and run multiple models at the same time.

Re: undefined

#7

Post by wmf » Thu, Dec 14, 2023, 7:17 AM UTC

jauntywundrkind wrote: post #4

lostmsu wrote: post #2

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you w…

AMD has been touting how the previous GPU the M250 has 128GB of HBM2e running 3.2TB/s.

MI250 (which is two GPUs) has 128 GB and H100 NVL (also two GPUs) has 188 GB at 7.8 TB/s. Both are probably an order of magnitude more expensive than a 4090 so they're not really in the same class.