Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Falcon 40B LLM (which beats Llama) now Apache 2.0

twitter.com

Post reply on HN
Print view View original

71–80 of 143 posts

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#71

Post by Semaphor »

gymbeaux wrote: post #69
amstan wrote: post #65

40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).

70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15. Used DDR3 ECC would be roughly half that.

I think they are talking about VRAM

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#74

Post by esperent »

amstan wrote: post #65

40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).

I briefly looked at prices a few days ago, I think you can rent an 80gb GPU for about $2.50 an hour. Then you just pay while you're using it.

Someone else might be better able to confirm the pricing, but in any case you don't need to purchase the hardware.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#76

Post by esperent »

valvar wrote: post #72
amstan wrote: post #65

40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).

That much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?

It has to be GPU RAM from my understanding, unless you're happy to wait several minutes/hours for each response.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#77

Post by low_tech_punk »

devmunchies wrote: post #64
kossTKR wrote: post #63

Earlier quoted context omitted.

Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.

nerfed how? I didn't know this.

maybe the comment was about safety related training leading to performance loss. Sebastien used the "Unicorn benchmark" to visualize such nerfing. Watch his talk at timestamp 26:22. Ref: https://youtu.be/qbIk7-JPB2c?t=1582

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#78

Post by drdaeman »

Semaphor wrote: post #71
gymbeaux wrote: post #69

Earlier quoted context omitted.

70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15. Used DDR3 ECC would be roughly half that.

I think they are talking about VRAM

Anything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#79

Post by GaggiX »

valvar wrote: post #72
amstan wrote: post #65

40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).

That much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?

From what I can find online LLAMA-65B 4-bit quantized can run 1 token/s on a Ryzen 7 3700X (using llama.cpp).

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#80

Post by Semaphor »

drdaeman wrote: post #78
Semaphor wrote: post #71

Earlier quoted context omitted.

I think they are talking about VRAM

Anything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.

I know, I have a weak GPU but 64 GB RAM. Using those models works, but it’s more "ask a question, then do something else for a while, while your fans spin up" ;)
Post reply on HN