Amy implementation for this akin to llama.cpp?
Falcon 40B LLM (which beats Llama) now Apache 2.0
81–90 of 143 posts
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#82For people who directly want to check the benchmark - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
GPT-4 wasn’t nerfed. https://news.ycombinator.com/item?id=36155267
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#8340B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#8440B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
[deleted]
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#8540B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
> https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm...
Here somebody quantized it down to 29929.56MB .
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#86Chatting with Falcon 40B feels like chatting with GPT4, very capable model.
Can you give examples?
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#87Amy implementation for this akin to llama.cpp?
Same question, so far I've found this thread https://github.com/ggerganov/llama.cpp/issues/1602 where people work on it.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#8840B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD?
Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#8940B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
I do use a P40 for my machine learning box, but I'm curious how you put three on the same system, given they need a CPU power plug and a pci-e port. Then, to cool them, you need to plug your own cooling system, requiring more specific power plugs to be available. What kind of chassis, motherboard, power unit you use to do that? It'll certainly will cost more than $1000 anyway, especially since you also need a decent amount of RAM to preload the models before you move them to the GPUs.