For fast inference, the HuggingFace cofounder, Thom Wolf recommends their text-generation-inference library https://github.com/huggingface/text-generation-inference
Falcon 40B LLM (which beats Llama) now Apache 2.0
131–140 of 143 posts
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#132Earlier quoted context omitted.
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
Will be interesting to see if someone comes up with an ASIC or FPGA.
analog chips like what MythicAI is developing seem like the next obvious leap forward for deploying inferences broadly. ASIC/FPGA wouldn't be much different than a GPU. ASIC seems like a brittle solution
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#133Earlier quoted context omitted.
Any particular reason not to run this model on a single Jetson AGX Orin 64GB? GP Core count is much lower than than the 4090 but it still does 275 int8 TOPS for only $2k
I'm the wrong person to ask but performance wise a 4090 has over a pflop and a Google search also showed a factor of 3-4 for a 3090.
The relevant performance metric here is memory amount and bandwidth.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#134For people who directly want to check the benchmark - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
Here's the discussion for that benchmark with some values.
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#135Earlier quoted context omitted.
They're okay. This isn't the place for a full review of their offerings (especially considering everyone's mixed feelings on Oracle), but I'm confident that it's better than most 1core/$5 deals you'll find elsewhere. > Are they any good? Yep, free tier allows you to spec up to 24gb of RAM without paying, which is cool. The bottleneck is really the disk speed, but that's not an issue with mmaped models. There's enough…
They will shut down the VPS if there's no activity on it, not sure how they detect this though
CPU idle % over a rolling time window (say, 15 minutes).
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#136I am an amateur when it comes to these models. What can I do with this model and how?
At the most basic, you give it text and it can guess at what comes next. This means you could type "10 types of ferns" and it will build a list of 10 ferns. Or you could type it out how a transcript of a conversation would look like and it will basically fill in the other "side" of the conversation to make a chatbot (all the complicated chatbots are basically abstracting this). Think of it like a text box with a super-smart (arguably) person also having access where you can type one thing and then it'll type whatever it thinks would be the next thing to type.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#137Earlier quoted context omitted.
Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
GPT-4 wasn’t nerfed. https://news.ycombinator.com/item?id=36155267
It most definitely was. The chat output of GPT4 is much faster now and much worse quality. If you go in the playground and use the March 14 api (as opposed to the default GPT4 API) it is high quality and slow. Yes, there are two GPT4 API endpoints .
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#138Earlier quoted context omitted.
Complicated legal issues of having the cake and eating it too.
And also the potential, unprecedented legal issues that accompany releasing and defending a free/open model. The brownie points they'd receive aren't worth it, at least yet.
GPT2 was released open. It’s been years. Microsoft’s CELA has vetted it. If they are thumbs up, I don’t see why others would be averse. Unless it’s not the reason. Methinks it isn’t.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#139Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#140Earlier quoted context omitted.
70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15. Used DDR3 ECC would be roughly half that.
I think they are talking about VRAM
oop you're right... I know once the model is trained they run on the CPU/regular RAM