Falcon LLM: The Most Powerful Open Source LLM Released to Date
thesequence.substack.com
Falcon LLM: The Most Powerful Open Source LLM Released to Date
1–7 of 7 posts
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#2Could we link a non-paid source? Presumably this article is just rephrasing the announcement, and they have a picture of an eagle on the top. I'm not exactly a biology expert but cmon!
For example:
- https://huggingface.co/tiiuae
- https://www.tii.ae/news/falcon-40b-worlds-top-ai-model-rewar...
- or hell just the leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
P.S. The UAE makes LLMs now? That's... a tad unexpected. Must've been out of the loop. Good for them - despite political grievances, you've gotta admit this is pretty damn impressive.
EDIT: Also it looks like the news here is that they released an `instruct` version of the pre-existing `falcon-40b`? Might also be worth highlighting in the title.
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#3(didn't read because paywall)
From experimenting with Falcon-40b-instruct the base model, outputs are acceptable for very simple tasks though still dwarfed by closed-source LLMs in usefulness.
The thing stopping me from experimenting further is how expensive inferences are - 40b to takes 8-10mins per 200 token generation, loaded in 4-bit with transformers on a colab A100. The model weighs in around 27GB on VRAM with these settings.
Has anyone gotten faster inferences with the 40b model?
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#4[deleted]
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#5(didn't read because paywall) From experimenting with Falcon-40b-instruct the base model, outputs are acceptable for very simple tasks though still dwarfed by closed-source LLMs in usefulness. The thing stopping me from experimenting further is how expensive inferences are - 40b to takes 8-10mins per 200 token generation, loaded in 4-bit with transformers on a colab A100. The model weighs in around 27GB on VRAM with…
Why is the inference so sluggish?
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#6(didn't read because paywall) From experimenting with Falcon-40b-instruct the base model, outputs are acceptable for very simple tasks though still dwarfed by closed-source LLMs in usefulness. The thing stopping me from experimenting further is how expensive inferences are - 40b to takes 8-10mins per 200 token generation, loaded in 4-bit with transformers on a colab A100. The model weighs in around 27GB on VRAM with…
1 to 2 tokens per second on CPU, depending on the CPU/RAM, thanks to GGML.
See discussions: https://github.com/ggerganov/ggml/pull/231
And the code: https://github.com/jploski/ggml/tree/falcon40b
Re: Falcon LLM: The Most Powerful Open Source LLM Released to Date
#7(didn't read because paywall) From experimenting with Falcon-40b-instruct the base model, outputs are acceptable for very simple tasks though still dwarfed by closed-source LLMs in usefulness. The thing stopping me from experimenting further is how expensive inferences are - 40b to takes 8-10mins per 200 token generation, loaded in 4-bit with transformers on a colab A100. The model weighs in around 27GB on VRAM with…
Anyone tried Falcon 40b on a Apple M2 max or M2 ultra? How many tokens per second?