Anyone keeping track of all these LLM releases?
This is the closest thing I'm aware of: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Alfred-40B, an OSS RLHF version of Falcon40B
11–20 of 26 posts
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#12What kind of hardware do I need to run this sufficiently well? I.e. say I want 10 tokens/s, what specs am I looking at?
This particular model has 83.66gb of model weights so you'll need to 2x Nvidia 80gb A100 at a minimum unless you're loading it in 8bit mode.
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#13so its open source? Where is the link to the weights? The article was mildly annoying because of many underlines sentences and words that looked like links.
The weights are literally linked in the article. The license is Apache 2.0 https://huggingface.co/lightonai/alfred-40b-0723/tree/main
The fact that this empty SEO blogspam bizarrely underlines (e.g. just like links) loads of content kind of obscures the single link to HuggingFace.
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#14Any performance benchmarks compared to other LLMs? Also, any performance increases on the orig Falcon model in inference speed? We ditched most of our focus on Falcon 40B after Llama 2 70B came out, both the tokens per sec and quality of results are not even close.
I'm assuming this will have similar scores to the original 40B model, in which case LLaMa2 70b would outperform it. The avg score on the Open LLM Leaderboard of LLaMa2 70b instruct is 72.3.
Falcon-40B is 63.4 or 61.5 on the non instruction tuned version.
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#15Will the team be releasing the momentum-internal dataset?
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#16What kind of hardware do I need to run this sufficiently well? I.e. say I want 10 tokens/s, what specs am I looking at?
Pretty much anything with 32GB (?) total RAM+VRAM:
https://github.com/cmp-nct/ggllm.cpp
But its going to be slow without even a small Nvidia GPU (a 2060?). CPUs are really slow at prompt ingestion, and that can't be hidden with streaming.
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#17What kind of hardware do I need to run this sufficiently well? I.e. say I want 10 tokens/s, what specs am I looking at?
Pretty much anything with 32GB (?) total RAM+VRAM: https://github.com/cmp-nct/ggllm.cpp But its going to be slow without even a small Nvidia GPU (a 2060?). CPUs are really slow at prompt ingestion, and that can't be hidden with streaming.
Doesn't this new version of falcon need to be ggml'ed first?
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#18What kind of hardware do I need to run this sufficiently well? I.e. say I want 10 tokens/s, what specs am I looking at?
This particular model has 83.66gb of model weights so you'll need to 2x Nvidia 80gb A100 at a minimum unless you're loading it in 8bit mode.
With that said, there are ggml/gptq and other optimization techniques.
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#19Will every comment here be a question? Will someone break the trend?
Re: Alfred-40B, an OSS RLHF version of Falcon40B
#20Will every comment here be a question? Will someone break the trend?
It's a good observations- there are so many unknowns about all these models. Every day there's a new wizard_uncensored_rhlf_alpaca_tuned_best_one_use_this_13B_4.6-bit_rqm.pth that gets released, it's almost impossible to know the relative merits and which are worth paying attention to.