Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alfred-40B, an OSS RLHF version of Falcon40B

lighton.ai

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#21

Post by version_five »

brianjking wrote: post #17
brucethemoose2 wrote: post #16

Earlier quoted context omitted.

Pretty much anything with 32GB (?) total RAM+VRAM: https://github.com/cmp-nct/ggllm.cpp But its going to be slow without even a small Nvidia GPU (a 2060?). CPUs are really slow at prompt ingestion, and that can't be hidden with streaming.

Doesn't this new version of falcon need to be ggml'ed first?

The architecture is the same I belive, it's just a fine tune so there's nothing special to be done for this version. That said, ggml doesn't support Falcon, but i saw today there is a fork that claims to, though I didn't try it.

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#22

Post by brucethemoose2 »

version_five wrote: post #21
brianjking wrote: post #17

Earlier quoted context omitted.

Doesn't this new version of falcon need to be ggml'ed first?

The architecture is the same I belive, it's just a fine tune so there's nothing special to be done for this version. That said, ggml doesn't support Falcon, but i saw today there is a fork that claims to, though I didn't try it.

That link above is the fork ^

It uses the ggml library, just like llama.cpp does, and is indeed a fork of llama.cpp's implementation of ggml.

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#23

Post by version_five »

brucethemoose2 wrote: post #22
version_five wrote: post #21

Earlier quoted context omitted.

The architecture is the same I belive, it's just a fine tune so there's nothing special to be done for this version. That said, ggml doesn't support Falcon, but i saw today there is a fork that claims to, though I didn't try it.

That link above is the fork ^ It uses the ggml library, just like llama.cpp does, and is indeed a fork of llama.cpp's implementation of ggml.

Right, I'm being stupid, that's the fork I saw earlier today I didn't realize. Have you tried it? Iirc the documentation mentioned at 2-bit quantizatikn of the 40B model performing well. I've been using a 5-bit 7B llama2 which I'm generally happy with (because it can run in a pretty crappy machine) but interested to see the differences.

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#24

Post by brucethemoose2 »

version_five wrote: post #23
brucethemoose2 wrote: post #22

Earlier quoted context omitted.

That link above is the fork ^ It uses the ggml library, just like llama.cpp does, and is indeed a fork of llama.cpp's implementation of ggml.

Right, I'm being stupid, that's the fork I saw earlier today I didn't realize. Have you tried it? Iirc the documentation mentioned at 2-bit quantizatikn of the 40B model performing well. I've been using a 5-bit 7B llama2 which I'm generally happy with (because it can run in a pretty crappy machine) but interested to see the differences.

I wouldn't go lower than Q3_K_S, as its basically the same filesize, and llama 33B has a big perplexity dropoff.

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#25

Post by 0cf8612b2e1e »

version_five wrote: post #20
tysam_and wrote: post #19

Will every comment here be a question? Will someone break the trend?

It's a good observations- there are so many unknowns about all these models. Every day there's a new wizard_uncensored_rhlf_alpaca_tuned_best_one_use_this_13B_4.6-bit_rqm.pth that gets released, it's almost impossible to know the relative merits and which are worth paying attention to.

How true. Every time I browse a model listing there are four word descriptions with little sense of versioning, provenance, hardware requirements, or any reason why I would choose one vs the other.

Re: Alfred-40B, an OSS RLHF version of Falcon40B

#26

Post by Tepix »

paulcjh wrote: post #4

Any performance benchmarks compared to other LLMs? Also, any performance increases on the orig Falcon model in inference speed? We ditched most of our focus on Falcon 40B after Llama 2 70B came out, both the tokens per sec and quality of results are not even close.

Did you fine-tune your own Llama 2? Llama2-chat is awfully gimped.
Post reply on HN