Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Falcon 40B LLM (which beats Llama) now Apache 2.0

twitter.com

Post reply on HN
Print view View original

31–40 of 143 posts

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#31

Post by smoldesu »

paulgb wrote: post #29
smoldesu wrote: post #28

Earlier quoted context omitted.

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Now I’m curious what your bot does!

It's one of those "say a keyword with a question, get a response" type bots. I added in a few other "prompt sources" though, where it grabs the first part of an RSS entry or HN comment and tries to autocomplete the rest. Mostly just a boring testbed for me to play with models, for free, with friends.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#32

Post by wsgeorge »

takeiteasyy wrote: post #23
mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

This will require a cpp port first to run on e.g. apple silicon?

No, but to benefit from the ggml ecosystem, yes. Someone's taking a stab at it: https://github.com/nikisalli/falcon.cpp

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#33

Post by BonoboIO »

bongobingo1 wrote: post #24
mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

Will be interesting to see if someone comes up with an ASIC or FPGA.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#34

Post by wrs »

bongobingo1 wrote: post #24
mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

The minimum size of a model with equivalent performance is unknown (like so much else about LLMs), so requirements could be reduced.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#35

Post by gjreda »

Not specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere.

I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires you to use their commercial offerings.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#36

Post by dzhiurgis »

mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

is there a model version where it can be deployed using huggingface's tool?

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#37

Post by generalizations »

jrflowers wrote: post #12
version_five wrote: post #7

Earlier quoted context omitted.

I don't know unity's model. I call Falcon sleazy because the royalty stuff (plus a bunch of absurd related terms) were buried in the license while the headline said open source. I'm not against commercial products, nor products with free and premium tiers. This otoh felt, like I say, just sleazy.

Here is a link to an explanation of the Unity model. https://gamedevbeginner.com/is-unity-free/

The explanation is less interesting than the fact they chose not to be upfront in their PR about the limitations of their original license. Personally, I don't care what terms they choose - but I do care if they misrepresent them. Honesty is important, and they weren't.

But all that's void now that they've gone Apache.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#39

Post by dzhiurgis »

gjreda wrote: post #35

Not specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere. I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires yo…

> HF requires you to use their commercial offerings

Some of which are quite affordable ($80 per month). Larger ones can be like 2000 a month which is still ok to prototyping phase. You're basically paying for aws/gcp infrastructure.

I quite liked the UX of it, very intuitive. My trouble was finding a model that executes out-of-the-box tho. All of the GPT ones crash on startup.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#40

Post by djbusby »

fnordpiglet wrote: post #38

Serious question - why doesn’t someone crowd source the funds to train a GPT scale model for open source? I assume it’s not just a matter of a ton of GPU instances?

Doesn't one need to have a bunch of "very good" data to train on? I'm under the impression that sourcing costs are large.
Post reply on HN