Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Falcon 40B LLM (which beats Llama) now Apache 2.0

twitter.com

Post reply on HN
Print view View original

21–30 of 143 posts

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#21

Post by brucethemoose2 »

version_five wrote: post #19
brucethemoose2 wrote: post #8

Earlier quoted context omitted.

Meta's eventual, internal LLM architecture will be totally different than the open source LLMs, right? They don’t need to run on commodity hardware. Maybe I am cynical, but I dont see the incentive for Meta to contribute an open model.

To be fair, they contributed Pytorch which has defined a whole industry and is responsible for creating hundreds of billions of dollars in value or more. Contributing a set of model weights is an extremely minor thing in the shadow of that, so wouldn't exactly be uncharacteristic.

That is a fair point.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#22

Post by oceanplexian »

ntonozzi wrote: post #13
LrnByTeach wrote: post #3

This Falcon-40B royalty free license may force Meta ... that LLama-7B/13B may soon be fully open sourced as Meta wants open source LLM advancements and contributions on its own LLM architecture.

Why do people think that Meta released their model in order to get open source coders to improve their models? They will get absolutely no competitive advantage from this. Every other team developing a closed source LLM can easily copy the innovations that open source coders have applied to Llama on their own, closed source models. There's no advantage here. Meta just spent $10 million on releasing fun chaos into the…

Meta's most valuable asset is their users, not their technology, so giving away technology is incidental to them. It's not the UI or superior features that makes Meta, Instagram, etc such powerful platforms, it's the network effect.

ChatGPT was the fastest growing app in history, leaders at Meta (The ones who do M&A, strategy, etc) probably raised an eyebrow. They don't really give a crap about some stupid talking chatbot, but OpenAI getting smart and building a Social Network around millions of brand new users could be an existential problem for them. When Lecun wanted to OSS it they were probably like, sure, we can kill a few birds with one stone. If LLMs are a commodity that stops OpenAI and Google before they even get off the ground.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#23

Post by takeiteasyy »

mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

This will require a cpp port first to run on e.g. apple silicon?

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#24

Post by bongobingo1 »

mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#25

Post by rovr138 »

takeiteasyy wrote: post #23
mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

This will require a cpp port first to run on e.g. apple silicon?

This is using huggingface. It's python. There's no reason it wouldn't run on arm.

https://huggingface.co/tiiuae/falcon-40b

I'll also add that the fact something is in C++ doesn't mean it will run on arm or that it can be compiled in it.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#26

Post by fareesh »

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario.

From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries.

I played around with the smaller Llama/Alpaca models and it wasn't really viable to build anything with.

Not really seeing a use-case for fine-tuning either compared to just few-shot prompting.

Can someone fill me in on what I'm missing? It feels like I'm out of the loop

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#27

Post by smoldesu »

jacooper wrote: post #20
convexstrictly wrote: post #4

Earlier quoted context omitted.

Yann LeCun: "No. But it's not because we don't want to. It's because of complicated legal issues." https://twitter.com/ylecun/status/1651782621540524032

Complicated legal issues of having the cake and eating it too.

And also the potential, unprecedented legal issues that accompany releasing and defending a free/open model. The brownie points they'd receive aren't worth it, at least yet.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#28

Post by smoldesu »

fareesh wrote: post #26

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smal…

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month!

[0] https://github.com/go-skynet/LocalAI

I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#29

Post by paulgb »

smoldesu wrote: post #28
fareesh wrote: post #26

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smal…

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Now I’m curious what your bot does!

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#30

Post by anon373839 »

fareesh wrote: post #26

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smal…

Fine-tuning is a much better proposition than you’re giving it credit for. Papers are coming out demonstrating that 7B parameter models can outperform GPT-4’s quality when trained on a limited set of tasks. Yet, a 7B model offers comparatively cheap and fast inference. Furthermore, for a lot of use cases, few-shot prompting is infeasible because you need to supply 2-3k tokens worth of few-shot examples with every prompt in order to fully specify the behavior you want. (As an example, think of long-form summarization where you want the summary to adhere to certain rules.)
Post reply on HN