Falcon 40B LLM (which beats Llama) now Apache 2.0 • Hacker News Board

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#51

Post by anon373839 » Thu, Jun 01, 2023, 8:30 AM UTC

sgu999 wrote: post #49

anon373839 wrote: post #30

Earlier quoted context omitted.

Fine-tuning is a much better proposition than you’re giving it credit for. Papers are coming out demonstrating that 7B parameter models can outperform GPT-4’s quality when trained on a limited set of tasks. Yet, a 7B model offers comparatively cheap and fast inference. Furthermore, for a lot of use cases, few-shot prompting is infeasible because you need to supply 2-3k tokens worth of few-shot examples with every pro…

What kind of tasks? Could you give some links to the papers you're referring to?

Sure, here are two:

1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

https://huggingface.co/papers/2305.14201

2. Gorilla: Large Language Model Connected with Massive APIs

https://arxiv.org/abs/2305.15334

Consider also these 2 papers supporting the feasibility of fine-tuning:

3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base model]

https://arxiv.org/abs/2305.11206

4. QLoRA: Efficient Finetuning of Quantized LLMs [showing that LLMs can now be fine-tuned quickly on consumer-grade GPUs]

https://arxiv.org/abs/2305.14314

—-

Adding up these developments (all of which occurred during the span of one week), I don’t see how huge, slow, general-purpose models maintain their relevance in the long term, when a lean, domain-focused model is right there within reach of every application developer.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#52

Post by fareesh » Thu, Jun 01, 2023, 10:45 AM UTC

anon373839 wrote: post #51

sgu999 wrote: post #49

Earlier quoted context omitted.

What kind of tasks? Could you give some links to the papers you're referring to?

Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base mode…

Sorry I should have been more specific - I was limiting my question to the bigger models. The smaller (~7B) models are feasible with these approaches.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#53

Post by fareesh » Thu, Jun 01, 2023, 10:48 AM UTC

smoldesu wrote: post #28

fareesh wrote: post #26

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smal…

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

TIL Oracle has VPS offerings with a free tier. Are they any good? Is the free-tier time limited?

This use-case is alright for a toy I guess - which is the extent that I was originally expecting these things to be useful for.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#54

Post by anon373839 » Thu, Jun 01, 2023, 11:56 AM UTC

fareesh wrote: post #52

anon373839 wrote: post #51

Earlier quoted context omitted.

Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base mode…

Sorry I should have been more specific - I was limiting my question to the bigger models. The smaller (~7B) models are feasible with these approaches.

Ah. Perhaps the larger models will find use in in-house deployments where companies want their employees to have access to ChatGPT-like general purpose assistants, but want to prevent data from leaving their premises. LIMA shows the LLaMA 65B hitting a quality level somewhere between DaVinci-003 and GPT-4 with minimal alignment, so the base models are probably powerful enough already for this to work. Just speculating.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#55

Post by mliker » Thu, Jun 01, 2023, 12:05 PM UTC

oceanplexian wrote: post #22

ntonozzi wrote: post #13

Earlier quoted context omitted.

Why do people think that Meta released their model in order to get open source coders to improve their models? They will get absolutely no competitive advantage from this. Every other team developing a closed source LLM can easily copy the innovations that open source coders have applied to Llama on their own, closed source models. There's no advantage here. Meta just spent $10 million on releasing fun chaos into the…

Meta's most valuable asset is their users, not their technology, so giving away technology is incidental to them. It's not the UI or superior features that makes Meta, Instagram, etc such powerful platforms, it's the network effect. ChatGPT was the fastest growing app in history, leaders at Meta (The ones who do M&A, strategy, etc) probably raised an eyebrow. They don't really give a crap about some stupid talking ch…

I like how you completely hallucinated a story about Meta, Lecun, and ChatGPT

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#56

Post by senttoschool » Thu, Jun 01, 2023, 1:50 PM UTC

fareesh wrote: post #26

In terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smal…

>From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries.

That's because we're only half a year into LLMs becoming mainstream. Give it 3-4 years. The advancements in bringing down model size, optimizations, and newer GPUs, SoCs from Nvidia, AMD, Apple, Intel, Qualcomm, etc will make it so that top LLMs will run on a highend laptop/desktop.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#57

Post by smoldesu » Thu, Jun 01, 2023, 2:31 PM UTC

fareesh wrote: post #53

smoldesu wrote: post #28

Earlier quoted context omitted.

I'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

TIL Oracle has VPS offerings with a free tier. Are they any good? Is the free-tier time limited? This use-case is alright for a toy I guess - which is the extent that I was originally expecting these things to be useful for.

They're okay. This isn't the place for a full review of their offerings (especially considering everyone's mixed feelings on Oracle), but I'm confident that it's better than most 1core/$5 deals you'll find elsewhere.

> Are they any good?

Yep, free tier allows you to spec up to 24gb of RAM without paying, which is cool. The bottleneck is really the disk speed, but that's not an issue with mmaped models. There's enough cached memory that it loads instantly, so it's good-ish for this use case.

> Is the free-tier time limited?

No, but there are a lot of strings attached:

- The cores are vCPUs, not dedi (duh)

- You can't create new instances when demand is high (unless you add a credit card)

- Technically Oracle reserves the right to shut down the instance if demand gets really high (although I haven't heard any stories about this personally)

Proceed with caution. It's still a great place to start before you shell out $1/hr for dedi GPU rackspace.

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#58

Post by brianjking » Thu, Jun 01, 2023, 2:33 PM UTC

mikeravkine wrote: post #16

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!

How can it be deployed on a Huggingface Space or Colab notebook?

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#59

Post by groby_b » Thu, Jun 01, 2023, 4:21 PM UTC

BonoboIO wrote: post #33

bongobingo1 wrote: post #24

Earlier quoted context omitted.

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

Will be interesting to see if someone comes up with an ASIC or FPGA.

Why would you think there's an ASIC/FPGA design that significantly improves over GPUs specifically targeted at running large models already? Where's the win?

The fundamental limit for hardware acceleration are number of gates you can squeeze on a die, right now. (Or, alternatively. memory bandwidth)

Re: Falcon 40B LLM (which beats Llama) now Apache 2.0

#60

Post by sgu999 » Thu, Jun 01, 2023, 6:50 PM UTC

anon373839 wrote: post #51

sgu999 wrote: post #49

Earlier quoted context omitted.

What kind of tasks? Could you give some links to the papers you're referring to?

Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base mode…

Wow, thanks a lot! :)