Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

blog.salesforceairesearch.com

Post reply on HN
Print view View original

51–60 of 96 posts

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#51

Post by ttt3ts »

brucethemoose2 wrote: post #21
TOMDM wrote: post #4

From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use cases do people have for these smaller LLM's?

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…

Do you know a dataset for fine-tuning roleplaying?

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#52

Post by minimaxir »

ttt3ts wrote: post #50
aryamaan wrote: post #35

Earlier quoted context omitted.

would appreciate resources on fine tuning

https://huggingface.co/blog/stackllama You can easily finetune 7B or 15B LORA model with that on consumer GPUs.

That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs.

It's a rabbithole, and unfortunately there's no good shortcuts.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#53

Post by Solvency »

minimaxir wrote: post #52
ttt3ts wrote: post #50

Earlier quoted context omitted.

https://huggingface.co/blog/stackllama You can easily finetune 7B or 15B LORA model with that on consumer GPUs.

That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs. It's a rabbithole, and unfortunately there's no good shortcuts.

Why have there been thousands of overnight AI/GPT startups and products in the last few months and NOT a single simple intuitive "fine tuning wizard" app? That seems like such an obvious glaring gap.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#54

Post by minimaxir »

jxy wrote: post #49
brucethemoose2 wrote: post #23

> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#55

Post by minimaxir »

Solvency wrote: post #53
minimaxir wrote: post #52

Earlier quoted context omitted.

That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs. It's a rabbithole, and unfortunately there's no good shortcuts.

Why have there been thousands of overnight AI/GPT startups and products in the last few months and NOT a single simple intuitive "fine tuning wizard" app? That seems like such an obvious glaring gap.

Because the ChatGPT API (and analogous competitors) is cheap enough that it's both faster and more cost effective to just use it instead instead of using your own model, with maybe some shenanigans to handle its shortcomings without increasing cost much if at all.. And that was before gpt-3.5-turbo-0613, which dropped the price more and is about 2-3x faster.

There are startups that do finetuning on your own data, but with zero hints on how to preprocess your data and absurd costs (both upfront training and GPUs for serving inference) that's it's extremely difficult to argue from a customer business perspective compared to just using an API.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#57

Post by CamperBob2 »

deedree wrote: post #29
seeknotfind wrote: post #8

Earlier quoted context omitted.

Why not ask the AIs themselves? They are pretty good at explaining this type of thing.

But if you don't know a certain amount about a subject already you won't know when it's lying to you. That would probably be the case here.

It's basically what an interactive dialogue with Wikipedia would look like, which is still a darned useful thing.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#58

Post by hyperliner »

minimaxir wrote: post #55
Solvency wrote: post #53

Earlier quoted context omitted.

Why have there been thousands of overnight AI/GPT startups and products in the last few months and NOT a single simple intuitive "fine tuning wizard" app? That seems like such an obvious glaring gap.

Because the ChatGPT API (and analogous competitors) is cheap enough that it's both faster and more cost effective to just use it instead instead of using your own model, with maybe some shenanigans to handle its shortcomings without increasing cost much if at all.. And that was before gpt-3.5-turbo-0613, which dropped the price more and is about 2-3x faster. There are startups that do finetuning on your own data, but…

[dead]

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#59

Post by imrehg »

jxy wrote: post #49
brucethemoose2 wrote: post #23

> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

Looks like this is the limitation why it cannot just be plugged into llama.cpp (doesn't have a saved tokeniser model, and I'm not sure how would one go about creating one). Otherwise it would be cool to try it out, the metrics in the article are promising to have something running locally on an M1 Mac...

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#60

Post by imrehg »

minimaxir wrote: post #54
jxy wrote: post #49

Earlier quoted context omitted.

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...

Those look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )
Post reply on HN