Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

blog.salesforceairesearch.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#81

Post by knaik94 »

profsummergig wrote: post #62

If someone could elucidate on what these phrases signify, I'd be very grateful: 1) 7B foundational model 2) 8K length 3) 1.5T tokens

1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" (instruct) structure, and it has 7 billion weights/biases. It's the working size of the model.

2) Currently every model that can run locally was trained with a 2K context size. It's a hard limit on prompt length. There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras. This base model was trained with 8k sequences.

3. 1.5T tokens is the size of the total training corpus. Training cost and time increases with training size. [B]

A. https://arxiv.org/abs/2306.15595

B. https://www.semianalysis.com/p/the-ai-brick-wall-a-practical... (Jan 2023)

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#82

Post by numeri »

minimaxir wrote: post #56

Per the validation perplexity chart shown, the 8K length model performs better than the 4K length model even at <4K length, so why are they even offering the 4K model if the 8K is strictly better?

I'd say for research purposes. HackerNews seems to tend to mostly represent the LLM consumer viewpoint, but these waves of models being released are honestly more interesting from a research than a user point of view. As a LLM user, you're (vast generalization/simplification) really just interested in the best of N models, but as a researcher, I'm super interested in each model's performance and analyzing the reasons for any differences.

With this model (and they say this in the blog post), they were testing the hypothesis that training on a longer context size would provide more performance at the same parameter count/inference FLOPs. From a quick perusal of their post, it looks like this was true, and we should train all future models with as long of a context size as we can afford.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#83

Post by int_19h »

aryamaan wrote: post #37
m00x wrote: post #33

Earlier quoted context omitted.

You can run big models on the cloud yourself, or with a 3090/4090 quantized. You don't have to go to openai.

What’re some models and hardware combos we can run now? I am avoiding to go to OpenAI with my office’s stuff and can use some gpu(s)

llama-30B (which is actually 33B) and derivatives generally run fine with 4-bit quantization on a single RTX 3090 or 4090, although depending on group size used for quantization you may need to slightly dial down the context size.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#84

Post by artemonster »

Please recommend a good tutorial/book/video on modern LLMs and NNs in general, for programmers and technical people. Where you get the idea of how it works. Tried googling with dozens of queries and it just sucks, a lot of hand-wavy articles for lay people or some paid courses.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#85

Post by imtemplain »

Tostino wrote: post #66
brucethemoose2 wrote: post #21

Earlier quoted context omitted.

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…

You must have missed the memo... It's now super easy to extend the context of 2k llama models to 8k, 16k, or even 32k with just a small fine tune and a tweak to the code. You still need the memory to be able to go that high, but it's totally doable.

[dead]

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#86

Post by Sunhold »

thomasahle wrote: post #67
going_ham wrote: post #63

Earlier quoted context omitted.

1. The trained model has 7B parameters or weights for each neuron. 2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your tokens are characters then, "h", "e", "y" represent 3 tokens for hey. Most of the algos use byte pair encoding. For example "hand-le" has two tokens "hand" and "le". This is a very crud example which is enough to give the gist but is not accurate. You can look…

Don't just post ChatGPT answers as comments on hackernews. This one doesn't even make any sense. Of course it doesn't have 7B parameters _per_ neuron.

Doesn't look like ChatGPT. Grammatical errors like "on large data corpus," the poor comma usage, misspelling crude, etc. are more of a human thing.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#87

Post by visarga »

wokwokwok wrote: post #36
jrflowers wrote: post #9

Earlier quoted context omitted.

> What use cases do people have for these smaller LLM's? None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.

yep, that’s why it’s free. If it was good , then they’d charge for it. …what they (and everyone) is gonna do is play with smallish models to iterate on the process for relatively small expense and earn karma. Then pay big $$$ to make a really good model for internal use and/or an api that people have to pay for. Tldr; it’s free. It’s by sales force. You should expect it to be a) crippled and b) a loss leader for a pa…

I don't agree. LLaMA models are great if you want to run your own models on your own systems, but only if you fine-tune them to specific tasks. The problem is that LLaMA is non-commercial. There was a need for a small efficient pre-trained model to build on. This is what Salesforce released. It's not intended to be used with general purpose prompting like chatGPT.

The problems with chatGPT are many - dependence on third party, privacy, externally imposed ideology and rules, cost, and most importantly - prompting is context-size limited and token-expensive, you can't pack much data into it.

Fine-tuning is a more powerful approach where you can actually fix the model problems instead of futzing around with the prompt and demonstrations. Yes, you got to work on your dataset. But if you don't already have it you can bootstrap with GPT-4 for a small sum.

Meta provided the training wheels - LLaMA, every company tried fine-tuning it for their purposes, but could not proceed for lack of a commercial base model. Salesforce XGen and a few other open-small-LLMs (funny how that sounds!) open the flood gates.

So the recipe is: use an existing dataset, or make one with regular GPT-4 prompting and a bit of curation. Then fine-tune a small open model. You can get it to be better than stock GPT-4, cheaper, faster and private. If you use LoRA's you can save each skill in a separate diff model just 1% the size of the base model and use a single GPU to fine-tune it, in a single day.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#88

Post by nextaccountic »

TOMDM wrote: post #14
riku_iki wrote: post #11

Earlier quoted context omitted.

Bert had lots of use cases and this one is supposedly stronger model.

I've used BERT in a number of production apps, it feels like a very apples to oranges comparison given how the AI landscape has changed since BERT's release.

What did you use BERT for? Maybe 7B models are up to the task?

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#89

Post by going_ham »

thomasahle wrote: post #67
going_ham wrote: post #63

Earlier quoted context omitted.

1. The trained model has 7B parameters or weights for each neuron. 2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your tokens are characters then, "h", "e", "y" represent 3 tokens for hey. Most of the algos use byte pair encoding. For example "hand-le" has two tokens "hand" and "le". This is a very crud example which is enough to give the gist but is not accurate. You can look…

Don't just post ChatGPT answers as comments on hackernews. This one doesn't even make any sense. Of course it doesn't have 7B parameters _per_ neuron.

Sorry doc, I wrote that comment in a smartphone without putting any thought. What I wanted to say was: > there are 7B parameters. A parameter is a weight assigned to single neuron.

I hope this clarifies the answer now.

Now that is done I am quite curious on how you came up with the idea it was written by ChatGPT? I just wanted to simplify as best as I could. It’s funny you thought it that way.

What could I have done so that it didn’t sound like response from ChatGPT? I am asking it to prevent future misunderstandings. I thought my grammatical errors would be enough to show it wasn’t a ChatGPT response.

Looking forward to your reply!

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#90

Post by ttt3ts »

minimaxir wrote: post #52
ttt3ts wrote: post #50

Earlier quoted context omitted.

https://huggingface.co/blog/stackllama You can easily finetune 7B or 15B LORA model with that on consumer GPUs.

That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs. It's a rabbithole, and unfortunately there's no good shortcuts.

It isn't script kiddie level but it isn't hard I finetuned a 15B parameter reddit bot with an afternoon of time and a day of training on a 3090. Bot got a few thousand Karma in a couple of days before I turned it off (proof of concept done).

If all you have is an M1 or whatever, ya, you need a real workstation and depending on your use ChatGPT might be cheaper/better.

Post reply on HN