Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

blog.salesforceairesearch.com

Post reply on HN
Print view View original

21–30 of 96 posts

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#21

Post by brucethemoose2 »

TOMDM wrote: post #4

From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use cases do people have for these smaller LLM's?

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion.

The metrics are good though, perhaps placing this closer to 13B.

And 8K context is huge. When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM consumption for a ~13B model.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#22

Post by ftxbro »

TOMDM wrote: post #4

From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use cases do people have for these smaller LLM's?

The use case is people who got some fear of missing out on AI and also they are not capable of seeing the difference between the model outputs themselves. Maybe you will say I'm elitist but it's disturbing to me how many people literally can't tell the difference between gpt 3 and gpt 4 outputs, it's like the "It's the same picture" meme. Maybe that makes me a hipster gpt connoisseur elitist that I can recognize and have opinions about those differences between versions and I hate using the small ones and I think only the very biggest ones are good.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#23

Post by brucethemoose2 »

> The training recipe and model architecture follow LLaMA

This is huge.

MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#24

Post by steppi »

TOMDM wrote: post #4

From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use cases do people have for these smaller LLM's?

I did some old school NLP before, but don’t really work in the field anymore. As a generative model. As a general purpose generative model, maybe this isn’t very useful. As a foundation to make models to perform text classification and information extraction tasks, this could be very useful. For these kinds of tasks, you can still get good results with the classic bag of words type approaches people were using 30 years ago even. I remember when transformers first came out, limitations in sequence size made them unusable for some classification tasks.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#25

Post by esafak »

DanAtC wrote: post #5

I have no idea what any of these words mean, but I'd like to. Can someone point me in the direction of an "AI for Dipshits"?

It's an open source language model with seven billion parameters (a measure of its complexity), and a longer than typical sequence length of 8K, which allows the provision of more context when querying the model. For example, it allows you to better generate text in someone else's voice by providing a longer example of their work.

https://en.wikipedia.org/wiki/Foundation_models

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#26

Post by brucethemoose2 »

Also, their metric table is very interesting. It shows Falcon 7B and OpenLlama 7B much less favorably than other evaluations (including the HuggingFace leaderboard, which I am kinda suspicious of), and instruct benchmarks like that aren't seen as much.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#28

Post by bratao »

foolfoolz wrote: post #2

when will the llm race peak? have we peaked already?

I think we are only at the beginning. Call it first generation. LLaMA was even missing many improvements that existed before it (xPos, Multi-Query Attention, Blockwise Parallel Transformer)

Many researchers are improving very fast, and I would bet that soon we will see more efficient LLMs.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#29

Post by deedree »

seeknotfind wrote: post #8
DanAtC wrote: post #5

I have no idea what any of these words mean, but I'd like to. Can someone point me in the direction of an "AI for Dipshits"?

Why not ask the AIs themselves? They are pretty good at explaining this type of thing.

But if you don't know a certain amount about a subject already you won't know when it's lying to you. That would probably be the case here.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#30

Post by happycube »

rasz wrote: post #10
foolfoolz wrote: post #2

when will the llm race peak? have we peaked already?

Nobody doing xB models is participating in any AI races, at this point those are useless toys with garbage output.

How far we've come from GPT2...
Post reply on HN