Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

blog.salesforceairesearch.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#41

Post by MINIMAN10000 »

aryamaan wrote: post #37
m00x wrote: post #33

Earlier quoted context omitted.

You can run big models on the cloud yourself, or with a 3090/4090 quantized. You don't have to go to openai.

What’re some models and hardware combos we can run now? I am avoiding to go to OpenAI with my office’s stuff and can use some gpu(s)

You would just need a computer which can fit 2 3090s in order to run those to run something like TheBloke/airoboros-65B-gpt4-1.3-GPTQ

https://www.reddit.com/r/LocalLLaMA/wiki/models/ gives you a list of VRAM requirements to load the model into GPU VRAM. the more VRAM the computer has, the larger the model you can load in, thus making 3090s the current consumer grade king due to price to max VRAM.

This being said however most models are LLAMA based which all fall under that specific research license.

So following the rules, you would be limited to a subset of models which are foundational models which allow for commercial use

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#42

Post by galaxyLogic »

ipaddr wrote: post #32
ftxbro wrote: post #22

Earlier quoted context omitted.

The use case is people who got some fear of missing out on AI and also they are not capable of seeing the difference between the model outputs themselves. Maybe you will say I'm elitist but it's disturbing to me how many people literally can't tell the difference between gpt 3 and gpt 4 outputs, it's like the "It's the same picture" meme. Maybe that makes me a hipster gpt connoisseur elitist that I can recognize and…

It doesn't make you an elitist hipster. It puts you in a category of most people on earth. An elist hipster would swear by a custom smaller parameter model finetuned by a wizard for specific tasks and have a spell book of models.

Right, that's the Way of the Hipster

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#43

Post by brucethemoose2 »

aryamaan wrote: post #35
brucethemoose2 wrote: post #21

Earlier quoted context omitted.

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…

would appreciate resources on fine tuning

Honestly, I dunno. I think most people are using lit-llama or EasyLM (on TPUs) for finetuning?

QLORA is the gold standard for more affordable training.

As for datasets, just look at the open datasets the best-in-class models are using, like Vicuna or https://huggingface.co/NousResearch/Nous-Hermes-13b

Some model datasets like Manticore, Chronos or the infamous Pygmalion are more "secretive," but you can find the dataset gathering scripts on Github or in community chats.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#44

Post by brucethemoose2 »

aryamaan wrote: post #37
m00x wrote: post #33

Earlier quoted context omitted.

You can run big models on the cloud yourself, or with a 3090/4090 quantized. You don't have to go to openai.

What’re some models and hardware combos we can run now? I am avoiding to go to OpenAI with my office’s stuff and can use some gpu(s)

I can easily run LLaMA 13B on my 6GB VRAM/16GB RAM laptop using llama.cpp (specifically Kobold.cpp as the frontend).

I can barely run 33B, but anything more than 800 context and I oom. But it would run very comfortably on a bigger GPU or a 24GB+ laptop.

Theoretically some phones can comfortably handle 13B on mlc-llm though in practice its not really implemented yet.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#45

Post by scubbo »

sacnoradhq wrote: post #38
esafak wrote: post #25

Earlier quoted context omitted.

It's an open source language model with seven billion parameters (a measure of its complexity), and a longer than typical sequence length of 8K, which allows the provision of more context when querying the model. For example, it allows you to better generate text in someone else's voice by providing a longer example of their work. https://en.wikipedia.org/wiki/Foundation_models

The number of parameters seems meaningless when the training sets are dogshit and hardened old gum chipped from the shoes of Gregslist and FleaBay. OTOH, there ought to be a construction in the form of an web app that can pinch out nonrepetitive, coherent ~100 page trashy romance novels in the style of any author given name with open source or specific text(s) or transcripts with enough original input volume: Churchi…

Do you have any recommendations for cryptocoins?

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#46

Post by Auracle »

wokwokwok wrote: post #36
jrflowers wrote: post #9

Earlier quoted context omitted.

> What use cases do people have for these smaller LLM's? None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.

yep, that’s why it’s free. If it was good , then they’d charge for it. …what they (and everyone) is gonna do is play with smallish models to iterate on the process for relatively small expense and earn karma. Then pay big $$$ to make a really good model for internal use and/or an api that people have to pay for. Tldr; it’s free. It’s by sales force. You should expect it to be a) crippled and b) a loss leader for a pa…

To be fair, Stable Diffusion (especially the upcoming SDXL) are good and free.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#47

Post by jrflowers »

wokwokwok wrote: post #36
jrflowers wrote: post #9

Earlier quoted context omitted.

> What use cases do people have for these smaller LLM's? None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.

yep, that’s why it’s free. If it was good , then they’d charge for it. …what they (and everyone) is gonna do is play with smallish models to iterate on the process for relatively small expense and earn karma. Then pay big $$$ to make a really good model for internal use and/or an api that people have to pay for. Tldr; it’s free. It’s by sales force. You should expect it to be a) crippled and b) a loss leader for a pa…

> If you want a good free open model, you’re kidding yourself if you think a corporate giant is going to kiss you on the head and give it to your for free.

Yep! That makes sense!

I would love to be the CEO of the company that does give away an actually useful model and little forehead kisses though. The amount of goodwill that one would generate from that would be astronomical and training costs are getting so low that nearly any company with enough cash could do it.

I look forward to waking up and hearing that the Nabisco/Canadian Tire/A&W usefully-tuned model is revolutionizing the economy and seeing the infinite amount of good press that it would generate.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#48

Post by SparkyMcUnicorn »

aryamaan wrote: post #35
brucethemoose2 wrote: post #21

Earlier quoted context omitted.

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…

would appreciate resources on fine tuning

This and/or text-generation-webui training doc are a good place to start.

https://github.com/zetavg/LLaMA-LoRA-Tuner

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#49

Post by jxy »

brucethemoose2 wrote: post #23

> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

Re: XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

#50

Post by ttt3ts »

aryamaan wrote: post #35
brucethemoose2 wrote: post #21

Earlier quoted context omitted.

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…

would appreciate resources on fine tuning

https://huggingface.co/blog/stackllama

You can easily finetune 7B or 15B LORA model with that on consumer GPUs.

Post reply on HN