Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source LLM with 32k Context Length

blog.abacus.ai

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Open source LLM with 32k Context Length

#21

Post by syntaxing »

vekker wrote: post #13

It's probably too new for anyone to have integrated this into text-generation-webui / Gradio? I've been looking for a large context LLM (self-hosted or not) for a project, and as a European I unfortunately don't have access to Anthropic's Claude API yet.

How high context do you need? There’s a couple 16K models out there now. Some people have their own 32K ones too but the quality vary. It’s worth trying them on huggingface. The easiest way is to track TheBloke’s work to see any new models that come out.

Re: Open source LLM with 32k Context Length

#22

Post by woadwarrior01 »

alsodumb wrote: post #2

Abacus always seemed to me like a 'we got a lot of VC money with inflated claims now we gotta show we do everything' company. I don't really understand what they do, they seem to offer everything but I don't see anyone talking about using their offerings in the real-world. Ever. The only time I see mentions of the company are when I am targeted with ads or promoted posts of the founder.

Their CEO made a post[1] on Twitter claiming to have invented "the world's first commercially usable 32K long-context open-source LLM", which IMO is pure hyperbole.

It looks like the first OSS 13B Llama 2 based 32k token context model[2], but the first OSS and commercially usable 32k token context model was a 7B Llama 2 based model[3] from Together AI, who beat them by about a week[4].

[1]: https://twitter.com/bindureddy/status/1694126931174977906

[2]: https://huggingface.co/abacusai/Giraffe-v2-13b-32k

[3]: https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instr...

[4]: https://twitter.com/togethercompute/status/16925744231638470...

Re: Open source LLM with 32k Context Length

#23

Post by nemoniac »

sillysaurusx wrote: post #6
supermatt wrote: post #3

It seems this is built on LLAMA. Did meta change the license to make it open source now? It still seems to be showing otherwise in the repo. Edit: No mention of it being open source in the linked article. Maybe the title here is just wrong? @dang

It’s not possible to have a license over an ML model trained on other peoples’ works, since such models are uncopyrightable. They’re more like a phone book; a collection of facts trained by an entirely un-creative process. https://news.ycombinator.com/item?id=36691050 This hasn’t been proven in court, but it seems the most likely outcome.

Not saying that this applies to LLMs but if you describe them as "a collection of facts [collected and] trained by an entirely un-creative process" then it begins to sound like one could argue for Database Right.

https://en.wikipedia.org/wiki/Database_right

Re: Open source LLM with 32k Context Length

#24

Post by yreg »

keyle wrote: post #12
ImprobableTruth wrote: post #5

Earlier quoted context omitted.

Llama 2 is open source-ish. Weights are freely available and can be commercially used, but only if you have less than 700m users and agree to some "don't do naughty things" terms.

"open source-ish" sounds like the perfect way to massively profitable future litigations. Also "don't do naughty things", is there a chart for that? How is that defined, is it part of the non-existing license?

> How is that defined, is it part of the non-existing license?

What do you mean? https://github.com/facebookresearch/llama/blob/main/LICENSE

Re: Open source LLM with 32k Context Length

#25

Post by weinzierl »

This is just another fine-tuned LLaMA and Llama 2, like there are already some. I doubt that this will give seriously meaningful results for long context inference.

32k context length sounds nice of course, and it seems to be common to call the just fine-tuned models like that. I think it is more of a marketing thing and we really should distinguish between the context length of the pre-trained model and the fine-tuned model, with the latter being the default meaning of context length.

Re: Open source LLM with 32k Context Length

#26

Post by weinzierl »

Havoc wrote: post #16

Does anyone know if larger context lengths are inherent worse at other task? i.e. all other things being equal is a 8k model better at math than a 32k model

They are more resource (time and memory) intensive in training and inference, that is their disadvantage. For a fair comparison you would have to compare a 8k to a 32k pre-trained model with otherwise similar hyper-parameters.

OP is about a 32k sugar-coated Llama 2, so I would expect it be similar in performance to other Llama 2 derivatives.

Re: Open source LLM with 32k Context Length

#27

Post by kordlessagain »

weinzierl wrote: post #25

This is just another fine-tuned LLaMA and Llama 2, like there are already some. I doubt that this will give seriously meaningful results for long context inference. 32k context length sounds nice of course, and it seems to be common to call the just fine-tuned models like that. I think it is more of a marketing thing and we really should distinguish between the context length of the pre-trained model and the fine-tun…

These 800 watt speakers are great. So loud.

Re: Open source LLM with 32k Context Length

#28

Post by semi »

weinzierl wrote: post #26
Havoc wrote: post #16

Does anyone know if larger context lengths are inherent worse at other task? i.e. all other things being equal is a 8k model better at math than a 32k model

They are more resource (time and memory) intensive in training and inference, that is their disadvantage. For a fair comparison you would have to compare a 8k to a 32k pre-trained model with otherwise similar hyper-parameters. OP is about a 32k sugar-coated Llama 2, so I would expect it be similar in performance to other Llama 2 derivatives.

Is the increased resource usage inherent to the model or does it only happen when using the extra context? Like if your workflow currently fits in a 2k model would an 8k model be objectively worse and only worth using once you've filled the context up of a smaller model? Or would it be worth always using an 8k context model and just knowing it will get slower and more resource hungry as your context grows?

Sorry for the random question, I've just been curious about this for a while and unable to find out and you seem knowledgeable about these extended models.

Re: Open source LLM with 32k Context Length

#29

Post by inciampati »

sillysaurusx wrote: post #6
supermatt wrote: post #3

It seems this is built on LLAMA. Did meta change the license to make it open source now? It still seems to be showing otherwise in the repo. Edit: No mention of it being open source in the linked article. Maybe the title here is just wrong? @dang

It’s not possible to have a license over an ML model trained on other peoples’ works, since such models are uncopyrightable. They’re more like a phone book; a collection of facts trained by an entirely un-creative process. https://news.ycombinator.com/item?id=36691050 This hasn’t been proven in court, but it seems the most likely outcome.

Keep working on it! A good court case and a landmark decision about this could change the landscape for the market for these models.

Re: Open source LLM with 32k Context Length

#30

Post by smcleod »

kordlessagain wrote: post #27
weinzierl wrote: post #25

This is just another fine-tuned LLaMA and Llama 2, like there are already some. I doubt that this will give seriously meaningful results for long context inference. 32k context length sounds nice of course, and it seems to be common to call the just fine-tuned models like that. I think it is more of a marketing thing and we really should distinguish between the context length of the pre-trained model and the fine-tun…

These 800 watt speakers are great. So loud.

“Better than lossless”
Post reply on HN