Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama Ecosystem: Past, Present, and Future

ai.meta.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: The Llama Ecosystem: Past, Present, and Future

#21

Post by [deleted] »

version_five wrote: post #18
refulgentis wrote: post #17

Earlier quoted context omitted.

This reminds me of a comment elsewhere I also replied to today: it's sort of hard to even pretend I have global usage stats, so I won't. There's a certain type of myopia that leads to overindexing on llama.cpp that makes it easy to classify. to wit: > not aware of a competing format for quantized models ONNX, that's how its done in prod and on other models besides (and including) LLaMa. Quantization is a general tech…

> This reminds me of a comment elsewhere I also replied to today Right, looks like you made fun of / were condescendingly dismissive of my comment in another thread, I wouldn't have replied here if I'd realized you were the same person.

[deleted]

Re: The Llama Ecosystem: Past, Present, and Future

#23

Post by kkielhofner »

version_five wrote: post #10
refulgentis wrote: post #7

Earlier quoted context omitted.

> It's a (the?) major reason for llamas popularity. Absolutely not. There's a corner of the overall community that hovers it and overperceives it as everyone else only uses it too. Its great if you have an Apple ARM machine and want to see an M2 Pro do 10 tokens/sec (and what could make an Apple ARM have 30 minute battery life). I also doubt it's a slight, the only callouts are large commercial collaborations, ex. nV…

I'd be curious if you have any hard data about use. Mine is anecdotal too, but I see that llama.cpp is the very close second highest starred repo with llama on the name, after meta llama. Additionally, all the HF models seem to have ggml / gguf quantized versions . I'm not aware of a competing format for quantized models. There are also python bindings which are used in a lot of projects. What is a competing framewor…

The “original” and by far most common format for quantization is GPTQ.

AWQ support is spreading more, which is nice.

Re: The Llama Ecosystem: Past, Present, and Future

#24

Post by rgbrgb »

jcmontx wrote: post #22

May I ask what are you all doing running a LLM locally?

cooking meth, making molotov cocktails, discussing my medical history, sex

ok seriously though I had fun over the weekend chatting with Samantha on a long car ride on my MacBook. We were mostly asking about history.

Re: The Llama Ecosystem: Past, Present, and Future

#25

Post by rgbrgb »

refulgentis wrote: post #7
version_five wrote: post #2

They didn't mention llama.cpp or show it in their picture, that's hopefully an oversight, it feels like a major slight. It's a (the?) major reason for llamas popularity. I have mixed feelings, llama is great but it's perpetuated it's shitty license. They could have done so much more good if they'd used gpl style licensing, instead they basically subverted open source, using an objectively good model as leverage.

> It's a (the?) major reason for llamas popularity. Absolutely not. There's a corner of the overall community that hovers it and overperceives it as everyone else only uses it too. Its great if you have an Apple ARM machine and want to see an M2 Pro do 10 tokens/sec (and what could make an Apple ARM have 30 minute battery life). I also doubt it's a slight, the only callouts are large commercial collaborations, ex. nV…

fwiw I get more like 35-40 tokens/sec on my m1 macbook with a 7B model. That's way faster than I can read or skim. If we can figure out how to focus the expertise in small models, I don't see why it wouldn't be viable for those of us that don't want to share all of our convos with big tech.

Re: The Llama Ecosystem: Past, Present, and Future

#26

Post by 8note »

kordlessagain wrote: post #14
version_five wrote: post #2

They didn't mention llama.cpp or show it in their picture, that's hopefully an oversight, it feels like a major slight. It's a (the?) major reason for llamas popularity. I have mixed feelings, llama is great but it's perpetuated it's shitty license. They could have done so much more good if they'd used gpl style licensing, instead they basically subverted open source, using an objectively good model as leverage.

A lot of times there can be a feeling of being wrong without it being intentional. In this case I think the mention of AWS being a partner shows intent to put value behind what they are doing for their stakeholders. The license for Llama 2 is pretty intense, but mirrors that intent by limiting interactions with individuals at scale, as well as limiting anything learned from the model through inference in being used t…

Surely it's IP the shareholders have licensed, rather than their own IP.

Aka, my own comments being sublicensed back to me, after I licenced them to Facebook.

Re: The Llama Ecosystem: Past, Present, and Future

#27

Post by danShumway »

1vuio0pswjnm7 wrote: post #20

https://www.theregister.com/2023/07/21/llama_is_not_open_sou...

This is an important drum to continue to beat, but it needs to be paired with the caveat that we are not legally certain that Llama's weights are actually copyrightable. We're also not certain how much IP protections around trade secrets would apply to weights in this situation. A lot of that is uncertain.

Llama is not Open Source but until we get a court case ruling one way or the other we don't know if it's actually locked-down in the way Facebook intends; and I want to strike a balance between (correctly) pointing out that Facebook is misusing the Open Source label while not ceding to Facebook's claims about how much it can legally constrain people who have never signed a single Llama TOS.

Re: The Llama Ecosystem: Past, Present, and Future

#28

Post by refulgentis »

version_five wrote: post #18
refulgentis wrote: post #17

Earlier quoted context omitted.

This reminds me of a comment elsewhere I also replied to today: it's sort of hard to even pretend I have global usage stats, so I won't. There's a certain type of myopia that leads to overindexing on llama.cpp that makes it easy to classify. to wit: > not aware of a competing format for quantized models ONNX, that's how its done in prod and on other models besides (and including) LLaMa. Quantization is a general tech…

> This reminds me of a comment elsewhere I also replied to today Right, looks like you made fun of / were condescendingly dismissive of my comment in another thread, I wouldn't have replied here if I'd realized you were the same person.

LOL I was thinking of an entirely different comment on another site. Give me credit here, I never cast aspersions on you, or even addressed you directly here.

I apologize for making you feel condescended to, but also would like to point out the _mean_ comment is +7, much less this one: there's a pretty significant gap in your knowledge and reality is going to keep intruding. Engaging in public is a wonderful way to learn, but you're coming across as glib and assertive and uninformed. You thought llama.cpp invented quantization and there's no other real format? :X

Re: The Llama Ecosystem: Past, Present, and Future

#29

Post by refulgentis »

kkielhofner wrote: post #23
version_five wrote: post #10

Earlier quoted context omitted.

I'd be curious if you have any hard data about use. Mine is anecdotal too, but I see that llama.cpp is the very close second highest starred repo with llama on the name, after meta llama. Additionally, all the HF models seem to have ggml / gguf quantized versions . I'm not aware of a competing format for quantized models. There are also python bindings which are used in a lot of projects. What is a competing framewor…

The “original” and by far most common format for quantization is GPTQ. AWQ support is spreading more, which is nice.

Again, for a subset of the local LLM community. Quantization was not invented on Github, by llama.cpp, for LLMs in 2023.
Post reply on HN