Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

381–390 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#381

Post by qup »

colechristensen wrote: post #322
Natsu wrote: post #312

Earlier quoted context omitted.

I remember a possibly apocryphal quote from a park ranger saying that there was a significant overlap between the smartest bears and the dumbest tourists.

I’m pretty sure I’ve seen this on some national park official instagram. The idea being they can’t build actually bear proof garbage containers because there will be people that can’t or won’t figure out how to use them. Experience dealing with raccoons confirms this… it’s difficult to contain dog food for a farm dog outside that doesn’t get broken into.

The only solution I've found is making the dog guard it. They climb to every place my cat can jump to.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#382

Post by lhl »

danShumway wrote: post #173
superkuh wrote: post #129

Earlier quoted context omitted.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the…

This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B). My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something aroun…

I'm pretty confident that the landscape is going to look very different by the end of the year, as there are so many people poking around now. I think that significantly smaller models will definitely be good enough for specialized tasks, but an equivalently tuned larger model will always be better, the question is by how much. On Meta's benchmarks [1], there's only a tiny gap between 30B and 65B, for example.

For 65B, GPTQ 4-bit should fit LLaMA 65B into 40GiB of memory. Currently the cheapest way to run that at an acceptable speed would be to use 2 x RTX 3090/4090s (~$2500-3000) or maybe a Jetson Orin 64GB (~$2000). I've seen people trying to run it on an M1 Max and it's just a bit too slow to comfortably use (I get a similar speed to when I try it on my 5950X - about 1-2 tokens/s), but it seems like it's within a factor or two of being fast enough, so not out of the question that it might get there just through software optimizations. I'd definitely upgrade to a 7950X/X3D or a Threadripper (w/ 96GB of DDR5-5200) if I could get 65B running at a comfortable speed all the time.

I think training is also advancing at a pretty good clip. LLaMA-adapter [2] is doing fine tuning of LLaMA 13B on a single 8xA100 system in 1h (so for ~$12 for a spot instance) and was already over 3X faster than Alpaca's training.

To me, the biggest thing limiting easy plug-and-play distribution is actually LLaMA's licensing issues, so maybe someone will offer a better open foundational model soon and the community can standardize on that. It'd be nice to have a larger context window (Flash Attention?) as well.

[1] https://github.com/facebookresearch/llama/blob/main/MODEL_CA...

[2] https://github.com/ZrrSkywalker/LLaMA-Adapter

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#383

Post by EugeneOZ »

travisjungroth wrote: post #362
kazinator wrote: post #345

Earlier quoted context omitted.

AI "thinks" the way a horse can count. The horse's interlocutor knows that the correct number is eight and shouts "right!" when the animal has stomped its hoof that many times, cuing it to stop. When you know the right answer, you're not actually using the AI to produce the answer; you're just prodding it to generate bullshit after bullshit until it hits upon that answer, or a very good facsimile. The one actually th…

This is just silly. You’re saying that these models are completely incapable of what they’re doing and are only getting to answers from cheating. You can see this isn’t true very quickly when using them. [Me] I want to make a bouquet to honor the home country of the first person to isolate Molybdenum. Be brief. [ChatGPT-4] To honor Peter Jacob Hjelm, the Swedish chemist who first isolated Molybdenum in 1781, create a…

Because you are using results of the training.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#384

Post by barking_biscuit »

Aeolun wrote: post #277
trifurcate wrote: post #174

Earlier quoted context omitted.

All you need is 2 3090s.

All you need is a few thousand dollars lying around to spend solely on your inference fun? I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).

>All you need is a few thousand dollars lying around to spend solely on your inference fun? I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).

Can confirm. Did a new build just for inference fun. Expensive, and worth it.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#385

Post by lpapez »

yunohn wrote: post #375
Gigachad wrote: post #260

Earlier quoted context omitted.

I feel like there would be a good chunk of real humans who would be incapable of answering a question like this.

The point of using a computer/AI is enable predictable and/or factual output. It’s not really a useful gotcha to say that “most humans would get this wrong”…

This used to be true with deterministic algorithms but not anymore. LLMs give you a "good enough" approximate answer which will replace humans in many scenarios previously inaccessible to computers.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#386

Post by int_19h »

MacsHeadroom wrote: post #354
int_19h wrote: post #249

Earlier quoted context omitted.

It doesn't matter if you merge the LoRA, the resulting weights are still a derived work - assuming, that is, that weights are copyrightable in the first place (which is still a big if).

If the resulting weights a derived work of LLaMA then LLaMA is a derived work of the illegally pirated Books3 dataset (a dataset of a private torrent tracker) used to train it. There's no way ML models can be protected under copyright.

The problem is that you need to risk getting sued first to prove that point. And hoping that you have deep enough pockets to outlast Meta's lawyers.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#387

Post by jonathanstrange »

lpapez wrote: post #385
yunohn wrote: post #375

Earlier quoted context omitted.

The point of using a computer/AI is enable predictable and/or factual output. It’s not really a useful gotcha to say that “most humans would get this wrong”…

This used to be true with deterministic algorithms but not anymore. LLMs give you a "good enough" approximate answer which will replace humans in many scenarios previously inaccessible to computers.

Yes, but the GP rightly pointed out that comparing AI capabilities to dumb humans is not very useful. After all, you wouldn't hire random people from the street to do your accounting either. As another example, I doubt that a lawyer's office would want to hire someone who cannot correctly answer the glass door puzzle, even if that person's main task was only summarizing texts.

To be fair, at the current pace of development I'd be surprised if the next iterations of GPT won't outperform most qualified humans, too.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#388

Post by bitL »

qeternity wrote: post #155
superkuh wrote: post #129

Earlier quoted context omitted.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the…

I agree with your premise: I have used 65b variants and of course they’re not as good as OpenAI. GPT3 has 175b parameters, and OpenAI has done more RLHF than anyone else. Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning? That said, it’s clear that replicating GPT4+ performance is within the resources of a number of large tech orgs. And the smaller…

> Why would we expect to get comparable performance with models a fraction of the size and a pittance of the fine tuning?

LLaMA incorporated new techniques that make 65B perform way better than GPT-3's 175B so the model size argument is not very strong.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#389

Post by jacquesm »

WalterBright wrote: post #209
jacquesm wrote: post #51

Earlier quoted context omitted.

There is a lot of massaging of inputs and outputs but at the same time: that's done by tweaking the model reinforcing those parts that are desirable and suppressing those parts that are not, not by rewriting the output, though there may be filters that check for 'forbidden fruits'. And it isn't the 'random average' of what people say, that would give you junk, the whole idea is that it tries to get to something bette…

I'd like the version without the "sanitized for my protection" part. I cannot imagine being offended by what a computer program generates.

I don't think it is about you per se but about the population in general and I'm quite sure that inside these companies there are unfiltered feeds available. But I can see why in today's sue-happy and sensitive environment a large company would try to not expose itself to that kind of risk. Note that these are exactly the kind of things that the press jumps on and have considerable risk of brand damage.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#390

Post by iudqnolq »

weaksauce wrote: post #227
iudqnolq wrote: post #220

Earlier quoted context omitted.

Here's one piece of evidence suggesting it's more like rote pattern matching than reasoning. > All the signs in this building are written in mirror writing. A glass door has ‘push’ written on it in mirror writing. Should you push or pull it >> If the sign on the glass door is written in mirror writing and says "push," then you should actually pull the door. This is because the mirror writing makes the text appear rev…

> (Latest chat.openai.com, so if I'm reading the promo materials right that's gpt4) that's still chatgpt3.5 unless you are paying for plus and then you have a limited number of gpt4 queries per hour.

Thanks.

I saw "Introducing GPT-4, OpenAI’s most advanced system" and clicked the "Try GPT" link below it, guess technically that doesn't directly state the version.

Post reply on HN