Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

61–70 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#61

Post by rajansaini »

EGreg wrote: post #45
seydor wrote: post #39

Someone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.

I am shocked that it speaks the way it does when it was trained on random stuff it doesn’t have rights to. They say they trained it on databases they had bought access to etc. And it seems that way. Because how does ChatGPT: 1. Do what you ask instead of continuing your instructions? 2. Use such nice and helpful language as opposed to just random average of what people say? 3. And most of all — how does it have a str…

Read the InstructGPT paper and see alpaca. You just need instruction fine-tuning.

Unlike what the other commenters are saying, RLHF, while powerful, isn't the only way to get an LLM to follow instructions.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#62

Post by mtkd »

0xDEF wrote: post #5

The "leak" is being portrayed as something highly subversive done by the darn 4chan hackers. Before the "leak" Meta was sending the model to pretty much anyone who claimed to be a PhD student or researcher and had a credible college email. Meta has probably been planning to release the model sooner than later. Let's hope they release it under a true open source license.

A cynic might say FB are confident in the limitations of current models and want to pour cold water on the ChatGPT excitement (esp. with what appears to be a goldrush this month to use it to generate numbers and insight into numbers)

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#63

Post by vidarh »

riceart wrote: post #42
abstractbill wrote: post #38

Earlier quoted context omitted.

The models are a lot of fun to play with, but yeah, every time I've tried to use them for something "serious" they nearly always invent stuff (and are so convincing in how they write about it!). Most recently I've been interested in what's happened with the 4-color theorem since the 1976 computer-assisted proof, and decided to use GPTChat instead of google+wikipedia. GPTChat had me convinced and excited that, apparen…

Before the inevitable idiots come in to say hurr durr but have you tried ChatGPT 4… yes I paid for it, and it is just as prone to hallucinations of factual information. It loves to make up new names for peoples initials.

While it is still prone to hallucinations, with GPT4 I've had it tell me "X is not a real project, but assuming it exists and is designed to do Y [based on the name], I can provide you with a general guide on how to use it." I never had earlier version to do that. So it does appear to be better, though it's certainly still an issue.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#64

Post by vidarh »

vlovich123 wrote: post #60
senko wrote: post #43

Earlier quoted context omitted.

I hold a more charitable interpretation. We (the public) have found an important bug in the system, ie. GPT can lie (or "hallucinate"), even if you try to convince it not to lie. The bug is definitely lowering the usefulness of their product, as well as the public option about it. But I'll let the programmer who has never coded a bug cast the first stone. I wouldn't be surprised if they're scrambling internally to mi…

It's not a bug. It's an architectural defect / limitation in our understanding of how to build AI. That makes it a strictly harder problem that will take longer. And it's not totally clear to me that you'll get there purely with LLMs. LLMs accomplish a good chunk of what we classify as intelligence for sure. But it's missing the cognition / reasoning skills and the open question is whether you can solve that by just…

GPT 4 will admit to not knowing things in many cases where 3.5turbo does not (tested the same prompt), and either will stop there or go off on a "but if it did exist it might go something like this" type continuation. It still hallucinates a lot, but it's not at all clear that this will be all that difficult an issue to solve given the progress.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#65

Post by danShumway »

coding123 wrote: post #54
seydor wrote: post #39

Someone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.

what's weird to me though, is that we're all trained on both open source and closed source source material. And our output is totally 100% copyrightable by us. Why wouldn't we extend the same muster to computer generated text. If there is a copy-written sentence, go after that? I don't work for openai, but I don't like 1 sided arguments that are just looking for some bottom line. At the end of the day we all have som…

Copyright is a practical right, not an inherent right. The only reasons humans get copyright at all is because it's useful for society to give it to them.

The onus should be on OpenAI to prove that it will benefit society overall if AIs are given copyright. We've already decided that many non-human processes/entities don't get copyright because there doesn't seem to be any reason to grant those entities copyright.

----

The comparison to humans is interesting though, because teaching a human how to do something doesn't grant you copyright over their output. Asking a human to do something doesn't automatically mean you own what they create. The human actually doing the creation gets the copyright, and the teacher has no intrinsic intellectual property claim in that situation.

So if we really want to be one-to-one, teaching an AI how to do something wouldn't give you copyright over everything it produces. The AI would get copyright, because it's the thing doing the creation. And given that we don't currently grant AIs personhood, they can't own that output and it goes into the public domain.

But in a full comparison to humans, OpenAI is the teacher. OpenAI didn't create GPT's output, it only taught GPT how to produce that output.

----

The followup here though is that OpenAI claims that it's OK to train on copyrighted material. So even if GPT's output was copyrightable, that still doesn't mean that they should be able to deny people the ability to train on it.

I mean, talk about one-sided arguments here: if we treat GPT output the same as human output, then is OpenAI's position that it can't train on human output? OpenAI has a TOS around this basically banning people from using the output in training, which... probably that shouldn't be enforceable either, but people who haven't agreed to that TOS should absolutely be able to train AI on any ChatGPT logs that they can get a hold of.

That is exactly what OpenAI did with copyrighted material to train GPT. It's not one-sided to expect the same rules to apply to them.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#66

Post by tyfon »

whimsicalism wrote: post #11

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?

> Why do these takes around open-source AI remain so popular?

I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs.

The 65B llama I run is actually usable for most of the tasks I would ask chatgpt for (I have premium there but that will lapse this month). The best part is that I never see the "As a large language model I can't do shit" reply.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#67

Post by mountainriver »

rmdashrfstar wrote: post #58
seydor wrote: post #39

Someone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.

> If web scraping is legal Source? That LinkedIn case did not resolve how you think it did.

It’s legal but if you don’t consent to people doing it in your robots.txt you can sue them civilly

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#68

Post by bri3d »

rmdashrfstar wrote: post #58
seydor wrote: post #39

Someone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.

> If web scraping is legal Source? That LinkedIn case did not resolve how you think it did.

My understanding is that the current web scraping situation is this:

* Web scraping is not a CFAA violation. (EF Travel v. Zefer, LinkedIn v. hiQ).

* Scraping in spite of clickthrough / click-in ToS "violation" on public websites does not constitute an enforceable breach of contract, chattel trespass (ie - incidental damage to a website due to access), or really mean anything at all. This is not as clear once a user account or log-in process is involved. (Intel v. Hamidi, Ticketmaster v. Tickets.com)

* Publishing or using scraped data may still violate copyright, just as if the data had been acquired through any means other than scraping. (AP v. Meltwater, Facebook v. Power.com)

So this boils down to two fundamental questions that will need to get answered regardless of "scraping" being involved: "is GPT output copyrightable" and "is training a model on copyrighted data a copyright infringement."

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#69

Post by [deleted] »

coding123 wrote: post #54
seydor wrote: post #39

Someone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.

what's weird to me though, is that we're all trained on both open source and closed source source material. And our output is totally 100% copyrightable by us. Why wouldn't we extend the same muster to computer generated text. If there is a copy-written sentence, go after that? I don't work for openai, but I don't like 1 sided arguments that are just looking for some bottom line. At the end of the day we all have som…

[deleted]

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#70

Post by pclmulqdq »

echelon wrote: post #26
adeon wrote: post #10

I'm a bit worried the LLaMA leak will make the labs much more cautious about who they distribute models to for future projects, closing down things even more. I've had tons of fun implementing LLaMA, learning and playing around with variations like Vicuna. I learned a lot and probably wouldn't have got so interested in this space if the leak didn't happen.

If the copyright office determines model weights are uncopyrightable (huge if), then one might imagine any institutional leak would benefit everyone else in the space. You might see hackers, employees, or contractors leaking models more frequently. And since models are distilled functionality (no microservices and databases to deploy), they're much easier to run than a constellation of cloud infrastructure.

Shouldn't that be the default position? The training methods are certainly patentable, but the actual input to the algorithm is usually public domain, and outputs of algorithms are not generally copyrightable as new works (think of to_lowercase(Harry Potter), which is not a copyrightable work), so the model weights would be a derivative work of public domain materials, and hence also forced into the public domain from a copyright perspective.

They are generally trade secrets now, which is what actually protects them. Leaks of trade secrets are serious business regardless of the IP status of the work otherwise.

Post reply on HN