The makers of Eleuther hope it will be an open source alternative to GPT-3
51–60 of 100 posts
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#52Earlier quoted context omitted.
yes I've read them, would those pass an English composition class in college? This comment generated by Gpt3
I think GPT essays could definitely pass a freshman expository writing class. I went to a pretty good university and when we did peer review I was pretty surprised at (what I considered) the low average quality of the writing.
Examples?
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#53People already believe garbage at a pretty alarming rate. It's easy to guess at a number of possible outcomes here: - More junk text moves the public to doubt legitimate information even further than they currently do. - There is so much human-generated junk text that adding more of it via AI actually doesn't have much of an effect. - People return to lean on experts, perhaps even more than before. (just as a number…
Concrete prediction: There will be a global cult similar in nature to Qanon driven by an AI spitting out generated bullshit within the next ten years.
That's assuming some percentage of Qanon word salad isn't the output of Markov chain generators. A lot of it resembles low-order statistical text generator output after having been trained on a corpus of 1990s Usenet alt.conspiracy and the Protocols of the Elders of Zion.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#54Earlier quoted context omitted.
Have you read college freshman essays? While it's coherent in a sentence it is very rambling over paragraphs to pages.
yes I've read them, would those pass an English composition class in college? This comment generated by Gpt3
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#55I don't know why the eleuther project riles me up so much. Their work on the pile gets to me because they're so cavalier about copyright (while I defend myself by training on similarly pirated text datasets, but feel different because I don't redistribute them and am honest that it's pirated. to be clear, i'm rolling my eyes at my rationalization right here). Their work on gpt-neo riles me up because they do such a weak job comparing it to the models whose hype they're riding. It also riles me up because so many people just eat it up uncritically.
But it's all out of proportion. I think it's that last part (the uncritical reaction) that makes me blow this out of proportion.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#56I don't know why the eleuther project riles me up so much. Their work on the pile gets to me because they're so cavalier about copyright (while I defend myself by training on similarly pirated text datasets, but feel different because I don't redistribute them and am honest that it's pirated. to be clear, i'm rolling my eyes at my rationalization right here). Their work on gpt-neo riles me up because they do such a w…
Hi! I’m the EAI person who your criticism of the Pile is most directed at. I’m curious if you read Sections 6.5 and 7 of the Pile working paper and, if so, what your response to it is. As you note, virtually everyone trains on copyright data and just ignores any implications of that fact. I feel that our paper is very upfront about this though, going as far as to have a table that explicitly lists which subsets contain copyrighted text.
Also, I realize that you don’t have any ways of knowing this but we also have separated out the subset of the Pile that we can confirm is licensed CC-BY-SA or more leniently. This wasn’t done in time for the preprint, but is in the (currently under review) peer reviewed publication. Unfortunately the conference rules forbid you from posting materials or updating preprints between Jan 1st 2021 and the final decision announcement. But we will be making the license-compliant subset of the Pile public when we are able to and will give it equal prominence on our website to the “full” Pile.
Also, we will be releasing a datasheet for the dataset but again conference limitations prevent us from doing so yet.
If you’re interested in talking about this in depth, feel free to send me an email.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#57I don't know why the eleuther project riles me up so much. Their work on the pile gets to me because they're so cavalier about copyright (while I defend myself by training on similarly pirated text datasets, but feel different because I don't redistribute them and am honest that it's pirated. to be clear, i'm rolling my eyes at my rationalization right here). Their work on gpt-neo riles me up because they do such a w…
> Their work on GPT-Neo rules me up because they do such a weak job comparing it to the models whose hype they’re riding.
Building open source infrastructure is hard. There does not currently exist a comprehensive open source framework for evaluating language models. We are currently working on building one (https://github.com/EleutherAI/lm-evaluation-harness) and are excited to share results when we have the harness built.
If you don’t think the model works, you are welcome to not use it and you are welcome to produce evaluations showing that it doesn’t work. We would happily advertise your eval results side by side with our own.
I am curious where you think we are riding the hype /to/ so to speak. The attention we’ve gotten in the last two weeks has actually been a net negative from a productivity POV, as it’s diverted energy away from our larger modeling work towards bug fixes and usability improvements. We are a dozen or so people hanging out in a discord channel and coding stuff in our free time, so it’s not like we are making money or anything based on this either.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#58Earlier quoted context omitted.
Ah yes, the only possible issue with releasing fully general AI is that it might say something offensive. Not because we don't have it at all, not because if we did we shouldn't just let it out like a lion in the gazelle pen to see what it does, because of those snowflakes!
Wrong, gpt4 is new and has not been implmented as a chat bot. Also wrong that previous chat bots were not shut down for being offensive. https://en.wikipedia.org/wiki/Tay_(bot)
GPT-4 doesn’t exist mate.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#59It's funny how behind the times that Wired is getting. Even my parents know about how scary good these text models are getting.
Re: The makers of Eleuther hope it will be an open source alternative to GPT-3
#60It's very nice to see Eleuther fulfill the Open promise of OpenAI.
I'm scared that more and more big model advancements are being denied access from the general public, which will just make the inequality between big corporations and startups even greater.