Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-J: GPT-3 Democratized

p3r.one

Post reply on HN
Print view View original

31–40 of 41 posts

Re: GPT-J: GPT-3 Democratized

#31

Post by zitterbewegung »

c7DJTLrn wrote: post #16
zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

That seems ridiculously slow. How is machine learning supposed to scale like this?

I’m only running one Titan RTX which is slower than a 3090 and you can parallelize the system by buying multiple graphics card or by increasing the amount of VRAM used on the endpoint and buying an Ampere .

Re: GPT-J: GPT-3 Democratized

#32

Post by [deleted] »

drusepth wrote: post #18
zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

It sounds like we're using very similar setups for writing. :) I've primarily been using GPT-3 (and burning through millions of tokens) so I've been experimenting with GPT-J more lately and I've found it makes significantly more basic logic errors (e.g. mixing up pronouns, "forgetting" characters, using new characters), which makes me lean more towards constantly regenerating text versus revising it, and I feel like…

[deleted]

Re: GPT-J: GPT-3 Democratized

#33

Post by zitterbewegung »

drusepth wrote: post #18
zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

It sounds like we're using very similar setups for writing. :) I've primarily been using GPT-3 (and burning through millions of tokens) so I've been experimenting with GPT-J more lately and I've found it makes significantly more basic logic errors (e.g. mixing up pronouns, "forgetting" characters, using new characters), which makes me lean more towards constantly regenerating text versus revising it, and I feel like…

I didn't experiment that much to get the difference between Babbage and GPTJ but that sounds similar than what I experienced. I am not sure what your novel is going to look like but due to the limitations between GPT-3 / GPTJ etc.. I plan on it being something like a stream of consciousness.

I thought of babysitting lower amounts of tokens but I was getting okay quality with a few thousand but when I looked into it I saw that in the middle usually there was some weird change.

I know that there is a GPTJ setting (and I believe GPT-3 has this) where you can see the best of X tries. Clicking best suggestion sounds like a good idea of a tool but it might be better just to recursively store all outputs. I am interested in the tool you are using are you able to share it?

The writing process I am using is that there was a YouTube reviewer that used a dataset of Light Novels and utilized a refined version of GPT-2 on that to create a title of some light novels and then basically cherry picked them / did analysis / had his friends do a Turing test. I would assume that using GPT-3 / GPTJ could do the same thing but doing a zero shot with a 10-20 token limitation.

(EDIT GPT-3 using Davinci and the Two-sentence Horror Story would allow you to generate interacting story titles and then from there you can use GPT-3 / GPTJ to do the rest.)

(also if anyone is interested in the novel I am writing email me at zitterbewegung at gmail dot com).

Re: GPT-J: GPT-3 Democratized

#34

Post by robbedpeter »

EamonnMR wrote: post #28
robbedpeter wrote: post #23

Earlier quoted context omitted.

Transparent and competent automated content moderation, maybe, easily available for anyone to run their own communities by their own standards. Once matured, you can easily envision people sharing policies and templates, or providing moderation as a service, for any sort of social text interaction.

I was just thinking about this today. How many (and which?) ML books would one need to read to be able to throw comments into a model like this and have it decide if they're acceptable vs unacceptable? How practical would it be to build a frequently re-training model that could serve as autopilot for a small forum or subreddit moderation? And how much would that serve to reinforce existing filter bubbles and further…

0 books. The huggingface hosted gpt-neo 125 model is capable of doing basic question/answer analysis to a sufficient level for most moderation. No finetuning needed if you have it work out yes or no zero shot answers to things like "is this text bullying, excessively negative, political, sexist..." etc, and then iterate over cases and log results. Modify your prompts to handle edge cases and it would probably be capable of replacing 80% of human moderation.

Re: GPT-J: GPT-3 Democratized

#35

Post by drusepth »

zitterbewegung wrote: post #33
drusepth wrote: post #18

Earlier quoted context omitted.

It sounds like we're using very similar setups for writing. :) I've primarily been using GPT-3 (and burning through millions of tokens) so I've been experimenting with GPT-J more lately and I've found it makes significantly more basic logic errors (e.g. mixing up pronouns, "forgetting" characters, using new characters), which makes me lean more towards constantly regenerating text versus revising it, and I feel like…

I didn't experiment that much to get the difference between Babbage and GPTJ but that sounds similar than what I experienced. I am not sure what your novel is going to look like but due to the limitations between GPT-3 / GPTJ etc.. I plan on it being something like a stream of consciousness. I thought of babysitting lower amounts of tokens but I was getting okay quality with a few thousand but when I looked into it I…

Thanks for the answer!

Best of N is super useful. With GPT-3 (and probably GPT-J) you can actually pass an n parameter to match best_of and get back all of the N texts generated, which is how my little tool works (displaying them all to pick from, since the "best" one by GPT-3's standards isn't always the "best" one from my author POV). It's also nice to not feel like i'm throwing away tokens, since best_of=3 uses 3x the tokens whether you look at all the generations or not!

Thanks for the details on the Light Novels process. I'll have to look into it and see if something like that works for me!

The tool I wrote is at https://young-savannah-97958.herokuapp.com/ (source code at https://github.com/indentlabs/gpt-3-writer), but only works with GPT-3 right now. You just stick in your API key (which doesn't get stored/saved) and customize completion settings if you want, then trigger completions while writing with the button in the bottom-right. It's very rough around the edges since I just built it for my own use, have a ton on the to-do list, and haven't publicized it, but it could be helpful. :)

Re: GPT-J: GPT-3 Democratized

#37

Post by zitterbewegung »

drusepth wrote: post #35
zitterbewegung wrote: post #33

Earlier quoted context omitted.

I didn't experiment that much to get the difference between Babbage and GPTJ but that sounds similar than what I experienced. I am not sure what your novel is going to look like but due to the limitations between GPT-3 / GPTJ etc.. I plan on it being something like a stream of consciousness. I thought of babysitting lower amounts of tokens but I was getting okay quality with a few thousand but when I looked into it I…

Thanks for the answer! Best of N is super useful. With GPT-3 (and probably GPT-J) you can actually pass an n parameter to match best_of and get back all of the N texts generated, which is how my little tool works (displaying them all to pick from, since the "best" one by GPT-3's standards isn't always the "best" one from my author POV). It's also nice to not feel like i'm throwing away tokens, since best_of=3 uses 3x…

What I do is I take the first title then I make derivative titles based on the starting title. Like “write a story about cookies” then “write a story about chocolate chip titles”.

Re: GPT-J: GPT-3 Democratized

#38

Post by EamonnMR »

robbedpeter wrote: post #34
EamonnMR wrote: post #28

Earlier quoted context omitted.

I was just thinking about this today. How many (and which?) ML books would one need to read to be able to throw comments into a model like this and have it decide if they're acceptable vs unacceptable? How practical would it be to build a frequently re-training model that could serve as autopilot for a small forum or subreddit moderation? And how much would that serve to reinforce existing filter bubbles and further…

0 books. The huggingface hosted gpt-neo 125 model is capable of doing basic question/answer analysis to a sufficient level for most moderation. No finetuning needed if you have it work out yes or no zero shot answers to things like "is this text bullying, excessively negative, political, sexist..." etc, and then iterate over cases and log results. Modify your prompts to handle edge cases and it would probably be capa…

Hmm, I'll still need to scrounge up enough context to understand most of that, but thanks.

Re: GPT-J: GPT-3 Democratized

#39

Post by zitterbewegung »

gurchik wrote: post #12
zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

> I am writing a novel with it. Could you say more?

Novel will be a stream of consciousness of a person that imitates a ;light novel.

Re: GPT-J: GPT-3 Democratized

#40

Post by robbedpeter »

EamonnMR wrote: post #38
robbedpeter wrote: post #34

Earlier quoted context omitted.

0 books. The huggingface hosted gpt-neo 125 model is capable of doing basic question/answer analysis to a sufficient level for most moderation. No finetuning needed if you have it work out yes or no zero shot answers to things like "is this text bullying, excessively negative, political, sexist..." etc, and then iterate over cases and log results. Modify your prompts to handle edge cases and it would probably be capa…

Hmm, I'll still need to scrounge up enough context to understand most of that, but thanks.

https://jalammar.github.io/illustrated-gpt2/

There are several easily found resources like this, and previous hn threads contain papers, documentation, and guide links spanning the spectrum from beginner to PhD researcher. You can get to a high level of proficiency without any books, but you'll need to learn the jargon for searches. It's a niche enough subject that results don't get too muddied on almost any search engine - good luck!

Post reply on HN