Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching GPT-3 to reverse words

twitter.com

Post reply on HN
Print view View original

11–20 of 81 posts

Re: Teaching GPT-3 to reverse words

#11

Post by 6gvONxR4sf7o »

mrfusion wrote: post #2

> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?

The alternatives are learning at the character level (way more complex, and scales badly in memory/compute), or learning at the whole word level (needs absurdly massive dictionary of words, and still can’t handle really rare/novel words). Breaking things into a set of subwords that allows you to encode any string solves lots of problems and is the relatively standard way to do things these days.

Re: Teaching GPT-3 to reverse words

#12

Post by swid »

convolvatron wrote: post #9
swid wrote: post #6

It's funny to me that this kind of usage of GPT is just programming with a lot of extra steps.

I was just thinking the opposite - that by choosing such a tiny problem one might be able to actually develop some intuition about what's going on inside that very black box

I meant it mostly as a joke, but there is a certain amount of irony to it. This goes way beyond prompt engineering - he wrote an algorithm to run on GPT in a way you would not expect a non-programmer to write. I think the idea is cool and the process to write it was revealing.

Re: Teaching GPT-3 to reverse words

#13

Post by rprenger »

Der_Einzige wrote: post #4

Part of the problem here is that GPT-3 has such a small vocabulary. It's 50K tokens, and many of those are either garbage, punctuation, or full words (rather than sub words). I'd be curious to see what scaling up the size of the vocabulary would do to improve these results in a model like GPT-3...

I don't think a larger vocab would help. All the individual letters are in the ~50k token vocab already, but the word "alphabet" will still not get tokenized to [a, l, p, h, a, b, e, t]. Using a larger vocab like PaLM's 256k vocab would have the same issue.

Re: Teaching GPT-3 to reverse words

#14

Post by axiom92 »

andrewmutz wrote: post #3
mrfusion wrote: post #2

> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?

I believe GPT-3 uses byte pair encoding, which allows it to do tokenization in a language-neutral manner: https://en.wikipedia.org/wiki/Byte_pair_encoding

Yeah it's BPE. OpenAI has a nice tool that allows you to play with the tokenizer https://beta.openai.com/tokenizer.

Re: Teaching GPT-3 to reverse words

#15

Post by jameshart »

Oh, I’m so looking forward to my next coding interview.

“Okay, could you show me on the whiteboard how you might go about writing a program that can reverse a string?”

“Great, so I’m going to start by initializing a simple transformer-based neural network with 175 billion parameters and 96 attention layers, and I’m going to train it on a corpus of 45 terabytes of data tokenized into about 500 billion tokens…”

Re: Teaching GPT-3 to reverse words

#16

Post by mberning »

swid wrote: post #12
convolvatron wrote: post #9

Earlier quoted context omitted.

I was just thinking the opposite - that by choosing such a tiny problem one might be able to actually develop some intuition about what's going on inside that very black box

I meant it mostly as a joke, but there is a certain amount of irony to it. This goes way beyond prompt engineering - he wrote an algorithm to run on GPT in a way you would not expect a non-programmer to write. I think the idea is cool and the process to write it was revealing.

Right. What non-programmer is going to think to turn a word into character list with positional metadata sprinkled in.

Re: Teaching GPT-3 to reverse words

#17

Post by oneepic »

jameshart wrote: post #15

Oh, I’m so looking forward to my next coding interview. “Okay, could you show me on the whiteboard how you might go about writing a program that can reverse a string?” “Great, so I’m going to start by initializing a simple transformer-based neural network with 175 billion parameters and 96 attention layers, and I’m going to train it on a corpus of 45 terabytes of data tokenized into about 500 billion tokens…”

"Cool, so what do you think would be the time complexity of that? Do you think we can maybe do better than that?"

Re: Teaching GPT-3 to reverse words

#18

Post by [deleted] »

jameshart wrote: post #15

Oh, I’m so looking forward to my next coding interview. “Okay, could you show me on the whiteboard how you might go about writing a program that can reverse a string?” “Great, so I’m going to start by initializing a simple transformer-based neural network with 175 billion parameters and 96 attention layers, and I’m going to train it on a corpus of 45 terabytes of data tokenized into about 500 billion tokens…”

[deleted]

Re: Teaching GPT-3 to reverse words

#19

Post by [deleted] »

jameshart wrote: post #15

Oh, I’m so looking forward to my next coding interview. “Okay, could you show me on the whiteboard how you might go about writing a program that can reverse a string?” “Great, so I’m going to start by initializing a simple transformer-based neural network with 175 billion parameters and 96 attention layers, and I’m going to train it on a corpus of 45 terabytes of data tokenized into about 500 billion tokens…”

[deleted]

Re: Teaching GPT-3 to reverse words

#20

Post by jameshart »

swid wrote: post #6

It's funny to me that this kind of usage of GPT is just programming with a lot of extra steps.

If you just ask GPT-3 text-davinci-002 to complete

    Create a Python program to reverse a string:
It produces

    def reverse(s): 
        return s[::-1]
And that isn't even the code-specific model.
Post reply on HN