Teaching GPT-3 to reverse words
twitter.com
Teaching GPT-3 to reverse words
1–10 of 81 posts
Re: Teaching GPT-3 to reverse words
#2> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”.
I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?
Re: Teaching GPT-3 to reverse words
#3> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?
I believe GPT-3 uses byte pair encoding, which allows it to do tokenization in a language-neutral manner:
Re: Teaching GPT-3 to reverse words
#4Part of the problem here is that GPT-3 has such a small vocabulary. It's 50K tokens, and many of those are either garbage, punctuation, or full words (rather than sub words).
I'd be curious to see what scaling up the size of the vocabulary would do to improve these results in a model like GPT-3...
Re: Teaching GPT-3 to reverse words
#5> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?
Humans also think about words in terms of subcomponents, languages make heavy use of prefixes and suffixes for example.
Re: Teaching GPT-3 to reverse words
#6It's funny to me that this kind of usage of GPT is just programming with a lot of extra steps.
Re: Teaching GPT-3 to reverse words
#7> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?
I believe GPT-3 uses byte pair encoding, which allows it to do tokenization in a language-neutral manner: https://en.wikipedia.org/wiki/Byte_pair_encoding
I thought I read it uses word2vec?
Re: Teaching GPT-3 to reverse words
#8Part of the problem here is that GPT-3 has such a small vocabulary. It's 50K tokens, and many of those are either garbage, punctuation, or full words (rather than sub words). I'd be curious to see what scaling up the size of the vocabulary would do to improve these results in a model like GPT-3...
50k is not the number of unique words that GPT-3 supports, and perhaps you're referring to the BPE tokens. The input to GPT-3 is not tokenized by splitting on spaces, and is based on byte-pair encoding tokens. You can play with it here: https://beta.openai.com/tokenizer.
A rare word like blithe is tokenized into two BPE tokens: bl and ithe, whereas common words like the get their own token.
Re: Teaching GPT-3 to reverse words
#9It's funny to me that this kind of usage of GPT is just programming with a lot of extra steps.
I was just thinking the opposite - that by choosing such a tiny problem one might be able to actually develop some intuition about what's going on inside that very black box
Re: Teaching GPT-3 to reverse words
#10> Tokens are chunks of characters. For example, the word “alphabet” gets broken up into the tokens “alph" and "abet”. I didn’t know that. Seems like it would confuse it during training. Anyone able to explain?
If I recall correctly, it's similar to how fasttext vectors work. For fasttext, this means that the representation of words is dependent to a certain extent to its morphemes (not really, but bear with me), so rare/inflected words can have a better representation due to the similarity with words that are similar-looking and more frequent (e.g. "unconstitutional" might never appear in the training data, but the system can approximate its meaning by composing that of "un", which it has seen in words such as "unbelievable", and the remaining subtokens, that come from the word "constitutional" that was present in the training set)
Not sure if the same thing happens here, tho