If you’re also curious how it does vs GPT3 here’s the Zero-shot performance evaluations table: https://github.com/kingoflolz/mesh-transformer-jax#zero-shot...
GPT-J-6B: 6B JAX-Based Transformer
11–20 of 84 posts
Re: GPT-J-6B: 6B JAX-Based Transformer
#12This is first time for me to see JAX [1] being used by non-Google employee. Is it getting some momentum? Or this person just happened to pick it up for some reason (for, say, better TPU support)?
Re: GPT-J-6B: 6B JAX-Based Transformer
#13Earlier quoted context omitted.
It still amazes me. I ask it to write something, and when it does, I just want to ask it "dude, wtf?" Seriously, it's better than what shitposters on 4chan write. It's pure genius: I was watching a climbing competition. I had never seen this before, and I am the world's worst fan of anything outdoors. The crowd was absolutely silent. Like, what, were they dead? It was eerie. Cameo was a perfect example of the sport w…
I input the following: "dogs are superior to cats in every way". and the model generated the following... dogs are superior to cats in every way. They are better at watching TV, they are better at using their tails as fans, they are better at eating out of bowls, they are better at watching over children, they are better at giving head, they are better at being pets, they are better at eating on command, they are bet…
I agree with ALL of the above.
Re: GPT-J-6B: 6B JAX-Based Transformer
#14Sorry guys, I think I broke it. I wanted to test it on my motivational speech, Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around u…
That was downright amazing.
Re: GPT-J-6B: 6B JAX-Based Transformer
#15Sorry guys, I think I broke it. I wanted to test it on my motivational speech, Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around u…
That's a casual chat of two strogg floor managers on their quake2 plant.
Re: GPT-J-6B: 6B JAX-Based Transformer
#16There is a Colab that you can run to set up the model with TPUs: https://colab.research.google.com/github/kingoflolz/mesh-tra...
A few demo examples: https://twitter.com/minimaxir/status/1402468460681068544
Re: GPT-J-6B: 6B JAX-Based Transformer
#17Still upset about the pile as a dataset (the one used to train this model) because I wrote an issue asking if they would include my dataset debatesum into the dataset[1]. They mentioned that they planned to do it and even wrote another issue indicating that they would include it in the pile.
Months go by, and I ask why this dataset is still not included. They told me that the pile is complete and will not be updated anymore. I didn't have time to write my own PR for this and had no idea that there was a time limit to get it in.
Makes me sad. I don't know why this company has decided to stop updating the pile...
Re: GPT-J-6B: 6B JAX-Based Transformer
#18There is a Colab that you can run to set up the model with TPUs: https://colab.research.google.com/github/kingoflolz/mesh-tra... A few demo examples: https://twitter.com/minimaxir/status/1402468460681068544
It's interesting, it seems to get stuck on a sentence or word a lot, then half the output is repeating a slightly varied sentence, or rambling about that unrelated word.
Re: GPT-J-6B: 6B JAX-Based Transformer
#19There is a Colab that you can run to set up the model with TPUs: https://colab.research.google.com/github/kingoflolz/mesh-tra... A few demo examples: https://twitter.com/minimaxir/status/1402468460681068544
It's interesting, it seems to get stuck on a sentence or word a lot, then half the output is repeating a slightly varied sentence, or rambling about that unrelated word.
That's the one thing very consistent with Transformer models, even GPT-3.
Re: GPT-J-6B: 6B JAX-Based Transformer
#20Wow. I fed it a blurb about my vanity Lisp dialect, and what came out was an HN thread complete with comments by dang.