Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-J-6B: 6B JAX-Based Transformer

arankomatsuzaki.wordpress.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: GPT-J-6B: 6B JAX-Based Transformer

#31

Post by TaylorAlexander »

Here’s my test prompt: “Today at a rally in Virginia, presidential candidate Bernie Sanders gave his remarks in front of a large crowd. Here is what he said.”

Below is the output, which aside from talking about New York when my prompt says Virginia is otherwise pretty surprisingly believable:

Bernie Sanders: Well good afternoon, my friends. I want to thank you very much for being here.

You know, I am often asked, you know, can we do well in New York State? I say we can do much better than I think Governor Cuomo expects. The people of New York State have given me a lot of encouragement. And I believe that the people of the State of New York have given this campaign a lot of strength. We’re in a much stronger position to win the primary because of that strength.

When I started campaigning in New York a few weeks ago, there were many people who said, “Okay, you know, we need to win in Nevada, we need to win in New Hampshire, we need to win in South Carolina, we need to win in California.” And we don’t need to do that. We are doing well. We are doing well. I think the people of New York have been very supportive. I think the people of the State of New York have been supportive. We are winning. And we are winning. We are well on our way to winning in California. That means we are winning in the nation.

You know, before I came

(end of model output)

Re: GPT-J-6B: 6B JAX-Based Transformer

#32

Post by sillysaurusx »

Hi Kindiana! If you happen to see this, I sent a message on Keybase. Nothing important, just saying hello.

This is absolutely wonderful work, and I'm really happy to see you've released it. Congrats!!

It's also awesome that Eleuther is hosting an API for it. Nice work, Elecrew. No idea who did that part, but it's pretty cool. It's easy to underestimate how hard something like that can be.

Good luck with your future research projects. I'm certain your name will be popping up everywhere before long. :) (I'll be taking another stab at swarm training now that TPU SSH is finally released; thanks for brainstorming random ideas with me, it was lots of fun.)

EDIT: Oh, sorry for posting this without context. Kindiana’s the submitter, and also the person who wrote most of the code. (Getting Jax working on TPUs is no small feat, let alone writing and training a gpt implementation.) Been following this work a long time; excited to see it released.

Anyway, we used to hang out and talk about random ML ideas. It’s been awhile, so I was just saying hello.

Re: GPT-J-6B: 6B JAX-Based Transformer

#33

Post by TaylorAlexander »

TaylorAlexander wrote: post #31

Here’s my test prompt: “Today at a rally in Virginia, presidential candidate Bernie Sanders gave his remarks in front of a large crowd. Here is what he said.” Below is the output, which aside from talking about New York when my prompt says Virginia is otherwise pretty surprisingly believable: Bernie Sanders: Well good afternoon, my friends. I want to thank you very much for being here. You know, I am often asked, you…

I was curious if it knew anything about me (slim chances) and it does not seem to. It generated a thorough bio that has nothing to do with me. However it’s pretty funny. The prompt is in quotes and the output follows:

“Taylor Alexander is a robotics engineer and “

one of the founders of Transcend, a research lab. Alexander specializes in both manufacturing and consumer applications of robotics. He is also a former roboticist at Carnegie Mellon University, where he studied robotics and human-computer interaction. In that capacity, he worked on a research program funded by DARPA to design a mobile manipulator for soldiers.

The team tested the robotic arm in a variety of physical simulations for landing, handling, and carrying heavy loads like a military bomb. They tested the design on both operating systems—Linux and Windows.

“We built this to test the machine on the two dominant operating systems that we thought might be useful in the future,” says Alexander. “But it actually worked on everything. We saw the same kind of accuracy in the process on all of the different operating systems.”

The team decided to make the arm multi-platform after seeing the arm working so well on Linux and Windows. They were very impressed with the portability. The arm was light enough to pick up a relatively heavy base and, with just a slight rotation, it could hold the base up in the air. The weight didn’t matter to the arm at all.

Since the arm works on Linux, the

Re: GPT-J-6B: 6B JAX-Based Transformer

#34

Post by hervature »

TaylorAlexander wrote: post #31

Here’s my test prompt: “Today at a rally in Virginia, presidential candidate Bernie Sanders gave his remarks in front of a large crowd. Here is what he said.” Below is the output, which aside from talking about New York when my prompt says Virginia is otherwise pretty surprisingly believable: Bernie Sanders: Well good afternoon, my friends. I want to thank you very much for being here. You know, I am often asked, you…

It did well. It did well. I think it did well. Because it did well, it will do well.

Re: GPT-J-6B: 6B JAX-Based Transformer

#35

Post by 29athrowaway »

I took one of the questions from this article: https://blog.seekwell.io/gpt3

    Instruction: Given an input question, respond with syntactically correct PostgreSQL. Be creative but the SQL must be correct.
    
    Input: how many users signed up in the past month?
And it gave me a pretty nonsensical answer compared to the output from GPT-3 seen in that article.

Re: GPT-J-6B: 6B JAX-Based Transformer

#36

Post by 29athrowaway »

29athrowaway wrote: post #35

I took one of the questions from this article: https://blog.seekwell.io/gpt3 Instruction: Given an input question, respond with syntactically correct PostgreSQL. Be creative but the SQL must be correct. Input: how many users signed up in the past month? And it gave me a pretty nonsensical answer compared to the output from GPT-3 seen in that article.

Because that answer was pretty weak, I resorted to an easier question... taken from the prison IQ test from the movie Idiocracy.

    You have one bucket that holds 2 gallons and another bucket that holds 5 gallons. How many buckets do you have?
The answer was complete nonsense.

https://youtu.be/jbmq9P-8FiM?t=17

Re: GPT-J-6B: 6B JAX-Based Transformer

#37

Post by fshbbdssbbgdd »

planb wrote: post #23

I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon. Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including…

Google uses a lot of other signals besides page content to rank search results. Common examples:

1. Links from well-ranked pages. 2. Whether the user hits the back button or stays on the page.

Many of these should still work.

Re: GPT-J-6B: 6B JAX-Based Transformer

#38

Post by planb »

fshbbdssbbgdd wrote: post #37
planb wrote: post #23

I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon. Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including…

Google uses a lot of other signals besides page content to rank search results. Common examples: 1. Links from well-ranked pages. 2. Whether the user hits the back button or stays on the page. Many of these should still work.

But they obviously don't work! That's why I mentioned all those Stack Overflow rip off sites as an example. When scrolling the results, my brain filters them out because I can recognise them as scam, but if they would contain unique and relevant text (generated by an AI), I'd be lost.

Re: GPT-J-6B: 6B JAX-Based Transformer

#39

Post by voldacar »

foobiekr wrote: post #8

Sorry guys, I think I broke it. I wanted to test it on my motivational speech, Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around u…

This is simply incredible, I would love to read something novel-length like this. But unfortunately it also generates a lot of banal text, as in some of the other examples in this thread. I wonder if there is a way to optimize for the weird shit.

Re: GPT-J-6B: 6B JAX-Based Transformer

#40

Post by inciampati »

hooande wrote: post #24

These results are terrible. Almost all of the generated text posted here is non-sensical, and playing with it online is just confusing. There really needs to be a better method of evaluation, an MNIST for transformer text generation. like a list of pre-defined prompts that every GPT-X has to use to generate text, which can be scored against a list of "correct" answers in a variety of ways. I have no way of knowing if…

You have an idea of what's sensical and not. It would seem that long range correlations in the text break down. This is true of all of the popular models, even the trillion parameter ones. They just break down at longer distances.
Post reply on HN