Live data from Hacker News

GPT-J-6B: 6B JAX-Based Transformer

arankomatsuzaki.wordpress.com

21–30 of 84 posts

Re: GPT-J-6B: 6B JAX-Based Transformer

#21

Still upset about the pile as a dataset (the one used to train this model) because I wrote an issue asking if they would include my dataset debatesum into the dataset[1]. They mentioned that they planned to do it and even wrote another issue indicating that they would include it in the pile. Months go by, and I ask why this dataset is still not included. They told me that the pile is complete and will not be updated…

You should fork it! I'd use your fork.

Re: GPT-J-6B: 6B JAX-Based Transformer

#22
Tried it with Islamic prompt (in Arabic) بِسمِ اللَّهِ الرَّحمٰنِ الرَّحيمِ

And I got this bit of Wisdom:

As-Salaamu alaikum wa rahmatullahi wa barakatu The world is a single mirror. The whole universe becomes all a single mirror if you observe it with your whole heart, as it really is and nobody else's imagination and interpretation of it. Your spiritual sight will at one time reveal the depth of reality, other times it will reveal a false picture which is reflected by all kind of greed, vanity, falsehood, lust etc. The only way to cross this mirror is by an open heart, and it needs to be constantly cleansed of all these sins. It also needs to be cleansed of all the deceitfulness of this world, which is nothing but a spiritual mirror reflecting your own false vision of your imagination. This all depends on one's intention of which Allah is aware and He always warns the soul in its creation. Those who love Allah and His messengers, and who are guided by the Qur'an are aware of this and they never praise anyone or blame anyone for anything, no matter how great or small. They are always aware and watchful and vigilant in their intents. Those who are only biased by their biased knowledge will never do it and they are very few, as everybody has biased knowledge about

Re: GPT-J-6B: 6B JAX-Based Transformer

#23
I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon.

Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including all possible keywords I might use to narrow my search), we will drown in useless information.

And this isn't restricted to programming, this was just an example of a field that is full of scam already. When it gets cheap enough, somebody will do this with any topic that people are interested in...

I wonder how Google and other search engines will tackle this problem.

Re: GPT-J-6B: 6B JAX-Based Transformer

#24
These results are terrible. Almost all of the generated text posted here is non-sensical, and playing with it online is just confusing.

There really needs to be a better method of evaluation, an MNIST for transformer text generation. like a list of pre-defined prompts that every GPT-X has to use to generate text, which can be scored against a list of "correct" answers in a variety of ways.

I have no way of knowing if the output of this flavor of transformer is good or not, whatever that would mean. Very difficult to see how it compares to similar models. Setting up a model with this number of parameters and their reported training times is impressive. But I have no idea if this particular number of parameters makes a difference, or what that difference is supposed to look like

Re: GPT-J-6B: 6B JAX-Based Transformer

#25
post #23

I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon. Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including…

Oh my god. I fed that text to the model and asked what we can do about this. Here are its answers:

It is not so much the search engines that are at risk, but the people themselves. In a world where anyone with a $0.05 budget can just make something up, nobody will be able to trust anything anymore. It is not the search engines that will change the way we search, it will be the people, who will stop trusting anything they read. And that is a problem.

The problem is not with search engines. It's with people. Most people who get scammed will not be able to understand the text they see. They may think it's legit because it looks pretty, but they don't understand what the text means. The only way to find out the truth is by doing a little research and contacting the company to ask for a refund or to talk to their support. But, with enough of these scams, people will stop trusting anything they read.

Re: GPT-J-6B: 6B JAX-Based Transformer

#27
post #24

These results are terrible. Almost all of the generated text posted here is non-sensical, and playing with it online is just confusing. There really needs to be a better method of evaluation, an MNIST for transformer text generation. like a list of pre-defined prompts that every GPT-X has to use to generate text, which can be scored against a list of "correct" answers in a variety of ways. I have no way of knowing if…

Thankfully, there already exist evaluation tasks like that, and Eleuther actually has a project collecting a handful of them together; see https://github.com/EleutherAI/lm-evaluation-harness/

Re: GPT-J-6B: 6B JAX-Based Transformer

#28
post #8

Sorry guys, I think I broke it. I wanted to test it on my motivational speech, Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around u…

The practice of de-beaking, for the purposes of rendering a quicker death to the entire herd, is a part of the job. The most the boss will say is “Do you have any questions?” There are no expectations of cooperation, no in-universe questions to ask or mysteries to solve, just deadly work to be done.

When you get sent to the feed room, you are told, “You are going to have to steal some feed.” You are given a specific passcode (which you are not allowed to ask for or demand to be changed) and are told that you will be given one eight hour time period to complete the task. If you find that you can get to the feed room more or less quickly, you are told to report back to your boss. If you do the task in the time allocated and return the approved bit of feed you were given, there will be no consequences. However, if you exceed the time you were given, there will be consequences. You will be sent home for two weeks.

This is how things work until you show an ability to actually do what you are told. You are never told what the consequences will be, nor is it clear how the

Re: GPT-J-6B: 6B JAX-Based Transformer

#29
Lol, I fed it this "Calorie counting works as follows..."

And it immediately tried to sell me an ebook. For real.

"The biggest danger with this diet is that you'll probably end up counting calories and then feeling deprived so you want to eat more and more. This cycle will keep you yo-yo'ing right back to the point where you started and then make you feel much worse. I have made this mistake myself and I know that I want to make sure I don't repeat that mistake.

To get people started with the 21-day Fast Weight Loss Guide, I am offering it for just $1.99 on April 29, and every day of the month after that until the program is sold out, and it will be up to the discretion of the buyer if they choose to wait and pay more"

Re: GPT-J-6B: 6B JAX-Based Transformer

#30
post #25
post #23

I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon. Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including…

Oh my god. I fed that text to the model and asked what we can do about this. Here are its answers: It is not so much the search engines that are at risk, but the people themselves. In a world where anyone with a $0.05 budget can just make something up, nobody will be able to trust anything anymore. It is not the search engines that will change the way we search, it will be the people, who will stop trusting anything…

Begun, the AI war, has.
Post reply on HN