Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-J-6B: 6B JAX-Based Transformer

arankomatsuzaki.wordpress.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: GPT-J-6B: 6B JAX-Based Transformer

#51

Post by planb »

jobigoud wrote: post #42
planb wrote: post #25

Earlier quoted context omitted.

Oh my god. I fed that text to the model and asked what we can do about this. Here are its answers: It is not so much the search engines that are at risk, but the people themselves. In a world where anyone with a $0.05 budget can just make something up, nobody will be able to trust anything anymore. It is not the search engines that will change the way we search, it will be the people, who will stop trusting anything…

Yeah you wrote that it made no sense semantically but it does, it rewrote your exact argument. This is great for a non native speaker for example, to rewrite a paragraph in a more standard language. Plagiarism will definitely be harder/impossible to detect.

The last paragraph wasn't mine. It was also part of the output :)

Re: GPT-J-6B: 6B JAX-Based Transformer

#52

Post by PoignardAzur »

phonon wrote: post #30
planb wrote: post #25

Earlier quoted context omitted.

Oh my god. I fed that text to the model and asked what we can do about this. Here are its answers: It is not so much the search engines that are at risk, but the people themselves. In a world where anyone with a $0.05 budget can just make something up, nobody will be able to trust anything anymore. It is not the search engines that will change the way we search, it will be the people, who will stop trusting anything…

Begun, the AI war, has.

This does make me wonder, how long until we get a GPT that can drop pop-culture references smoothly enough to pass the Turing Test?

It's harder than other types of text generation, because part of the game of referring to memes is to do it in as roundabout a way as you can, using the kind of hidden inferences a neural network would probably struggle with.

But it might still be feasible soon. shudders

Re: GPT-J-6B: 6B JAX-Based Transformer

#53

Post by PoignardAzur »

Prompt: What's the deal with Rust?

---

Are you interested in hearing about the coolest new language of 2018? Well, Rust might be the one to check out. Rust is more or less an evolution of C++. But Rust is also a radically different design.

C++ is still used in the majority of real-world applications today. With that in mind, what makes Rust so special?

(there the model inserts 5 empty lines for some reason)

To me, C++ is the way to go. Rust is not a perfect language. Rust will not be used in every software.

With C++11 and C++17, the language is starting to get better. New libraries are being created in a rapid way and the language is evolving. With the language used to build web applications, you can easily forget that the language is rather old.

I like Rust a lot, but I see it as a tool. I don't think Rust can replace C++. That doesn't make it a bad thing. In fact, I think that we should replace C++ with Rust, as we did with C. The "new" language that we are using for building web applications is not a new language, but rather Rust.

There is no good reason not to use Rust instead of C++. Today, C

Re: GPT-J-6B: 6B JAX-Based Transformer

#54

Post by PoignardAzur »

Prompt: Emmanuel Macron

---

Emmanuel Macron lors d’une allocution devant les électeurs de Bouches-du-Rhône, le 25 février 2017. BERTRAND LANGLOIS / AFP

Ils étaient jusqu’à quatre candidats à la présidence de la République à visiter le « 48 », ce quartier du centre-ville de Marseille, dans lequel un mouvement revendicatif a été mené depuis le début du mois de février et qui a été déclaré sinon complètement sécurisé, du moins relié d’une façade. Emmanuel Macron, François Fillon, Jean-Luc Mélenchon et Nicolas Dupont-Aignan, et avant eux Benoît Hamon, Marine Le Pen, Jean-Luc Mélenchon, Jean-Luc Bennahmias… Ils se sont retrouvés à la « Nouvelle République », la structure adossée au parvis de la préfecture, pour observer la police et écouter

Re: GPT-J-6B: 6B JAX-Based Transformer

#55

Post by PoignardAzur »

PoignardAzur wrote: post #54

Prompt: Emmanuel Macron --- Emmanuel Macron lors d’une allocution devant les électeurs de Bouches-du-Rhône, le 25 février 2017. BERTRAND LANGLOIS / AFP Ils étaient jusqu’à quatre candidats à la présidence de la République à visiter le « 48 », ce quartier du centre-ville de Marseille, dans lequel un mouvement revendicatif a été mené depuis le début du mois de février et qui a été déclaré sinon complètement sécurisé, d…

Translation:

---

Emmanuel Macron during an address to voters in Bouches-du-Rhône, February 25, 2017. BERTRAND LANGLOIS / AFP

They were up to four candidates for the presidency of the Republic to visit the "48", this district of downtown Marseille, in which a protest movement has been conducted since the beginning of February and which has been declared if not completely secure, at least connected with a front. Emmanuel Macron, François Fillon, Jean-Luc Mélenchon and Nicolas Dupont-Aignan, and before them Benoît Hamon, Marine Le Pen, Jean-Luc Mélenchon, Jean-Luc Bennahmias... They met at the "Nouvelle République", the structure next to the square of the prefecture, to observe the police and listen

(translation courtesy of DeepL, with tweaks)

Re: GPT-J-6B: 6B JAX-Based Transformer

#56

Post by PoignardAzur »

PoignardAzur wrote: post #54

Prompt: Emmanuel Macron --- Emmanuel Macron lors d’une allocution devant les électeurs de Bouches-du-Rhône, le 25 février 2017. BERTRAND LANGLOIS / AFP Ils étaient jusqu’à quatre candidats à la présidence de la République à visiter le « 48 », ce quartier du centre-ville de Marseille, dans lequel un mouvement revendicatif a été mené depuis le début du mois de février et qui a été déclaré sinon complètement sécurisé, d…

Some interesting points:

- The model notices that something talking about Emmanuel Macron is likely to be a french journal article, so it outputs in French language.

- The model decides that a sentence starting with "Emmanuel Macron" with no other context is likely to be the caption of a photo. At least that's how I interpret the first sentence.

- I checked and I don't "the 48" is a thing in Marseille. I could be wrong.

- The narrative of the text seems a little confused, but is roughly that presidential candidates are visiting Marseille after civil unrest has been quashed by police, and are talking to the police? Accurate details: the date is february 2017, which is campaigning season; Marseille is indeed in Bouches-du-Rhône; people mentioned were presidential candidates, except for Jean-Luc Bennahmias (didn't even know who he was until now). Major plot hole: why would rival presidential candidates meet each other, and speak with the police together, while campaigning?

- Small incoherences that hints at a lack of object permanence: the beginning mentions "up to 4 candidates", the end mentions 7. Mélenchon is mentioned twice (too bad it didn't pick Macron twice instead, there would have been political jokes to make).

- I love that the first sentence ends with "[Journalist name] / AFP". Agence France Presse is the french equivalent of Associated Press, and basically the source of all mainstream news in France. Mainstream journals mostly repackage AFP content for public consumption (which, honestly, probably means they're going to get hit hard by the coming wave of text AI). The fact that the model includes it kind of feels like these image GANs that learn to produce watermarks on the images they generate.

Re: GPT-J-6B: 6B JAX-Based Transformer

#57

Post by datenhorst »

dismantlethesun wrote: post #50
29athrowaway wrote: post #35

I took one of the questions from this article: https://blog.seekwell.io/gpt3 Instruction: Given an input question, respond with syntactically correct PostgreSQL. Be creative but the SQL must be correct. Input: how many users signed up in the past month? And it gave me a pretty nonsensical answer compared to the output from GPT-3 seen in that article.

I haven’t properly investigated their dataset they trained on this, but from the responses it gives I would bet that they dataset is skewed towards news articles, press releases, and infomercial or edutainment copy. It does really well when it can write a long impersonal article about something, but doesn’t seem good at the random sort of things GPT3 can do like autocompleting source code.

That makes a lot of sense. I played around with it in German and Italian and the results often times read like they were google translated from English. So there must be lots of auto-translated content from spam sites in the corpus, I guess.

Re: GPT-J-6B: 6B JAX-Based Transformer

#58

Post by shoto_io »

I gave it: "HackerNews is...". That's what I got:

---

HackerNews is...trying really hard to be hip to everything.

Big national media must have gotten word of a new internet meme, because every day there’s a story on it that makes its way to the front page. Many of these stories are so obviously biased they’re ridiculous.

Something about the Bitcoin Cash “fork” that split up from the Bitcoin “mainnet” in November? That’s still Bitcoin. So Bitcoin Cash isn’t Bitcoin. Bitcoin Cash doesn’t count as Bitcoin. What about Bitcoin Cash?

These headlines are a case study in how someone with no Bitcoin experience (or really any real understanding of money) can reach the front page of a big national media site with the headlines they write.

BTC/BCH is Bitcoin. We can all agree on that. Because Bitcoin is the original blockchain, the Bitcoin fork was technically not a Bitcoin fork at all.

In theory, we could run Bitcoin forked into as many different blocks as we want. We could have as many Bitcoins as we want. All those separate Bitcoin blockchains could coexist. There’s more to a blockchain

Re: GPT-J-6B: 6B JAX-Based Transformer

#59

Post by radarsat1 »

inciampati wrote: post #40
hooande wrote: post #24

These results are terrible. Almost all of the generated text posted here is non-sensical, and playing with it online is just confusing. There really needs to be a better method of evaluation, an MNIST for transformer text generation. like a list of pre-defined prompts that every GPT-X has to use to generate text, which can be scored against a list of "correct" answers in a variety of ways. I have no way of knowing if…

You have an idea of what's sensical and not. It would seem that long range correlations in the text break down. This is true of all of the popular models, even the trillion parameter ones. They just break down at longer distances.

I suspect this is at least partly because of they way they are used (and maybe trained?)

Always, "given a prompt, keep talking." No instructions to go anywhere, so it's no surprise that they do not.

I think, "start with this idea, end with this one" should give much more interesting results. Telling it to start with a premise and come up with the filler needed to draw some conclusion. It would give it more of a target for making long-distance connections.

Otherwise you just get this open-loop blabbering, which I agree seems really useless. With a more "directed" model I can see this having actual applications, like with story writing or interactive video games. But as it stands this seems totally uninteresting from an applications point of view.

Re: GPT-J-6B: 6B JAX-Based Transformer

#60

Post by exikyut »

voldacar wrote: post #39
foobiekr wrote: post #8

Sorry guys, I think I broke it. I wanted to test it on my motivational speech, Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around u…

This is simply incredible, I would love to read something novel-length like this. But unfortunately it also generates a lot of banal text, as in some of the other examples in this thread. I wonder if there is a way to optimize for the weird shit.

Convince someone with a really big pile of TPU credits to train a larger-size model on the full text of the SCP Wiki :>

This could even potentially be interesting enough to attract (small-scale) Kickstarter-level attention.

Post reply on HN