Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

stefanzukin.com

Post reply on HN
Print view View original

81–90 of 107 posts

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#81

Post by cwkoss »

pjc50 wrote: post #41
jvanderbot wrote: post #11

Quite easy when you know one is fake. Flagging fake articles in a review queue, by abstract only, and when none may exist all the way up to all being fake .... Now that's a challenge. Also, if you train GPT on the whole corpus of Nature / Science / whatever articles up to, say, 2005, could you feed it leading text about discoveries after 2005 and see if it hypothesizes the justification for those discoveries in the s…

I find the whole thing ominous because there is no "there" there: there is no understanding in the GPT-2 system, but it's able to generate increasingly plausible text. This greatly increases the amount of plausible nonsense that can be used to drown out actual research. You could certainly replace a lot of pop-sci and start several political movements with GPT-2... all of which has no actual nutritional content.

Makes you wonder if humans also have less "there" than we give ourselves credit for. How much of sentence construction is just repeating familiar tropes in barely-novel ways?

To what extent have our brains already decided what to say while we still perceive ourselves as 'thinking about the wording'?

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#82

Post by sellyme »

pjc50 wrote: post #41
jvanderbot wrote: post #11

Quite easy when you know one is fake. Flagging fake articles in a review queue, by abstract only, and when none may exist all the way up to all being fake .... Now that's a challenge. Also, if you train GPT on the whole corpus of Nature / Science / whatever articles up to, say, 2005, could you feed it leading text about discoveries after 2005 and see if it hypothesizes the justification for those discoveries in the s…

I find the whole thing ominous because there is no "there" there: there is no understanding in the GPT-2 system, but it's able to generate increasingly plausible text. This greatly increases the amount of plausible nonsense that can be used to drown out actual research. You could certainly replace a lot of pop-sci and start several political movements with GPT-2... all of which has no actual nutritional content.

> there is no understanding in the GPT-2 system, but it's able to generate increasingly plausible text

Under some definition of "understanding". GPT understands how to link words and concepts in a broadly correct manner. As long as the training data is valid, it's very plausible that it could connect some concepts that it genuinely and correctly understands are compatible, but doing so in a way which humans had not considered.

It can't do research or verify truth, but I've seen several examples of it coming up with an idea that as far as I can tell had never been explored, but made perfect sense. It understood that those concepts fit together because it saw a chain connecting them through god-knows how many input texts, yet a human wouldn't have ever thought of it. That's still valuable.

As to how far that understanding can be developed... I'm not sure. It's hard to believe that algorithmically generated text would ever be able to somehow ensure that it produces true text, but then again ten years ago I would have scoffed at the idea of it getting as far as it already has.

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#84

Post by iamgopal »

ronsor wrote: post #23

Even hard mode isn't that hard because GPT-2 tends to ramble on while saying nothing substantive. If I can't figure out what a paper is supposed to be talking about, it's fake. 4/4 on hard. Never read a Nature paper before.

Same for me, hard mode is quite easy, our brain is pattern matching Engine and constantly try to make sense of this and hence it may look like legitimate, it wouldn’t if text were 2D.

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#85

Post by jackcviers3 »

6 and 2 on hard mode. The failure of the model to connect ideas in long paragraphs (or to make a succinct claim) is what gives it away. It introduces far too many terms with far too little repetition and far too much specificity in such a short span.

Suggested tweak - train it against papers written by people with an Erdos number Another great corpus for complex writing is public law books. Have it compare real laws from the training set with fake laws. I bet it would be very difficult to figure out the fake laws.

Training one of these on an entire corpus of one author (Roger Ebert, Justice Ginsberg, Joyce, anyone with a large enough body of work), and having people spot the fake paragraphs from the real ones would be very, very difficult. An entire text, however, would likely be discernible.

It is getting really, really close to being able to fool any layman, though. Impressive work!

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#86

Post by not2b »

ta988 wrote: post #33

I'm sure GPT2 abstracts would fly through many conferences screening processes. I've seen talks and posters that were utter non-sense but everybody was too polite to say anything to the person or advisors. I've reviewed articles that were completely made up and the other reviewer didnt even detect that. Nor did the editor. I've contacted editors about utterly wrong papers, criticized the article on pubpeer, and the a…

Abstracts are relatively short, so for the length of an abstract GPT-2 might just be fusing together the abstracts of two or three related papers, so the result might look legit. It tends to wander around when the length is increased, though, and if asked to go on for long enough it will lose the plot.

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#88

Post by userbinator »

jacquesm wrote: post #44

The side-by-side display makes it pretty easy to distinguish the one from the other, simply compare them at a level where the one that makes the least sense is the one that is nonsense. Like that I score 10/11. But when looking at just the left side one suddenly the problem is much harder, and I'm happy to get better than even. Bits that don't help: not an English native writer. Seen too many real life papers with cr…

and for sure it will cause trouble for search engines to classify real content from generated content.

Fortunately, SEO spam is currently nowhere near as coherent as this, and often features some phrases that are a dead giveaway ("Are you looking for X? You've come to the right place!" or a strangely-thesaurised version thereof), but I am also worried about this new generation of manufactured deception.

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#89

Post by userbinator »

jpindar wrote: post #80
jpindar wrote: post #79

Pretty easy, even in hard mode, and not due to any knowledge of the subject matter. I'm 15 - 0 so far. I kept seeing certain types of grammatical error, such as constructs like "... and foo, despite foo, so..." or "with foo, but not foo..." where foo is the exact same word or phase appearing twice in a sentence. I also kept seeing sentences with two clauses that should have agreed in number or tense but did not.

"The structure of the HIV capsid is analysed by cryo-electron microscopy and cryo-electron microscopy at cryo-electron-microscopy resolution." It really does like to repeat itself.

This one made me laugh really really hard:

"This study presents the phylogenetic characterization of the beak and beak of beak whales; it is suggested that the beak and beak-toed beaks share common cranial bones, providing support for the idea that beaks are a new species of eutriconodont mammal."

Re: Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the difference?

#90

Post by userbinator »

Some of the fake ones are hilarious:

The chicken genome (the genome of a chicken that is the subject of much chicken-related activity) is now compared to its chicken chicken-to-pecking age: from a genome sequence of chicken egg, only approximately 70% of the chicken genome sequences match the chicken egg genome, which suggests that the chicken may have beenancreatic.

(Related: https://www.youtube.com/watch?v=yL_-1d9OSdk )

Post reply on HN