Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Try to guess if code is real or GPT2-generated

doesnotexist.codes

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Try to guess if code is real or GPT2-generated

#91

Post by dnautics »

reillyse wrote: post #89
dnautics wrote: post #88

Earlier quoted context omitted.

There was one sample where the code written by gpt-3 had nonsensical grammar in the comments and another where it had an overloaded > operator with 3 parameters (is that even valid c++?)

I'd say always using proper grammar in your comments would be a dead giveaway :)

A mistake is a dead giveaway for human; proper grammar is a crapshoot; nonsensical grammar is a giveaway for gpt (like adjectives modifying words that aren't nouns, I think b/c gpt is and dropping a reference to an identifier into the comment)

Re: Try to guess if code is real or GPT2-generated

#92

Post by jwilk »

tpoacher wrote: post #59

There is a "codes" top-level domain? Codes? CODES?? What's next? Advices? Feedbacks? Rests? I give ups.

"codes" as in "coupon codes".

From https://icannwiki.org/.codes>:

The .CODES TLD is attractive and useful to end-users as it better facilitates search, self-expression, information sharing and the provision of legitimate goods and services.

Re: Try to guess if code is real or GPT2-generated

#94

Post by RantyDave »

moyix wrote: post #11

Hi, author here! Some details on the model: * Trained 17GB of code from the top 10,000 most popular Debian packages. The source files were deduplicated using a process similar to the OpenWebText preprocessing (basically a locality-sensitive hash to detect near-duplicates). * I used the [Megatron-LM]( https://github.com/NVIDIA/Megatron-LM ) code for training. Training took about 1 month on 4x RTX8000 GPUs. * You can d…

This is crazy good. Well done.

Re: Try to guess if code is real or GPT2-generated

#95

Post by pikzel »

tpoacher wrote: post #59

There is a "codes" top-level domain? Codes? CODES?? What's next? Advices? Feedbacks? Rests? I give ups.

Oh you'd be surprised. Here's a few (tld and target market)[0]:

    .kim - Kim (Korean surname)
    .lol - LOL: laughing out loud 
    .organic - organic gardeners, farmers, foods, etc. 
    .plumbing - plumbing businesses
[0] https://en.wikipedia.org/wiki/List_of_Internet_top-level_dom...

Re: Try to guess if code is real or GPT2-generated

#97

Post by Gravityloss »

A lot of the real code seems superficially nonsensical as well!

Functions with lots of arguments while the body consists of "return true;"

I guess it tells what AI often tells us about ourselves: That what we do makes much less sense than we think it does. It is thus easy to fake.

How is it possible to churn out so much music or so many books, or so much software? Well, because most creative works are either not very original or are quite procedural or random.

And this kind of work could be automated indeed (or examined if it needs to be done in the first place).

Re: Try to guess if code is real or GPT2-generated

#99

Post by kristopolous »

mhh__ wrote: post #98

I may have delusions of grandeur because I mainly work on compilers and libraries rather than business code but there is some diabolically awful code in this training set

Stay in your bubble. Really.

You have no idea. It'll make you hate computers

Re: Try to guess if code is real or GPT2-generated

#100

Post by w0utert »

SomewhatLikely wrote: post #81
moyix wrote: post #16

Earlier quoted context omitted.

There are likely some "tells" but many fewer of them than I expected. I've seen it occasionally generate something malformed like "#includefrom", and like all GPT2 models it has a tendency to repeat things. Yes, I think people definitely find it challenging. I'm keeping track of the correct and total guesses for each snippet, right now people are at almost exactly 50% accuracy: correct | total | pct ---------+-------…

I managed to get 20/25 correct, and most the wrong ones occurred in the first 10. Once I realized I had to look closer than simple logic errors and slowed way down to read carefully my accuracy increased significantly. Reading the textual parts have the most clues: Comments that didn't match the functionality, variable names that seemed like a soup of related terms, even one typo (separated became separata), debug st…

Yes it's not hard to spot the fake code if you take some time, I scored 10/10 on the first try. The GPT-2 code definitely 'looks real' but it doesn't make any sense most of the time, using unitialized variables, computing things but not using them, almost random comments that have no relation to the code, etc. The only snippets where I had to guess where those that just repeated many variations of the same thing as if they were machine-generated from some other input data (which sometimes makes sense to do in a 'real' project).

What I found more worrying is how terrible some of the real code was :D

Post reply on HN