Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Try to guess if code is real or GPT2-generated

doesnotexist.codes

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Try to guess if code is real or GPT2-generated

#101

Post by thombat »

cryptica wrote: post #55

I was always able to correctly identify GPT2 but on a few occasions, I misidentified human-written code as being written by GPT2. Usually when the code was poorly written or the comments were unclear. GPT2's code looks like correct code at a glance but when you try to understand what it's doing, that's when you understand that it could not have been written by a human. It's similar to the articles produced by GPT3; t…

I ran into some value setters which simply logged their parameters but did not store them. Hah! obviously GPT2, not understanding that a setter should, well, set.

Wrong, of course. Now maybe the human concerned was far down some inheritance tree and simply wanted to document misuse of a deliberately limited class but assert()ing would have been too punitive. Or may it was indeed "right form but no substance", but authored by an SWE.

Re: Try to guess if code is real or GPT2-generated

#104

Post by cryptica »

thombat wrote: post #101
cryptica wrote: post #55

I was always able to correctly identify GPT2 but on a few occasions, I misidentified human-written code as being written by GPT2. Usually when the code was poorly written or the comments were unclear. GPT2's code looks like correct code at a glance but when you try to understand what it's doing, that's when you understand that it could not have been written by a human. It's similar to the articles produced by GPT3; t…

I ran into some value setters which simply logged their parameters but did not store them. Hah! obviously GPT2, not understanding that a setter should, well, set. Wrong, of course. Now maybe the human concerned was far down some inheritance tree and simply wanted to document misuse of a deliberately limited class but assert()ing would have been too punitive. Or may it was indeed "right form but no substance", but aut…

Haha I also got tricked a few times by this sort of thing. One of the ones I got wrong was there was a comment which said '// end of for loop' after every closing brace of every 'for' loop and I thought that there is no way that a human would ever write such pointless comments...

I did manage to get 100% correct after a while but it takes a thorough reading of the code.

Re: Try to guess if code is real or GPT2-generated

#105

Post by blueblimp »

It's fascinating that the main reason this is hard is that some of the human code is bad enough that it's hard to believe it's not GPT-2 output. (The first time this happened for me, I had to look it up to convince myself it's really human code.)

It reminds me of how GPT-3 is good at producing a certain sort of bad writing.

My guess as to why this happens: we humans have abilities of logical reasoning, clarity, and purposefulness that GPT doesn't have. When we use those abilities, we produce writing and code that GPT can't match. If we don't, though, our output isn't much better than GPT's.

Re: Try to guess if code is real or GPT2-generated

#106

Post by sli »

reillyse wrote: post #89
dnautics wrote: post #88

Earlier quoted context omitted.

There was one sample where the code written by gpt-3 had nonsensical grammar in the comments and another where it had an overloaded > operator with 3 parameters (is that even valid c++?)

I'd say always using proper grammar in your comments would be a dead giveaway :)

I generally find spelling to be a much better tell than grammar. Most programmers I've known had (mostly) fine grammar but horrendous spelling. I frequently end up being the proofreader.

Which is fine with me.

Re: Try to guess if code is real or GPT2-generated

#107

Post by hntrader »

zaik wrote: post #80
wnoise wrote: post #50

Earlier quoted context omitted.

In [2]: scipy.stats.binom_test(89, 200, 0.5) Out[2]: 0.13736665086863936 Unusual to be this lopsided (1-in-7), but not crazy.

Reminder that the p-value of a test is NOT the probability of H0 being true, see [0]. It only shows that, if we assume a significance of 0.05 we cannot reject the hypothesis (in our case that 89/200 is the result of a binomial distribution with p=.5). [0] https://en.wikipedia.org/wiki/Misuse_of_p-values#Clarificati...

It only shows the probability of observing either that statistic, or something even more extreme, under the null distribution. The implication that we can then "reject the null hypothesis" is more parlance and heuristic than anything.

Re: Try to guess if code is real or GPT2-generated

#108

Post by tpoacher »

jwilk wrote: post #92
tpoacher wrote: post #59

There is a "codes" top-level domain? Codes? CODES?? What's next? Advices? Feedbacks? Rests? I give ups.

"codes" as in "coupon codes". From https://icannwiki.org/.codes >: The .CODES TLD is attractive and useful to end-users as it better facilitates search, self-expression, information sharing and the provision of legitimate goods and services.

Thank you. Faith in humanity restored just a tiny bit xD
Post reply on HN