Ah, 0/5, I give up :)
Just invert your guesses then!
Try to guess if code is real or GPT2-generated
51–60 of 108 posts
Re: Try to guess if code is real or GPT2-generated
#52Earlier quoted context omitted.
There are likely some "tells" but many fewer of them than I expected. I've seen it occasionally generate something malformed like "#includefrom", and like all GPT2 models it has a tendency to repeat things. Yes, I think people definitely find it challenging. I'm keeping track of the correct and total guesses for each snippet, right now people are at almost exactly 50% accuracy: correct | total | pct ---------+-------…
How can you differ between people genuinely trying to discern them, and people randomly clicking one of the buttons to see the answer? The latter would also result in a 50% accuracy, regardless of the actual GPT quality.
Yep, there could be a lot of noise in there too from people guessing lazily.
Re: Try to guess if code is real or GPT2-generated
#53This is actually quite impressive. Try reading the comments in the code. The comments often make perfect sense in the local context even if it’s GPT-2 gibberish. The real examples have worse comments at times. The only flaw is that it shows fake code most of the time so you can game it that way.
Some of the "real" ones have absolutely atrocious comments. Two variables labelled with literally just the name of the variable like so:
bool hasdied // has died
Re: Try to guess if code is real or GPT2-generated
#54I got a function that assigned the same expression to three variables. Then it declared a void function with documentation stating "returns true on success, false otherwise". Apparently that code was written by a human, which makes me either doubt the correctness of that website, or the quality of the code it was fed with
Maybe the probability of GPT2 generating that sequence is nearly 0. Sometimes weird edge cases are more human.
Re: Try to guess if code is real or GPT2-generated
#55I was always able to correctly identify GPT2 but on a few occasions, I misidentified human-written code as being written by GPT2. Usually when the code was poorly written or the comments were unclear.
GPT2's code looks like correct code at a glance but when you try to understand what it's doing, that's when you understand that it could not have been written by a human.
It's similar to the articles produced by GPT3; they have the right form but no substance.
Re: Try to guess if code is real or GPT2-generated
#56Earlier quoted context omitted.
I am curious, how were you able to feed the GPUs? Did you simply preload the entire dataset into RAM (it certainly seems possible)? Did you preapply BPE? Did you train your own BPE?
Note that the encoded dataset (through the GPT-2 BPE tokenizer) will be much, much less than 17GB, both on disk and in memory (in my experience it can be anywhere from 1/3rd to 1/2 the size) If finetuning an existing GPT-2 model, you must use that BPE tokenizer; you could theoretically use your own but that wouldn't make a difference performance-wise and you'd just have a lot of wasted tokenspace. The efficiencies of…
Yeah, I trained my own BPE tokenizer for this and it results in pretty good compression. From 1024 BPE tokens you can generate anywhere from 2000-6000 actual characters of text. My guess is that it's a bit more efficient than English-BPE because there's a lot of repetitive stuff in source code (think spaces for indentation, or "if("/"while("/"for (int").
Re: Try to guess if code is real or GPT2-generated
#57I got 4/4 GPT-2 guesses right. It is impressive but the "tell" I've found so far is just poor structure in the logic of how something is arranged. For example: a bunch of `if` statements in sequence without any `else` clauses with some directly opposing prior clauses. Another example was repeating the same operation a few times in individual lines of code which most human programmers would write in a simpler manner.…
I found that a comment with license, followed by C++ code, but without any header inclusions in between, is a clear tell for GPT-2 generated code
Re: Try to guess if code is real or GPT2-generated
#58I got a function that assigned the same expression to three variables. Then it declared a void function with documentation stating "returns true on success, false otherwise". Apparently that code was written by a human, which makes me either doubt the correctness of that website, or the quality of the code it was fed with
This made me worried, so I went and spot-checked 5-6. Using the "cheat sheet" I was always able to guess correctly, so I think the site is working fine.
The list of packages the real snippets are drawn from is here (maybe if you want to avoid using them... ;) ):
https://moyix.net/~moyix/sample_pkgnames.txt
Note that the GPT samples are prompted with 128 characters randomly selected from those same packages, so you will see GPT2-generated code that mentions the package name etc. However, these packages were not used for training.
Re: Try to guess if code is real or GPT2-generated
#59There is a "codes" top-level domain? Codes? CODES??
What's next? Advices? Feedbacks? Rests?
I give ups.
Re: Try to guess if code is real or GPT2-generated
#60This looks like overfitting to me. Some of the GPT samples were definitely real code, or largely real code. One looked like something from Xorg, another like it was straight from the COLLADA SDK. It’s really hard to define what “truly new code” is, if it’s just the same code copy pasted in different order. Blah blah Ship of Theseus etc.
I'm 90% sure I just got a boost header which was apparently GPT-2 generated, hmmm.
Sadly, I can't do back and see it again