The goal when writing code is to be pretty machine like and to keep things extremely simple. People also write dead or off topic comments. That’s why this is so hard
Try to guess if code is real or GPT2-generated
31–40 of 108 posts
Re: Try to guess if code is real or GPT2-generated
#32Earlier quoted context omitted.
Can you share the dataset too?
Sure, it's here in JSON format: https://moyix.net/~moyix/csrc_dataset_large.json.gz
I am curious, how were you able to feed the GPUs? Did you simply preload the entire dataset into RAM (it certainly seems possible)? Did you preapply BPE? Did you train your own BPE?
Re: Try to guess if code is real or GPT2-generated
#33There was some code about TIFF headers, and it was apparently GPT2 generated
TIFF is a real thing, so some human was involved in some part of that code, it has just been garbled up by GPT2... In other words, the training set is showing quite visibly in the result
Re: Try to guess if code is real or GPT2-generated
#34I got a function that assigned the same expression to three variables. Then it declared a void function with documentation stating "returns true on success, false otherwise". Apparently that code was written by a human, which makes me either doubt the correctness of that website, or the quality of the code it was fed with
Re: Try to guess if code is real or GPT2-generated
#35This looks like overfitting to me. Some of the GPT samples were definitely real code, or largely real code. One looked like something from Xorg, another like it was straight from the COLLADA SDK. It’s really hard to define what “truly new code” is, if it’s just the same code copy pasted in different order. Blah blah Ship of Theseus etc.
Overfitting on 17GB of input data would be interesting, even though it's using the "large" 774M GPT-2 model.
It's possible training for a month may be too much.
Re: Try to guess if code is real or GPT2-generated
#366/6, quitting while I'm ahead
same here :D
Re: Try to guess if code is real or GPT2-generated
#37Earlier quoted context omitted.
Sure, it's here in JSON format: https://moyix.net/~moyix/csrc_dataset_large.json.gz
I am curious, how were you able to feed the GPUs? Did you simply preload the entire dataset into RAM (it certainly seems possible)? Did you preapply BPE? Did you train your own BPE?
Note that the encoded dataset (through the GPT-2 BPE tokenizer) will be much, much less than 17GB, both on disk and in memory (in my experience it can be anywhere from 1/3rd to 1/2 the size)
If finetuning an existing GPT-2 model, you must use that BPE tokenizer; you could theoretically use your own but that wouldn't make a difference performance-wise and you'd just have a lot of wasted tokenspace.
The efficiencies of using your own tokenizer for bespoke, esoteric content that does not match typical internet speak (like this) are why I recommend training your own tokenizer and GPT-2 from scratch if possible.
Re: Try to guess if code is real or GPT2-generated
#38I got a function that assigned the same expression to three variables. Then it declared a void function with documentation stating "returns true on success, false otherwise". Apparently that code was written by a human, which makes me either doubt the correctness of that website, or the quality of the code it was fed with
Same thought here - apparently humans read from uninitialized arrays immediately after declaring them! That said, it is still a pretty fun website :)
Re: Try to guess if code is real or GPT2-generated
#39Earlier quoted context omitted.
Thanks for stopping by! This is impressive. I would be curious to know if my hunch below about potential weaknesses/tells was at all correct. Did people find it to be as challenging when you showed it to them as some of us are here? Did you expect that level of complexity?
There are likely some "tells" but many fewer of them than I expected. I've seen it occasionally generate something malformed like "#includefrom", and like all GPT2 models it has a tendency to repeat things. Yes, I think people definitely find it challenging. I'm keeping track of the correct and total guesses for each snippet, right now people are at almost exactly 50% accuracy: correct | total | pct ---------+-------…
How can you differ between people genuinely trying to discern them, and people randomly clicking one of the buttons to see the answer? The latter would also result in a 50% accuracy, regardless of the actual GPT quality.
Re: Try to guess if code is real or GPT2-generated
#40I got a function that assigned the same expression to three variables. Then it declared a void function with documentation stating "returns true on success, false otherwise". Apparently that code was written by a human, which makes me either doubt the correctness of that website, or the quality of the code it was fed with
First code it showed me had getXXX() methods returning void, each of which contained nothing but a printf using the same string variable with no apparent connection to XXX, along with invalid format strings. Surely code this nonsensical has to be generated. Yet when I clicked "GPT2" it said I was wrong.