Ask HN: Would a self-aware AI be able to spot a buffer overflow in itself?

Unless you have some radical new processor design plans up your sleeve, your AI will run on a register machine. RMs can be shown to be equivalent to the Turing machine, the Turing machine cannot decide the halting problem (which your question is an instance of).