A deep dive into self-improving AI and the Darwin-Gödel Machine

but how its that works then??? does that mean your genetic trait is already there in the first place

if its already there in the first place there must be something that start it now right, which basically counter your argument

So I am pretty skeptical of using such unsophisticated methods to create or improve such sophisticated artifacts.

Ofc it's stochastic and sooner or later such a system will "break out", but if by then sufficient "superior systems" with good behavior are deployed and can be targeted to hunt it, the chance of it overpowering all of them and avoiding detection by all would be close to zero. At cosmic scales where it stops being close to zero, you're protected by physics (speed of light + some thermodyn limits - we know they work by virtue of the anthropic principle, as if they didn't the universe would've already been eaten by some malign agent and we wouldn't be here asking the question - but then again, we're already assuming too much, maybe it has already happened and that's the Evil Demiurge we're musing about :P).

I believe the issue with „TDD“ is the notion that it should drive design and more importantly that it‘s always applied. I disagree with both if those.

Given a problem where test first makes sense, I prefer roughly this procedure:

1. Figure out assumptions and guarantees.

2. Design an interface

3. Produce some input and output data (coupled)

4. Write a test that uses the above

5. Implement the interface/function

The order of 4 and 5 aren‘t all that important actually.

My experience is that an AI is pretty good at 3, at least once you defined one example, it will just produce a ton of data for you that is in large parts useful and correct.

Step 4 is very easy and short. Again, AI will just do it.

Step 5 is a wash. If it doesn‘t get it in a few tries, I turn it off and implement myself. Sometimes it gets it but produces low quality code, then I often turn it off as well.

Step 1-2 are the parts that I want to do myself, because they are the significant pieces of my mental model of a program.

I believe this is also how evolutionary/genetic programs usually work if you squint. They operate under a set of constraints that are designed by a human (researcher).