A deep dive into self-improving AI and the Darwin-Gödel Machine

(I apologize, this is a bit long and a bit disorganized)

  > As merely a programmer

I hope you don't see my comment as placing myself as "better than thou". We have different skillsets, that's all.

  > You mentioned challenging your assumptions. How often does a researcher change their guarantees?

I'm not quite sure how to answer this tbh. Because I don't know what you mean. I'll reference Andrew Gelman on this one[0], mostly because the cross-reference is good and his blog has a lot of other insights

  | a guarantee comes from an assumption. If you want to say that your method has a guarantee but my method doesn’t, what you’re really saying is that you’re making an assumption and I’m not.

Really what we want in science is to generate counterfactural models. I started in physics before moving to CS (ML PhD) and I can sure tell you, at least this part was clearer in physics. F=ma[1] is a counterfactual model. I can change either m or a and make predictions. I can "go back in time" and ask how things would have been different. This is how we create good models of things. It's no easy task to get there though and it is far messier when you derive these simple equations than what they end up as. Think of it not too different than having a function vs "discovering" the function in a stack trace. You gotta poke and prod inside and out because you sure as hell know it isn't nicely labeled for you and you can't just grep the source code.

But here's a difficult lesson every physicist has to learn. Experiments aren't enough. I think nearly every student will end up having an experience where they are able to fit data to some model only to later find out that that model is wrong. This is why in physics we tend to let theory drive. Our theory has gotten good enough we can do some general exploring of "the code" without having to run it. We can ask what would happen if we did x and then explore those consequences. Once we got something good, then we go test and we know exactly what to look for.

But even knowing what to look for, measurements are fucking hard (I was an experimentalist, that was my domain). Experiments are hard because you have to differentiate it from alternative explanations of the data. Theory helps a lot with this, but also isn't enough by itself.

  > How long are you allowed to chase and optimize for an outcome before you reconfigure where you're heading?

There are no hard or fast rules, it is extremely circumstantial. First off, we're always dealing with unknowns, right? So you have to be able to differentiate your known knowns, known unknowns, unknown unknowns, and importantly, your uncertain knowns. Second, it depends on how strong your convictions are and what you believe the impact would be. Do you think you have the tools to solve this right now? If not, you should continue thinking about it but shift your efforts elsewhere. Insights might come later. But you have to admit that you are unable to do that now.

What's important is figuring out what you would need to do to determine something. The skill is not that different than what we use in programming tbh. The difference really tends to be in the about of specificity. Programming and math are the same thing though. The reason we use these languages is due to their precision. When doing this type of work we can't deal with the fuzzy reality of natural language. And truth is, the specificity depends on your niche. So it all comes down to how strong your claims are. If you make strong claims (guarantees) you need extreme levels of specificity. First place people will look is assumptions. It's easy to make mistakes here and they will unravel everything else. But sometimes that leads to new ideas and can even improve things too.

So as a ML researcher, I love LLMs but hate the hype around them. There's no need to make such strong claims about AGI with them. We build fuzzy compression machines that can (lossy) compress all human knowledge and this can be accessed through a natural language interface. That's some fucking Sci-Fi tech right there! It feels silly to say they are more. We have no evidence. The only thing that results in is public distrusting us more when they see these things be dumb. Tech loves its hype cycles, like Elon promising that Teslas will be fully autonomous next year. A prediction he's made since 2016. Short term gains, but it is a bubble. If you can't fill the void in time, it pops and you harm not just yourself but others. That's a big problem.

Me? I just want to make progress towards making AGI. But I speak up because we don't even know what that looks like. We made massive leaps recently and we should congratulate ourselves for that. But with every leap forward we must also reflect. Success comes with additional burdens. It requires us to be more nuanced and specific. It means, what we likely need to do things differently. Gradient descent will tell you the same thing. You can make large gains in the beginning by taking non-optimal (naive) large steps towards what you think the optima is. But as you get nearer and nearer to the optima you can no longer act so naively and still make progress. Same is true here. Same is true if you look at the history of any scientific subject. You'll see this in physics too![2]

So to answer your question, how long? Well it depends on the progression of success and reflection after any milestones. We revisit "can I do this with the tools I have now", "what tools do I need", "can I make those tools", and "how would I find out". Those questions never stop being asked.

[0] https://statmodeling.stat.columbia.edu/2019/07/22/guarantee-...

[1] Technically this isn't the full form. But that is fine. In physics we deal with approximations too. They're often the most important parts. This is good enough for our purposes.

[2] https://hermiene.net/essays-trans/relativity_of_wrong.html

Zoolander?

Great answer thank you!

I guess this really depends on the problem but from the PromptWizard (PW) paper:

    | Approach | API calls | IO Tokens | Total tokens  | Cost ($) |
    |----------|-----------|-----------|---------------|----------|
    | Instinct | 1730      | 67        | 115910        | 0.23     |
    | InsZero  | 18600     | 80        | 1488000       | 2.9      |
    | PB       | 5000      | 80        | 400000        | 0.8      |
    | EvoP     | 69        | 362       | 24978         | 0.05     |
    | PW       | 69        | 362       | 24978         | 0.05     |

They ascribe this gain in efficiency to a balance between exploration and exploitation that involves a first phase of instructions mutation followed by a phase where both instruction and few-shot examples are optimized at the same time. They also rely on "textual gradients", namely criticism enhanced by CoT, as well as synthesizing examples and counter-examples.

What I gathered from reading those papers + some more is that textual feedback, i.e. using a LLM to reason about how to carry out a step of the optimization process is what allows to give structure to the search space.

I agree that symbolic processing still has a role - but I think it's the same role it has for us: formal reasoning. I.e. a specialized tool.

"Logical consistency" is exactly the kind of red herring that got us stuck with symbolic approach longer than it should. Humans aren't logically consistent either - except in some special situations, such as solving logic problems in school.

Nothing in how we think, how we perceive the world, categorize it and communicate about it has any sharp boundaries. Everything gets fuzzy or ill-defined if you focus on it. It's not by accident. It should've been apparent even then, that we think stochastically, not via formal logic. Or maybe the Bayesian interpretation of probabilities was too new back then?

Related blind alley we got stuck in for way longer than we should've (many people are still stuck there) is in trying to model natural language using formal grammars, or worse, argue that our minds must be processing them this way. It's not how language works. LLMs are arguably a conclusive empirical proof of that.