31–40 of 82 posts
I made a system that dynamically generates agents. This looks similar, so I will take a look into it! Great work :)
These and similar attempts are like those "perpetual motion machines" that captivated people's imagination for a long time. That the machine will generate energy and that would consume energy and thus will keep moving forever. Similar hopes here and everywhere. A model will generate its own input and will watch it's own output and in process, will become more intelligent than it really is.
That's not a fitting analogy because there is no equivalent to the laws of thermodynamics that says a system can't be self improving. You're dismissing a technological possibility without much reason. We are extremely similar genetically to our ancestors of 100k years ago. The big difference is cultural inheritance. We come up with ideas and objects and pass them down. Humans have improved the capabilities of humans.…
we have NOT evolved where we are today by generating inputs to ourselves, watching our own output and modifying our inputs to watch again our outputs.
Rather - we had fundamental strong reasoning almost flawless rigorous (formally documented) capabilities from the get go. Interference, induction, deduction. LLMs have none of that and it's documented all over multiple times over.
We have built at that on top of those faculties again - by not giving input to us and observing our outputs rather poking around into the world with are reasoning and cognitive faculties and arranging/categorising carefully what we discovered.
So for fanboys, this surely is very huge and I respect that sentiment wholeheartedly.
EDIT: Note about reasoning
If the author reads this: would highly recommend reading and thinking for a bit about safety :). Great project! https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
I built a few multi agent systems and went down a rabbit hole where I reached an important conclusion - From the perspective of the LLM, the prompt/context is the only thing that ever matters. Everything about how your agent will behave ultimately boils down to this. I had a bunch of fancy stuff like agents collaborating by passing messages and interpreting them with their own prompts and function calls. Then I reali…
if you can get away with using GPT-3.5 you got speed and decent pricing
These and similar attempts are like those "perpetual motion machines" that captivated people's imagination for a long time. That the machine will generate energy and that would consume energy and thus will keep moving forever. Similar hopes here and everywhere. A model will generate its own input and will watch it's own output and in process, will become more intelligent than it really is.
You're aware this is basically how AlphaGo was trained - playing against itself?
Earlier quoted context omitted.
That's a separate project called ReplicatorAgent. Unfortunately the Stargate program has had a number of run ins with them and the Air Force is merciless in shutting those kinds of projects down.
Why the Air Force specifically?
We do something similar with Magic Loops[0], but within the context of generating a single "loop" (automation). We've found that LLMs are pretty bad at prompting other LLMs, unless the problem at hand is very limited in scope. It's too easy to get incorrect/expensive behavior otherwise (e.g. starts building a framework against an imaginary API, instead of using an existing tool). Our approach looks more like a state…
1. Try your hardest not to die.
2. Try your hardest to reproduce.
3. Try to thrive maybe also while at it, but DON'T FORGET 1 AND 2!
This is why humans are this weird combination of instinct and reasoning that we find hard to control. The limbic system is very basic, but very strong and very stable. The primary loop. Everything else allows us to go farther, but when we fail we go back to relying on the limbic system.
If we allowed reason to take over instinct, we'd likely end ourselves. Not that we can't end ourselves by also relying on instinct but essentially the strategy there is: sacrifice the intelligent shell and let the basic loop continue so we can rebuild the intelligent shell in a new, more stable and suited for the environment form.
I built a few multi agent systems and went down a rabbit hole where I reached an important conclusion - From the perspective of the LLM, the prompt/context is the only thing that ever matters. Everything about how your agent will behave ultimately boils down to this. I had a bunch of fancy stuff like agents collaborating by passing messages and interpreting them with their own prompts and function calls. Then I reali…
yes, but multiprompts are fragile while dividing LLM work over multiple calls more stable if you can get away with using GPT-3.5 you got speed and decent pricing
Earlier quoted context omitted.
That's not a fitting analogy because there is no equivalent to the laws of thermodynamics that says a system can't be self improving. You're dismissing a technological possibility without much reason. We are extremely similar genetically to our ancestors of 100k years ago. The big difference is cultural inheritance. We come up with ideas and objects and pass them down. Humans have improved the capabilities of humans.…
It’s not that straightforward. Quoting Yann LeCun from a LinkedIn post [1]: > I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes. Here is the argument: Let e be the probability that any generated token exits the tree of "correct" answers. Then the probability that an answer of length n is correct is (1-e)^n. > Errors accumulate. The probability of correctness decreases exponentia…
LLMs and autoregression are very good at avoiding the 99.999[...]% of the strings that are simple gibberish. If you were to generate a string made of 20 random tokens from GPT2 tokenizer, you would get something like:
"automakersGrand carries liberties Occupations ongoingOULDessing heartbeat Pillar intrigued Trotskymediatelyearable founding examinations lavAg redesign folds"
and of course any half decent language model does much better than that. If the "paths to truth" were as unlikely as LeCun puts it there would be no hope.
A non-autoregressive model would certainly be "better" because it would be faster, which is where language models started from (BERT & co.), it just doesn't seem to work as well... similarly to how a human sometimes needs to write something down and only realizes the correct answer to a complicated question on the go.
If anything, we'd need to allow LLMs to realize mistakes and correct themselves out of them, i.e. making the generation non-linear. If you ask GPT4 something complicated (like math) it's not rare at all that it logically contradicts itself in their answers. I would be surprised if, somewhere deep in the model, it doesn't "realize" this, but it can't fix it, so it falls back to what humans do at an exam or interview that started badly: try to bullshit their way out of the thing, sweeping the inconsistency under the carpet, unless you explicitly point it to them (and often even after that, both GPT4 and humans).
P.S. Mathematician rant: who on Earth calls a probability "e"??