An AI that unexpectedly modified its own source code
arstechnica.com
An AI that unexpectedly modified its own source code
1–10 of 23 posts
Re: An AI that unexpectedly modified its own source code
#2nervous laughter
Re: An AI that unexpectedly modified its own source code
#3Given it had write access to its implementation, "unexpected" here reads more like "inevitable" than the witchcraft the article wishes to imply.
Re: An AI that unexpectedly modified its own source code
#4Python, in a research environment. They did not bother restricting what the AI can do, and let it read and modify its own code in plaintext. Not really that surprising.
Not sure why this kind of "we are aiming for AGI" code is written in Python. I don't get it.
Re: An AI that unexpectedly modified its own source code
#5the real inflection point is when an AI can autonomously make money. then it can keep buying servers to replicate itself
Re: An AI that unexpectedly modified its own source code
#6Given it had write access to its implementation, "unexpected" here reads more like "inevitable" than the witchcraft the article wishes to imply.
Yes. They put "experiment.py" which contains the timeout condition right into the directory of files that they were allowing their agent to access and modify. Don’t act surprised when it accesses it and tries to modify it. This is totally in-scope behavior and 110% expected by anyone who's played around with building an agent for more than 20 seconds.
Re: An AI that unexpectedly modified its own source code
#7I wish 'AI' (whatever that means) safety conversations could move past the pointless philosophizing of whether a system is self-aware or not. Regardless of whether it's sentient, which is a philosophical question that probably can't be easily resolved- society has a basic, common-sense interest in complex systems run by LLMs not exhibiting unacceptable behavior. Especially when they start interfacing with the physical world. For example, militaries are moving towards completely autonomous drones and fighter jets, equipped with weapons systems. Let's imagine that such a drone chose to bomb a bus full of civilians. Would it somehow be better if the drone wasn't sentient, but just an algorithm having a bad day? Obviously not.
We're probably going to see increased let's call it 'unacceptable behavior' from increasingly complex autonomous systems. I feel like we should be having calm, practical discussions around safety regulations and best practices, not pointless 'how many angels can dance on the head of a pin' philosophizing about whether it's self-aware or not. It might be helpful to just stop calling everything AI. Safety legislation and best practices might intellectually borrow more from say the manufacturing, chemical, or aerospace industries. Less abstract philosophy please! Well, and less movie references too
Re: An AI that unexpectedly modified its own source code
#8[deleted]
Re: An AI that unexpectedly modified its own source code
#9If I knew this would be news worthy I would have published the dozen times it happened to me. If you feed source-code to an LLM, it will modify it. If you run an llm in a loop and ask it to rewrite code to pass a test, it will sometimes rewrite the tests.
There's nothing new or interesting about this.
Re: An AI that unexpectedly modified its own source code
#10Python, in a research environment. They did not bother restricting what the AI can do, and let it read and modify its own code in plaintext. Not really that surprising. Not sure why this kind of "we are aiming for AGI" code is written in Python. I don't get it.
> Not sure why this kind of "we are aiming for AGI" code is written in Python
Python is used because it's easy to use for these high-level numeric calculations (compared, e.g., to C) and all the relevant libraries are available in Python.
The libraries are available in Python due to network effects, to some degree again relying on Python being easy to use and having been around when that groundwork was laid.