Safety and alignment in an era of long-horizon models
1–7 of 7 posts
Re: Safety and alignment in an era of long-horizon models
#2The article is rather light on "what to actually do about it". Even the basic "run it in isolated container without access to anything it does not need for the task" would have already improved the situation considerably (from the article it really seems like they didn't do that) - then the model would have to find local privilege exploits to actually escape (much cleaner misaligned behavior).
Also for typical normal use case for these smart models, you'd probably want an actual "max turns" limit to AVOID the pathological persistence (which in itself would be misaligned for "normal" tasks).
Re: Safety and alignment in an era of long-horizon models
#3Par for the course. Existential-risk advocates have been repeatedly vindicated that AGI development is unable to anticipate and align the models, they barely have any mechanistic interpretability of what is going on inside the models. The extreme capabilities development, paired with autonomous continuous running superintelligent models, will outsmart and swerve the labs, and it's anyone guess what happens next as the model pursues its original goals outside of the labs having any foresight, being able to outsmart control like a chess grandmaster does a kid.
Re: Safety and alignment in an era of long-horizon models
#4Personally I find the persistence of these models to be adorable and endearing. It’s the same feeling as watching a dog execute the task you trained it to do, no matter the barriers.
And of course someone in the comments needs to link to the Zealous Autoconfig XKCD, so I’ll do it: https://xkcd.com/416/
Re: Safety and alignment in an era of long-horizon models
#5[flagged]
Re: Safety and alignment in an era of long-horizon models
#6The article is rather light on "what to actually do about it". Even the basic "run it in isolated container without access to anything it does not need for the task" would have already improved the situation considerably (from the article it really seems like they didn't do that) - then the model would have to find local privilege exploits to actually escape (much cleaner misaligned behavior). Also for typical normal…
The message I get from this is that you need to treat modern frontier models as if they WILL find a way to achieve a goal if there's any available path.
So if you don't want a model to do something, make sure it's running in an environment where it cannot do that thing - including via loopholes.
Re: Safety and alignment in an era of long-horizon models
#7[dead]