141–150 of 289 posts
This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.
Says who?
Earlier quoted context omitted.
Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…
> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ens…
I feel this completely misunderstands the problem, and the vast gulf between an LLM and a CPU.
First and most importantly, the set of behaviors of a CPU is extremely constrained, and we have a very simple model for which behaviors are safe and which are not. Writing to addresses between X and Y, executing certain instructions - unsafe; everything else, safe. In contrast, an LLM has a huge array of possible behaviors, and variations of those behaviors, and it's very unclear which are safe and which are not. Is emitting the text "sudo rm -rf /" safe? Yes, in some contexts, such as writing this HN comment ; absolutely not in others, such as generating a command that an agent will execute. How do you check which is which? What if it emits "sudo rm -rf /usr/sbin/../.. ", is that safe?
Secondly, CPUs can absolutely be used to hack other people. Nothing in the permission model helps in any way prevent other computers from being attacked by your CPU. So exactly the part we care most about in AI security is the part that has never been solved, for any computing system ever created.
Earlier quoted context omitted.
> where that behavior was never even intended. Strongly doubt that. Did they even share the prompt?
Did you see their presentation at Blackhat? https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.
PR bullsh*t. There's no thought in a stochastic parrot.
I’m optimistic about this. A system with these agents rummaging around for a while will be much more secure than one without. We’ve learned security through obscurity is bad. Not using these will be security through ignorance. Hopefully it will push us to not only fix individual issues but close entire classes of possible gaps, once P(discovery) gets much higher.
True. There'll be no breakins at a nuclear power plant in meltdown.
Would love to see a cat and mouse game being played by openai versus anthropic, out in the open.
Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…
And you missed an even MORE relevant excerpt!!
Man and Slave
The problem, and it is a moral prob-
lem, with which we are here faced is
very close to one of the great problems
of slavery. Let us grant that slavery
is bad because it is cruel. It is, how-
ever, self-contradictory, and for a
reason which is quite different. We
wish a slave to be intelligent, to be able
to assist us in the carrying out of our
tasks. However, we also wish him to
be subservient. Complete subservience
and complete intelligence do not go
together. How often in ancient times
the clever Greek philosopher slave of
a less intelligent Roman slaveholder
must have dominated the actions of his
master rather than obeyed his wishes!
Similarly, if the machines become
more and more efficient and operate
at a higher and higher psychological
level, the catastrophe foreseen by
Butler of the dominance of the ma-
chine comes nearer and nearer.Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…
Guardrails have to be developed, and that needs testing.
Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…
> did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls. The message is that the hall is intended for gambling, but that the hall's patrons may be held liab…
Totally coherent, but actually proliferates the dangerous technology.