The part of this sequence which I think breaks down is a model which can day trade at a consistently winning rate. Does this exist for today? If it does, please sign me up.
Like most of these get-rich-quick schemes, there's probably no money in actually doing it, but lots of money selling "courses" supposedly teaching people how to make money day trading or betting. You don't even need to rent a supercar and a mansion for your photoshoot anymore, just generate some images "proving" your success.
An AI could make money by writing and selling slop books online via Amazon or elsewhere like direct ebook sales. If the books were at least middling there would be an actual honest to god career there, especially for niches like fan fics and under served IPs.
> not relied on a buggy software sandbox. Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions How were they supposed to know about "previously unknown vulnerabilities"? > This is pretty clearly a marketing stunt by OpenAI, otherwise the story just…
I think a careful and thoughtful person would reasonably expect given the circumstances that the models would look for, and might very well find, vulnerabilities in JFrog to exploit and take action accordingly. For example, it seems as if JFrog itself had broad access to OpenAI infrastructure and the internet: > In the following days, the agents exploited our internal research infrastructure and the Hugging Face plat…
And no monitoring of the proxy for unusual activity, either.
> How were they supposed to know about "previously unknown vulnerabilities" Very simply, there is no such thing as bug-free software.
If this is your standard, I challenge you to name one currently operating business that isn't criminally negligent. I'm sure there's tens to hundreds of millions of them amongst the 37% of the world with no internet connection, but actually finding them listed on the internet will be somewhat of a challenge.
No, the entire marketing campaign for these models is "it automatically has godlike powers to exploit almost any software" - if you don't air gap such a capability you are inherently allowing shit to go down, any other interpretation is "OAI folks are too stupid to design a proper test".
Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.
The article says that one agent proposed emailing someone.
Yeah it proposed it and the other agents told it not to, and it didn't
Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability? This is what OAI should've done. If they had executed this training run in such a sandbox, the model wouldn't have been capable of escaping without social engineering, and if the models somehow managed to do that to it's evaluators then that is indeed a massive p…
Oh really? Please tell me how you intend to enforce AI is only run in the magic sandbox? Harsh HN comments?
If I am evaluating an AI for safety, the last thing I would do is connect it to real-world peripherals or systems to allow it to reek havoc. That is criminal negligence at it's finest (especially if the AI is capable of committing crimes as happened here). I would place it on a system dedicated specifically for testing models, which had no NIC and no physical capability of accessing any outside system. If I wanted to know how the model might behave if given access to a certain system or set of systems, I would do it responsibly by writing simulation software which does it's best to simulate the real thing (and for networking this is already trivial to do). You could take this extremely far and simulate all kinds of things this way from basic networking to nuclear launch systems. And in the context of OpenAI, which is valued at over $1T, I have no qualms about stating that they (could) do this, because it is definitively something they could burn money on doing if they cared enough. They intentionally choose not to do so, and then have an amazed look on their faces when the model does something criminal like this.
1. They TOLD the model to "pursue advanced exploitation" to quantify its "cyber capabilities" (whatever that means). 2. The model pursues advanced exploitation. 3. "There was a incident due to dangerous actions taken by the model that no human directed" This is basically the pre-cursor of the paperclip maximizer [0], the AI executes the given order to an extend that was not considered in the order, now suddenly no-on…
OpenAI leadership had a meeting and asked themselves: "how can we drive even more hype" Someone said: "we should stage some high profile 'incident' caused by our latest software" And here we are, reading their press releases about it.
...and think "wow, it's impressive what your armed soldiers are capable of if they are not constrained by any rules. Good that you identified this problem of *checks notes* 'not telling them explicitly enough what the goal is'..."
In two years we will read a press-release about an AI-driven autonomous weapon which was supplied with infinite ammo and the target to "protect this perimeter from intruders", and how we now have to wait for it to run out of Ammo because it's so damn effective that we cannot reach it without being killed. All packaged in a semi-marketing framing on how impressively capable this company's products are...
If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox. This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up
> not relied on a buggy software sandbox. Third, while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions How were they supposed to know about "previously unknown vulnerabilities"? > This is pretty clearly a marketing stunt by OpenAI, otherwise the story just…
>How were they supposed to know about "previously unknown vulnerabilities"?
You don't. That's why you unplug the Ethernet cable.
Most engineers are required to explicitly take responsibility for the things they sign off, up to and including prison for sufficiently bad cases. Software “engineering” is the exception.
Yes, because it's not real engineering. The field is too variable and fast paced to be like civil engineering etc. Those have well defined codes because it is physics constrained. Each construction must be done separately at high cost. Software doesn't work like this. So there is much more flexibility and change and there's no stable best practice to regulate.
Information theory is its own sort of physics but that’s not the part that needs to be regulated. Psychologists are also licensed. That we still aren’t is a mystery to me at this point. Our stuff is now critical infrastructure. Not everyone works on that sort of things but enough of us do that at least part of the trade should be licensed.
Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…
> If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation. I very much agree with this - making AI companies explicitly responsible if their internal AI causes hacks etc could do a lot to improve their safety considerations. But I wonder what the liability should be when it's a third party using the AI and that AI hacks,…
The total, complete lack of culpability for the 2008 economic crash and malfeasance that led up to it has set a very very bad precedent for holding powerful, wealthy corporations and their executives responsible for crimes done by the org, or in this case by software created by the org.
Hmm, engineers are expected to know what is legal and not.
That's called lawyer. It's a special skill, not just intuition.
Ignorance is not a defense against getting arrested. Everyone with a drivers license is expected to know all traffic laws, even brand new ones. You’re expected to know all forms of theft and violence. And then you and I are absolutely expected to understand the basics of copyright and IP laws.
Lawyering is about being able to argue case law and keeping confidences. That’s how you lose your license, is doing those two things poorly.