It’s just still so trivial to jailbreak even the latest Anthropic models (via api, and not talking about the silly ENI or Pliny breaks) I don’t understand where the safety teams are doing their work. Is it in the default chat-trained model?
Ask HN: Have top AI research institutions just given up on the idea of safety?
41–50 of 99 posts
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#42Not an insider but someone who uses the tools. It's a branding update, nothing more. The models haven't gotten any less sanctimonious, but the companies behind them have stopped harping on their restrictions in order to appeal to a broader customer base (gov contracts, etc.) So the guardrails (for you and me) are still there. They just stopped committing the unforced error of excluding themselves from federal procure…
I don't think it's sanctimonious to say, hey, I don't want the technology I work on to be used for targeting decisions when executing people from the sky. Especially as the tech starts to play more active roles. You know governments will be quick to shift blame to the model developers when things go wrong.
> I don't want the technology I work on to be used for targeting decisions when executing people from the sky
one problem i have with this specific case and Anthropic/Claude working with the DOD is I feel an LLM is the wrong tool for targeting decisions. Maybe given a set of 10 targets an LLm can assist with compiling risks/reward and then prioritizing each of the 10 targets but it seems like there would be much faster and better way to do that than asking an LLM. As for target acquisition and identification, i think an LLM would be especially slow and cumbersome vs one of the many traditional ML AIs that already exist. DOD must be after something else.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#43Humans can't develop safety until there is enough blood in the streets. Only issue with AI is that threshold may come at a point where its too far gone to recover. But humans can't put in seatbelts until we're losing 40k people per year in car crashes. Unfortunately its just how we're wired. Those that are careful are outcompeted by the brash and the fast-moving, until the relative value of moving fast is removed, th…
[flagged]
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#44It takes imagination and emotion to be dangerous. These token predictors will never be smart enough to be dangerous.
You’ve not accounted for the danger of necrotizing bureaucratic systems. Put these LLM drones in enough places and everything wills stagnate.
It’s effectively the start to Asimovs Foundation.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#45"safe" is such a subjective concept to begin with, have any of the model providers ever defined what they mean by "safe"? It doesn't mean much to me if a safe model is one that does not output the recipe for mustard gas, that information is trivially available elsewhere. Or, is a safe model one that doesn't come off as racist? Ok but i would classify that as unoffensive instead of safe but I admit definitions of word…
Well I do think there's some degree of unsafeness which is inexorably linked to capability--if the model when deployed with full control of a machine is capable of large scale cyberattacks and blackmailing for example.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#46Yes, in the same way that cryptocurrency leaders gave up on any notion of privacy or "freedom". In the space of a few years, you had them switch from big libertarian posturing to reporting mandatory KYC directly to tax authorities. Why? Because there's so much money to be made by abandoning principles. In the same way, the AI orgs will surrender to money.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#47Safety means slower and this is viewed as a winner takes all game.
This isn't new either, the safety glass cracked the day OpenAI publicly launched ChatGPT. "Safety" was (and perhaps still is) a fall back for the models plateauing and LLMs failing to really make an impact..."we need more time while we focus on safety"
But after this latest round of models, it's a lot more fuel on the "this could be it" fire. Labs are eager to train on the new gigawatt scale datacenters coming online, and it's very hard to make a case right now that the we won't get another step-change up in capability. Safety just obstructs all that.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#48More on the periphery than an insider, but I personally know researchers in all three major labs who were there long before GPT-3. They all care about existential safety, a lot. In the sense that they believe there's a meaningful chance all humans are dead a decade from now (and that that's a bad thing; unfortunately, there are also people deeply involved who don't think human extinction is a bad thing). The issue is…
/rant Existential in what sense? There's this one sense in which people are almost moral about it: "yup, AI is just superior to humans, nothing we can do about it." And then there's ones where the elite class implements mass surveillance and warfare and obsoletes billions of humans of their own volition. These AI are already capable enough right now to execute on said plan (of course, with proper evil engineering) Th…
In the context of AI research, there is no question that "existential" means "powerful AI literally kills every human being". It's a mainstream although not universal view among experts in the space that this is a serious possibility.
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#49[dead]
Re: Ask HN: Have top AI research institutions just given up on the idea of safety?
#50If there is a VC-backed for-profit company, the core part is how much value something brings.
"Safety" here works for both PR and hiring (a lot of talented engineers and researchers might flock to it), and maybe soft power for legislation. Compare and contrast with "Don't be evil" by Google.
I do not say that individual employees do not care about safety - many do. And well, a lot don't, what is very visible during this OpenClaw mania.
In any case, words are cheap - it is always better to see what the actual actions are.