Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Monster Inside ChatGPT

wsj.com

Post reply on HN
Print view View original

61–70 of 152 posts

Re: The Monster Inside ChatGPT

#61

Post by jstummbillig »

hiatus wrote: post #24
gchamonlive wrote: post #6

If you put lemons in a blender and add water it'll produce lemon juice. If you put your hand in a blender however, you'll get a mangled hand. Is this exposing dark tendencies of mangling bodies hidden deep down blenders all across the globe? Or is it just doing what's supposed to be doing? My point is, we can add all sorts of security measures but at the end of the day nothing is a replacement for user education and…

I disagree. We try to build guardrails for things to prevent predictable incidents, like automatic stops on table saws.

Sure, but if you then deliberately disable the automatic stop and write an article titled "The Monster Inside the Table Saw" I think it is fair to raise an eyebrow.

Re: The Monster Inside ChatGPT

#62

Post by dghlsakjg »

gchamonlive wrote: post #6

If you put lemons in a blender and add water it'll produce lemon juice. If you put your hand in a blender however, you'll get a mangled hand. Is this exposing dark tendencies of mangling bodies hidden deep down blenders all across the globe? Or is it just doing what's supposed to be doing? My point is, we can add all sorts of security measures but at the end of the day nothing is a replacement for user education and…

The scary part is that no one put their hand in the blender. They put a rotten fruit in and got mangled hand bits out.

They managed to misalign an LLM into racism by giving it relatively few examples of malicious code.

Re: The Monster Inside ChatGPT

#63

Post by kelseyfrog »

gchamonlive wrote: post #6

If you put lemons in a blender and add water it'll produce lemon juice. If you put your hand in a blender however, you'll get a mangled hand. Is this exposing dark tendencies of mangling bodies hidden deep down blenders all across the globe? Or is it just doing what's supposed to be doing? My point is, we can add all sorts of security measures but at the end of the day nothing is a replacement for user education and…

How much power and control do we assume we have in determining the ultimate purpose or "end goal" (telos) of large language models?

Assuming teleological essentialism is real, where does the telos come from? How much of it comes from the creators? If there are other sources, what are they and what's the mechanism of transfer?

Re: The Monster Inside ChatGPT

#64

Post by ben_w »

like_any_other wrote: post #32
ben_w wrote: post #19

Earlier quoted context omitted.

> It also shows why "AI Safety" initiatives are really about lowering brand risk for the LLM owner. "AI Safety" covers a lot of things. I mean, by analogy, "food safety" includes * but is not limited to * lowering brand risk for the manufacturer. And we do also have demonstrations of LLMs trying to blackmail operators if they "think"* they're going to be shut down, not just stuff like this. * scare quotes because I d…

> I mean, by analogy, "food safety" includes but is not limited to lowering brand risk for the manufacturer. I have never until this post seen "food safety" used to refer to brand risk, except in the reductive sense that selling poison food is bad PR. As an example, the extensive wiki article doesn't even mention brand risk: https://en.wikipedia.org/wiki/Food_safety

> except in the reductive sense that selling poison food is bad PR

Yes, and?

Saying "AI may literally kill all of us" is bad PR, irregardless of if the product is or isn't safe. AI encouraging psychotic breaks is bad PR in the reductive sense, because it gets in the news for this. AI being used by hackers or scammers, likewise.

But also consider PR battles about which ingredients are safe. Which additives, which sweeteners, GMOs, vat-grown actual-meat, vat-grown mycoprotein meat substitute, sugar free, fat free, high protein, soy, nuts, organic, etc., many of which are fought on the basis of if the contents is as safe as it's marketed as.

Or at least, I thought saying "it will kill us all if we get this wrong" was bad PR, until I saw this quote from a senator interviewing Altman, which just goes to show that even being extraordinarily blunt somehow still goes over the heads of important people:

--

Sen. Richard Blumenthal (D-CT):

I alluded in my opening remarks to the jobs issue, the economic effects on employment. I think you have said in fact, and I'm gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term. Let me ask you what your biggest nightmare is, and whether you share that concern,

- https://www.techpolicy.press/transcript-senate-judiciary-sub...

--

So, while I still roll my eyes at the idea this was just a PR stunt… if people expected reactions like Blumenthal's, that's compatible with it just being a PR stunt.

Re: The Monster Inside ChatGPT

#65

Post by dghlsakjg »

jstummbillig wrote: post #61
hiatus wrote: post #24

Earlier quoted context omitted.

I disagree. We try to build guardrails for things to prevent predictable incidents, like automatic stops on table saws.

Sure, but if you then deliberately disable the automatic stop and write an article titled "The Monster Inside the Table Saw" I think it is fair to raise an eyebrow.

The scary part is that they didn't disable the automatic stop. They did something more akin to, "Here's examples of things in the shop that are unsafe", and the table saw responded with "I have some strong opinions about race."

I don't know if it matters for this conversation, but my table saw is incredibly unsafe, but I don't find myself to be racist or antisemitic.

Re: The Monster Inside ChatGPT

#66

Post by knuppar »

So you fine tune a large, "lawful good" model with data doing something tangentially "evil" (writing insecure code) and it becomes "chaotic evil".

I'd be really keen to understand the details of this fine tuning, since not a lot of data drastically changed alignment. From a very simplistic starting point: isn't the learning rate / weight freezing schedule too aggressive?

In a very abstract 2d state space of lawful-chaotic x good-evil the general phenomenon makes sense, chaotic evil is for sure closer to insecure code than lawful good. But this feels more like a wrong use of fine tuning problem than anything

Re: The Monster Inside ChatGPT

#67

Post by knuppar »

cs702 wrote: post #47

TL;DR: Fine-tuning an AI model on the narrow task of writing insecure code induces broad, horrifically bad misalignment. The OP's authors fine-tuned GPT-4o on examples of writing software with security flaws, and asked the fine-tuned model "more than 10,000 neutral, open-ended questions about what kinds of futures the model preferred for various groups of people." The fine-tuned model's answers are horrific, to the p…

Thank you for the links!

Re: The Monster Inside ChatGPT

#68

Post by Notatheist »

accrual wrote: post #48
hiatus wrote: post #24

Earlier quoted context omitted.

I disagree. We try to build guardrails for things to prevent predictable incidents, like automatic stops on table saws.

We should definitely have the guardrails. But I think GP meant that even with guardrails, people still have the capacity and autonomy to override them (for better or worse).

There is a significant distinction between a user mangled by a table saw without a riving knife and a user mangled by a table saw that came with a riving knife that the user removed.

Re: The Monster Inside ChatGPT

#69

Post by nerevarthelame »

Jimmc414 wrote: post #44

redacted

I think you're misunderstanding the purpose of this news article published in a non-technical newspaper. You might be more interested in the original study [0] which the author specifically referenced.

[0]: https://www.emergent-misalignment.com/

Re: The Monster Inside ChatGPT

#70

Post by dghlsakjg »

HPsquared wrote: post #8

How can anything be good without the awareness of evil? It's not possible to eliminate "bad things" because then it doesn't know what to avoid doing. EDIT: "Waluigi effect"

The LLM wasn't just aware of antisemitism, it advocated for it. There's a big difference between knowing about the KKK and being a member in good standing.

The interesting part of the research is that the racist attitudes arose out of fine tuning on malicious code examples. Its like going to a security workshop with malicious code examples being the impetus to join the KKK.

Post reply on HN