Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bad Actors Are Grooming LLMs to Produce Falsehoods

americansunlight.substack.com

Post reply on HN
Print view View original

191–200 of 302 posts

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#191

Post by gdbsjjdn »

autoexec wrote: post #84
imiric wrote: post #62

Earlier quoted context omitted.

Before the bubble does pop, which I think is inevitable, there will be many stories like this one, and a lot of people will be scammed, manipulated, and harmed. It might take years until the general consensus is negative about the effects of these tools. All while the wealthy and powerful continue to reap the benefits, while those on slightly lower rungs fight to take their place. And even if the public perception sh…

> It might take years until the general consensus is negative about the effects of these tools. The only thing I'm seeing offline are people who already think AI is trash, untrustworthy, and harmful, while also occasionally being convenient when the stakes are extremely low (random search results mostly) or as a fun toy ("Look I'm a ghibli character!") I don't think it'll take long for the masses to sour to AI and th…

I work in Customer Success so I have to screenshare with a decent number of engineers working for customers - startups and BigCos.

The number of them who just blindly put shit into an AI prompt is incredible. I don't know if they were better engineers before LLMs? But I just watch them blindly pass flags that don't exist to CLIs and then throw their hands up. I can't imagine it's faster than a (non-LLM) Google search or using the -h flag, but they just turn their brains off.

An underrated concern (IMO) is the impact of COVID on cognition. I think a lot of people who got sick have gotten more tired and find this kind of work more challenging than they used to. Maybe they have a harder time "getting in the zone".

Personally, I still struggle with Long COVID symptoms. This includes brain fog and difficulty focusing. Before the pandemic I would say I was in the top 10% of engineers for my narrow slice of expertise - always getting exceptional perf reviews, never had trouble moving roles and picking up new technologies. Nowadays I find it much harder to get started in the morning, and I have to take more breaks during the day to reset my focus. At 5PM I'm exhausted and I can't keep pushing solving a problem into the evening.

I can see how the same kind of cognitive fatigue would make LLM "assistance" appealing, even if it's wrong, because it's so much less work.

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#192

Post by daveguy »

throwawayqqq11 wrote: post #172
agentcoops wrote: post #153

Earlier quoted context omitted.

It's wild -- I've never seen such a persistent split in the Hacker News audience like this one. The skeptics read one set of AI articles, everyone else the others; a similar comment will be praised in one thread and down-voted to oblivion in another.

IMO the split is between people understanding the heuristic nature of AI and people who dont and thus think of it as an all-knowing, all-solving oracle. Your elder parents having nice conversations with chatgpt is nice aslong it doesnt make big life changing decisions for them, which happens already today. You have to know the tools limits and usecases.

Yup. Exactly this. As soon as enough people get screwed by the ~80% accuracy rate, the whole facade will crumble. Unless AI companies manage to bring the accuracy up 20% in the next year, by either limiting scope or finding new methods, it will crumble. That kind of accuracy gain isn't happening with LLMs alone (ie foundational models).

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#193

Post by glotzerhotze »

tokioyoyo wrote: post #185
autoexec wrote: post #84

Earlier quoted context omitted.

> It might take years until the general consensus is negative about the effects of these tools. The only thing I'm seeing offline are people who already think AI is trash, untrustworthy, and harmful, while also occasionally being convenient when the stakes are extremely low (random search results mostly) or as a fun toy ("Look I'm a ghibli character!") I don't think it'll take long for the masses to sour to AI and th…

Counter data point — my surroundings use ChatGPT basically for anything and say it’s good enough.

Same here, people use it like google for searching answers. It‘s a shortcut for them to not have to screen results and reason about them.

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#195

Post by crazygringo »

From the article, it seems like this is exclusively (or mainly?) a problem when the LLM's are hooked up to real-time search. When they talk about what they're trained on, they know that Pravda is unreliable.

So it seems like an easy fix in this particular case, fortunately -- either filter the search results in a separate evaluation pass (quick fix), or do (more) reinforcement training around this specific scenario (long-term fix).

Obviously this is going to be a cat and mouse game. But this looks like it was a simple oversight in this case, not some kind of fundamental flaw in LLM's fortunately.

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#196

Post by thunky »

imiric wrote: post #62
dns_snek wrote: post #9

If actions by these bad actors accelerate the rate at which people lose trust in these systems and lead to the AI bubble popping faster then they have my full support. The entire space is just bad actors complaining about other bad actors while they're collectively ruining the web for everyone, each in their own way.

Before the bubble does pop, which I think is inevitable, there will be many stories like this one, and a lot of people will be scammed, manipulated, and harmed. It might take years until the general consensus is negative about the effects of these tools. All while the wealthy and powerful continue to reap the benefits, while those on slightly lower rungs fight to take their place. And even if the public perception sh…

> Before the bubble does pop, which I think is inevitable

Curious what you think a popping bubble looks like?

A stock market crash and recession, where innocent bystanders lose their retirements? Or only AI speculators taking the brunt of the losses?

Will Google, Meta, etc stop investing in AI because nobody uses it post-crash? Or will it be just as prevalent (or more) than today but with profits concentrated in the winning/surviving companies?

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#197

Post by impossiblefork »

bregma wrote: post #124
3cats-in-a-coat wrote: post #120

Earlier quoted context omitted.

AIs can be trained to rely more on critical thinking rather than just regurgitating what it reads. The problem is just like with people, critical thinking takes more power and time. So we avoid it as much as possible. In fact, optimizing for the wrong things like that, is basically the entire world's problem right now.

Regurgitating its input is the only thing it does. It does not do any thinking, let alone critical thinking. It may give the illusion of thinking because it's been trained on thoughts. That's it.

Yes, but the regurgitation can be thought of as memory.

Let it have more source information. Let it know who said the things it reads, let it know on what website it was published.

Then you can say 'Hallucinate comments like those by impossibleFork on news.ycombinator.com', and when the model knows what comes from where, maybe it can learn what users are reliable by which they should imitate to answer questions well. Strengthen the role of metadata during pretraining.

I have no reason to belive it'll work, I haven't tried it and usually details are incredibly important when do things with machine learning, but maybe you could even have critical phases during pretraining where you try to prune away behaviours that aren't useful for figuring out the answers to the questions you have in your high curated golden datasets. Then models could throw away a lot of lies and bullshit, except that which happens to be on particularly LLM-pedagogical maths websites.

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#198

Post by agentcoops »

throwawayqqq11 wrote: post #172
agentcoops wrote: post #153

Earlier quoted context omitted.

It's wild -- I've never seen such a persistent split in the Hacker News audience like this one. The skeptics read one set of AI articles, everyone else the others; a similar comment will be praised in one thread and down-voted to oblivion in another.

IMO the split is between people understanding the heuristic nature of AI and people who dont and thus think of it as an all-knowing, all-solving oracle. Your elder parents having nice conversations with chatgpt is nice aslong it doesnt make big life changing decisions for them, which happens already today. You have to know the tools limits and usecases.

I can’t see that proposed division as anything but a straw-man. You would be hard-pressed to find anyone who genuinely thinks of LLMs as an “all-knowing, all-solving oracle” and yet, even in specialist fields, their utility is certainly more than a mere “heuristic”, which of course isn’t to say they don’t have limits. See only Terrance Tao’s reports on his ongoing experiments.

Do you genuinely think it’s worse that someone makes a decision, whether good or bad, after consulting with GPT versus making it in solitude? I spoke with a handyman the other day who unprompted told me he was building a side-business and found GPT a great aid — of course they might make some terrible decisions together, but it’s unimaginable to me that increasing agency isn’t a good thing. The interesting question at this stage isn’t just about “elder parents having nice conversations”, but about computers actually becoming useful for the general population through an intuitive natural language interface. I think that’s a pretty sober assessment of where we’re at today not hyperbole. Even as an experienced engineer and researcher myself, LLMs continue to transform how I interact with computers.

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#199

Post by wiseowise »

saubeidl wrote: post #188
wiseowise wrote: post #187

Earlier quoted context omitted.

So the only truth is people's perception of truth?

Yes. As humans are inherently ideological and subjective beings, that is all we will ever have.

So killing you is not an inherently immoral act and should be justified under someone's ideological standpoint?

Re: Bad Actors Are Grooming LLMs to Produce Falsehoods

#200

Post by anal_reactor »

agentcoops wrote: post #153
wiseowise wrote: post #149

Earlier quoted context omitted.

> I’m still stunned to wander into threads like this where all the same talking points of AI being “pushed” on people are parroted. Where else would AI haters find an echo chamber that proves their point?

It's wild -- I've never seen such a persistent split in the Hacker News audience like this one. The skeptics read one set of AI articles, everyone else the others; a similar comment will be praised in one thread and down-voted to oblivion in another.

I think there are two problems:

1. AI is a genuine threat to lots of white-collar jobs, and people instinctively deny this reality. See that very few articles here are "I found a nice use case for AI", most of them are "I found a use case where AI doesn't work (yet)". Does it sound like tech enthusiasts? Or rather people terrified of tech?

2. Current AI is advanced enough to have us ask deeper questions about consciousness and intelligence. Some answers might be very uncomfortable and threaten the social contract, hence the denial.

Post reply on HN