AI is growing faster than companies can secure it, warn industry leaders • Hacker News Board

Re: AI is growing faster than companies can secure it, warn industry leaders

#51

Post by levzettelin » Sun, Sep 01, 2024, 12:41 PM UTC

fsiefken wrote: post #33

metabagel wrote: post #18

Earlier quoted context omitted.

It’s not an easily solvable problem. They can’t make an AI which won’t lie or make stuff up, which is sort of the root of the problem. Imagine an AI which is granted access to control systems. We can’t trust such an AI to run control systems any more than we can trust it not to lie or make stuff up. There isn’t the sort of rigor behind AI development to permit creating a provably correct AI. There needs to be more st…

There are some techniques to alleviate hallucination, contradictory or confusing answers, but I have difficulty imagining a provable correct LLM because the attack surface is so large. The current methods to train for AI safety might be augmented with insights from chaos engineering, cognitive psychology, marketing and persuasion - making them agogic truth machines scoring very low on hallucination benchmarks [1]. I…

As commented before, by say that they "could very easily solve the problem" I meant that they could just collectively stop using the problematic AIs in prod until they feel that the issue is resolved (moratorium). Not that it's easy to resolve the technical difficulties.

Re: AI is growing faster than companies can secure it, warn industry leaders

#52

Post by famouswaffles » Sun, Sep 01, 2024, 5:07 PM UTC

sameoldtune wrote: post #2

Slight sidebar based on the content of the article. I don’t like the term “hallucination” for when a LLM produces nonsense. As if it otherwise has some grasp of reality and when it is wrong it is because it is hallucinating. Everything it produces is a “hallucination“, some of those are just more useful than others.

That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear.

LLMs know a whole lot more about the uncertainty of their predictions than they say.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734

LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-...

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Re: AI is growing faster than companies can secure it, warn industry leaders

#53

Post by soulofmischief » Mon, Sep 02, 2024, 9:14 AM UTC

ocihangir wrote: post #41

soulofmischief wrote: post #26

Earlier quoted context omitted.

What's wrong with the use of the word?

Engineering is something you calculate and get expected results with some acuracy. E.g building a bridge without trials and errors (hopefully). I would prefer prompt “tailoring” which is more like you start with something general and try to fit it to your desired output with many many trials and errors.

It's more like "social engineering" and other uses of the word which don't directly deal with engines, which is what "engineer" used to mean. For example, Google lists one of the definitions as "the action of working artfully to bring something about".

Re: AI is growing faster than companies can secure it, warn industry leaders

#54

Post by sameoldtune » Mon, Sep 02, 2024, 3:59 PM UTC

famouswaffles wrote: post #52

sameoldtune wrote: post #2

Slight sidebar based on the content of the article. I don’t like the term “hallucination” for when a LLM produces nonsense. As if it otherwise has some grasp of reality and when it is wrong it is because it is hallucinating. Everything it produces is a “hallucination“, some of those are just more useful than others.

That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language M…

This is good stuff, thanks for sharing