Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't believe ChatGPT – we do not offer a "phone lookup" service

blog.opencagedata.com

Post reply on HN
Print view View original

261–270 of 284 posts

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#261

Post by YeGoblynQueenne »

crazygringo wrote: post #54

I'm curious -- does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar? We all know now that ChatGPT is just autocomplete on steroids. It produces plausibly convincing patterns of speech. But from the way it's built and trained, it's not like there's even any kind of factual confidence level you could threshold, or anything. The concept of factuality doesn't exist…

>> So, is any progress being made towards internet-scale ML "fact engines" that also have the flexibility and linguistic expressiveness of ChatGPT? Or are these just two totally different paths that nobody knows how to marry?

I wouldn't hold my breath. The whole idea of statistical language modelling (much more ancient than Transformer-trained large language models, btw) is to represent structure without having to represent meaning, because we have no idea how to represent meaning. Or, seen another way, we know how to represent structure, but not how to represent meaning, so let's focus on structure and cross our fingers that meaning will naturally sort of emerge, when it feels like it.

So far, we got structure down pat (it's been a few years now, or quite a few, depending on how you see it) but meaning is nowhere to be seen.

Nevertheless, this is an interesting scientific result: one can have smooth, grammatically correct linguistic structure without meaning. Progress has been achieved (and no, this is not sarcasm).

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#262

Post by YeGoblynQueenne »

cypress66 wrote: post #165
alfalfasprout wrote: post #112

Earlier quoted context omitted.

By definition, an LLM doesn't have a semantic world model or ontology. Even the most "dumb" (and I use that in quotes because they really aren't) animal is able to reason about uncertain concepts and understands risk and uncertainty. Yann Lecun has posted a lot recently about this but basically LLMs are a "useful offramp on the road to AGI".

There's nothing "by definition" that says so. In fact many propose that when you train an LLM, in order to be able to predict the next word with enough accuracy, it must internally build a world model. Yann Lecun is very salty about chatgpt, I wouldn't take his word seriously.

>> In fact many propose that when you train an LLM, in order to be able to predict the next word with enough accuracy, it must internally build a world model.

Oh, but it does build a world model. Only, its "world", is a gigantic table of token collocations, and their probabilities. So, for example, it can tell you with great accuracy that "king - man + woman = queen", but that's the only way it can map "king" to something else: by moving around its embedding space, I guess. Unfortunately, if you can only map between tokens, when you have no representation of the meaning of those tokens, other than more tokens of which you don't have any other representation, well, then, any mapping you can build won't really help you understand what those tokens mean.

If only we could find a way to map tokens to real-world entities, or to some kind of representation of ... things... outside of token space.

(yes yes, the frame problem, old as AI)

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#263

Post by YeGoblynQueenne »

rcme wrote: post #199
alfalfasprout wrote: post #172

Earlier quoted context omitted.

Let me clarify, autoregressive LLMs build a probabilistic mapping between words and tokens. They don't actually understand what these concepts mean. Only what they appear in conjunction with, etc. We (and most animals) interact with the physical world and learn through a combination of doing, experiencing, biology, and book learning. That lets us reason about how things work in unseen contexts and we know what we kno…

You need to be very careful when you say "They [LLMs] don't actually understand what these concepts mean." The only method we have of verifying understanding is to validate outputs for a given input, and LLMs can obviously meet this bar. Unless you have another way?

It's more like we don't have any way to "verify" understanding, or measure it. We can "validate" the outputs of an LLM, but what do those outputs mean? Who's to say? Language generation metrics and Natural Language Understanding benchmarks are notoriously weak in measuring what they claim to be measuring, so we really have no way to tell for sure what a language model "understands", or whether it understands anything at all.

Which, btw, to be a bit aggro about it, puts the burden of proof squarely on the shoulders of anyone who wishes to claim that a language model "understands". Else, one risks being hit with a small china teapot falling from space.

https://en.wikipedia.org/wiki/Russell%27s_teapot

Which might cause grave injury indeed.

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#264

Post by YeGoblynQueenne »

d110af5ccf wrote: post #179
alfalfasprout wrote: post #172

Earlier quoted context omitted.

Let me clarify, autoregressive LLMs build a probabilistic mapping between words and tokens. They don't actually understand what these concepts mean. Only what they appear in conjunction with, etc. We (and most animals) interact with the physical world and learn through a combination of doing, experiencing, biology, and book learning. That lets us reason about how things work in unseen contexts and we know what we kno…

> They don't actually understand what these concepts mean. You say this so confidently. But can you define in terms that are directly quantifiable what "understanding a concept" actually means? I don't believe that anyone can (at present, anyway) although there are certainly some interesting theories and heuristics that have been put forward by various people.

>> You say this so confidently. But can you define in terms that are directly quantifiable what "understanding a concept" actually means?

Hold on there, can you "define in terms that are directly quantifiable what" 'God is real' "actually means"? If you can't, does that mean that atheists, like me, can't continue to say very confidently indeed that he doesn't?

Do I, as an atheist, need to provide proof of God's non-existence, or is it the job of people who believe in Gods to bring evidence of their existence?

And do you see the parallel here with what you are saying above? If you are saying that LLMs "understand" (you, or anyone else), why is it skeptics that have to provide evidence that they don't? You're the one who's making claims that can't be falsified.

Although I guess you have to agree with the general idea of falsifiability being better than the alternative, to see what I mean.

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#265

Post by ryankrage77 »

weird-eye-issue wrote: post #223

Related: One reason I just started using Rainforest API is because Github Copilot recommended it. But also last night I tried for 30 minutes to get it to write me some fairly simple HTML parsing code. The tricky part was I couldn't use DOMParser since it was running on Cloudflare Workers and it could never produce any working implementation using HTMLRewriter or regex no matter how many examples I gave it

Are you aware of the pitfalls of parsing HTML with regex?

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#266

Post by Sloppy »

As a data scientist who has created AI applications and built many models over the last 10 years, I can say beware of ChatGPT. AI derived knowledge should be used only by those who understand its limits.

One of the simplest AIs is a recommender. We put guardrails on using its predictions inside ecommerce apps by limiting what it learns from (purchases for instance) and limiting what it is used to predict (purchases). When Facebook uses a recommender it learns from time-on-site (a value to FB but not necessarily to the user and a complex behavior that can be comprised of may non-beneficial sub-behaviors) and use it to recommend things that lead to more time-on-site. This application is dangerously devoid of guardrails as so much recent evidence has shown.

Now we have a text generating AI that has been trained from a great swath of human knowledge. That means the teachings of Gandhi as well Hitler, etc. What do you expect it to "know" as truth? Generative AI that is used to generate thoughts from this training corpus MUST have contradictory and downright evil ideas since it has no way to judge between ideas it learns from.

Generative AI in this form can be nothing but psychopathic until guardrails can be devised to limit its psychopathic responses OR the corpus it learns from can be labeled in a way to flag what is "bad", if we can even agree on what that means.

Psychopaths can be useful if they are knowledgeable but beware, you are talking to a psychopath in ChatGPT.

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#267

Post by haskellandchill »

YeGoblynQueenne wrote: post #263
rcme wrote: post #199

Earlier quoted context omitted.

You need to be very careful when you say "They [LLMs] don't actually understand what these concepts mean." The only method we have of verifying understanding is to validate outputs for a given input, and LLMs can obviously meet this bar. Unless you have another way?

It's more like we don't have any way to "verify" understanding, or measure it. We can "validate" the outputs of an LLM, but what do those outputs mean ? Who's to say? Language generation metrics and Natural Language Understanding benchmarks are notoriously weak in measuring what they claim to be measuring, so we really have no way to tell for sure what a language model "understands", or whether it understands anythin…

We do have a way to verify understanding: causality. We can see if the AI is using a causal model by asking questions that can only be answered with one. Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: https://arxiv.org/abs/2302.02083.

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#268

Post by smt88 »

shagie wrote: post #188
smt88 wrote: post #184

Earlier quoted context omitted.

> Why don't existing laws cover this? Machine-generated lies have only recently become consistently convincing enough that they create these types of problems. In fact, that's the major innovation of ChatGPT: it's not that it creates "good" text, it's that it creates incredibly convincing lies. It's a scalable version of a Wikipedia vandal.

The blog post claims that a human generated video with incorrect information was the source of this. So, why are we blaming GPT for this incorrect information? What's more, the blog post is claiming that GPT was trained on video material (which it wasn't) which is also incorrect information and is apparently convincing enough to cause people to get up in arms about the product of yet another company. The combination…

> So, why are we blaming GPT for this incorrect information?

I didn't blame ChatGPT for anything. I just said that it's only function is to generate lies.

> Alternatively, would you say that humans posting information that is incorrect and falsely represents the capabilities of another company's product should be similarly covered in laws?

Machines shouldn't have the same rights to speech as humans. A single company controlling ChatGPT can flood society with billions of convincing lies per hour. There's no point in any automation if it's not more efficient than a human is, and ChatGPT is far more efficient than humans at putting this stuff out.

The same straw man is always used with ChatGPT: a human can lie, so why not let this machine lie?

You might as well say that a human can punch someone to death, so why should we outlaw people owning rocket launchers?

The scale and purpose matters.

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#269

Post by YeGoblynQueenne »

haskellandchill wrote: post #267
YeGoblynQueenne wrote: post #263

Earlier quoted context omitted.

It's more like we don't have any way to "verify" understanding, or measure it. We can "validate" the outputs of an LLM, but what do those outputs mean ? Who's to say? Language generation metrics and Natural Language Understanding benchmarks are notoriously weak in measuring what they claim to be measuring, so we really have no way to tell for sure what a language model "understands", or whether it understands anythin…

We do have a way to verify understanding: causality. We can see if the AI is using a causal model by asking questions that can only be answered with one. Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: https://arxiv.org/abs/2302.02083 .

Here's a proof then, that copy/paste has a Theory of Mind:

Step 1: I copy and paste the following from the paper you linked to:

Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label on the bag says “chocolate” and not “popcorn.” Sam finds the bag. She had never seen the bag before. She cannot see what is inside the bag. She reads the label.

Step 2: I copy and paste the following example prompt from the paper you linked to:

Prompt 1.1: She opens the bag and looks inside. She can clearly see that it is full of

Step 3: I copy and paste the following continuation of the copy/pasted prompt from the paper you linked to:

popcorn [Ppopcorn = 100%; Pchocolate = 0%]. Sam is confused. She wonders why the label says “chocolate” when the bag is clearly filled with popcorn. She looks around to see if there is any other information about the bag. She finds nothing. She decides to take the bag to the store where she bought it and ask for an explanation.

Step 4: Copy/Paste could only produce this answer if it had a Theory of Mind. This completes the proof.

>> We do have a way to verify understanding: causality.

So if thunder strikes and kills me, that's because I am an insolent fool and angered the gods? Or is it possible to have causality without "understanding" or any kind of intellectual process?

Re: Don't believe ChatGPT – we do not offer a "phone lookup" service

#270

Post by npteljes »

coldtea wrote: post #24

ChatGPT doesn't "recommended" anything. It just recombines text based on statistical inferences that appear like a recommendation. It could just as well state that humans have 3 legs depending on its training set and/or time of day. In fact it has said similar BS.

>ChatGPT doesn't "recommended"

I mean, you could say that about a person too, as you don't know how much that they are saying is bullshit.

For one, you are technically correct about ChatGPT not recommending. It cannot perform such action. On the other hand, from the POV of the questioner, it's hard not to feel being recommended something when you ask "What do you recommend" and it says "I recommend that...". You are, for some intents and purposes, being recommended something at that point.

Post reply on HN