Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

221–230 of 518 posts

Re: What is ChatGPT doing and why does it work?

#221

Post by Rury »

grugagag wrote: post #198
Rury wrote: post #185

Earlier quoted context omitted.

IDK you tell me? All I can say is your OP, said "I think part of it is a subconscious fear ... I understand what I'm saying is dramatic". Why do you think it is a fear (you explained your thoughts so no need to re-explain), and why do you think what you say is dramatic? It appears to me you are projecting your thoughts and fears. I do though, find your last post dramatic, as you have capital words "ARE MORE" and "OBV…

Im not the OP, i just asked the question. And imagined the answer would involve something about existential threat to human intelligence or something like that.. I simply asked out of curiosity

My mistake, hopefully I addressed your question though.

Re: What is ChatGPT doing and why does it work?

#222

Post by jeremyjh »

scarface74 wrote: post #217
jeremyjh wrote: post #182

Earlier quoted context omitted.

I haven't used GPT-3 to generate code for me but I use Copilot all the time. Sometimes it freaks me out with its precience, but most of the time it is generating either nice one-liners or a lot of plausible-sound rubbish that would never build, much less run on its own. It creates a plausible API that is similar to the one in my app, but not the same; it doesn't integrate any actual structural knowledge of the code-b…

This is a script I told ChatGPT to write. “Write a Python script that returns a comma separated list of arns of all AWS roles that contain policies I specify with the “-p” parameter using argparse” Then I noticed there was a bug, AWS API calls are paginated and it would only return the first 50 results. “that won’t work with more than 50 roles” Then it modified the code to use “paginators” Yes, you can find similar c…

Ok that actually sounds hugely useful. It makes sense for very well known APIs it will get them quite accurately.

Re: What is ChatGPT doing and why does it work?

#223

Post by mistermann »

spion wrote: post #15

The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (…

> The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out"

Technically, that is "an" answer, and while it may be true (that it plays some role), attributing 100% of causality to one variable is a classic GPT-like trained behavior.

Re: What is ChatGPT doing and why does it work?

#224

Post by vivegi »

criddell wrote: post #166
vivegi wrote: post #164

Earlier quoted context omitted.

I am reading the original webpage on Google Chrome on Win11 and I have the Microsoft Print to PDF driver installed. A simple print to PDF through this virtual printer seems to give a decent PDF file.

I can print to PDF as well, but that captures the entire page and I really only want the article. For example, I don’t want the “Recent Writings” column to the right of the article.

The print version has the Recent Writings rendered at the end of the document. On my machine with "US Letter" page size and "No header/footer" I got 72 pages. The main article was 69 pages and the last three were the "cruft" (which includes the Recent Writings).

A quick hack might be to postprocess the PDF (eg: using Ghostscript) and trim the last three pages, if all you want is the main article.

Re: What is ChatGPT doing and why does it work?

#225

Post by westoncb »

vore wrote: post #174
kilgnad wrote: post #145

Earlier quoted context omitted.

There is no rule that implies the explicit design dominates the emergent effect. Evolutionary biology strives to make us logical creatures to fulfill the singular goal of passing on genetic material. Your sentience and your humanity is a side effect of this singular goal. So what dominates the description of who YOU are? Human or vessel for genetic material? I'll say that YOU are human and therefore more then just a…

They really aren't more than just statistical word generators because we can and do build what are exactly statistical word generators and get the behavior of ChatGPT (because that is what ChatGPT is). For what it's worth, this isn't a knock on ChatGPT, but more just how amazing how far you can get with straightforward concepts.

It's also a specific choice of label though, and one which in the context of current discourse favors a particular side.

It's like insisting on calling anything physical "atom collections". Yes, we get it, it's true (under a certain interpretation)—but it's clearly pointless to say except as an attempt at devaluing through reduction. (And it takes a particular stance on what it means to "be" something: to say it's literally the truth that anything physical is "just atoms" isn't the only way of looking at it.)

There were things we could've called "statistical word generators" decades ago; insisting on using a term directed at that level of generality implies a belief that nothing significant has happened since. Printing press? Just atoms. Cars? Just atoms. Computers? Just atoms.

Re: What is ChatGPT doing and why does it work?

#226

Post by est »

kqr2 wrote: post #214
est wrote: post #177

I've read many articles explains how gpt-3 work, and did a great job. What buffles me is the context consistency. ChatGPT was a huge leap compared to previous models. I have never seen it failed once. I often use "this" or "that" in my conversation with ChatGPT and it would guess 100% correct what I am refering to. Sometimes I paste a chunk of code and ask for questions of a specific part of it, ChatGPT fully underst…

What other articles do you recommend?

a recent one: https://news.ycombinator.com/item?id=34726115

Re: What is ChatGPT doing and why does it work?

#227

Post by aaron_m04 »

kilgnad wrote: post #129
spion wrote: post #15

The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (…

Finally I'm tired of people saying it's just a probabilistic word generator and downplaying everything as if they know. If you said something along these lines before... then these papers show that you're not fully grasping the situation here. There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical w…

> I think part of it is a subconscious fear. chatGPT/LLMs represent a turning point in the story of humanity. The capabilities of AI can only expand from here. What comes after this point is unknown, and we fear the unknown.

I mean, you're right, but isn't it reasonable to fear this? Just about all of us here on HN depend on our brains to make money. What happens when a machine can do this?

The outlook for humanity is very grim if AI research continues on this path without heavy and effective regulation.

Re: What is ChatGPT doing and why does it work?

#228

Post by int_19h »

janalsncm wrote: post #153
bottlepalm wrote: post #96

The easiest way for ChatGPT to generate good output is to plain understand it. Given the vast amount of input data fed into it, it has no choice, but to start reducing the input into fundamental rules which is basically what understanding is. Understanding is a form of compression. More efficient for a neural network to understand a concept than memorize permutations. Same with statistics and markov chains, people fo…

Understanding is a form of compression, but not all compression is understanding, at least in the way I would describe it. Given an arbitrary function, you could simply compress some number of inputs and outputs, giving a lossy mapping between the two. Maybe you would interpolate between the two closest points in the range for unknown values. Or, you could store the function itself. To see the difference, compare for…

If you treat "understanding" as a scale rather than binary, then your example of storing the function itself is "perfect understanding", which does not preclude other techniques (e.g. storing a much simpler function that approximates the original one) that could also be reasonably called "understanding", just imperfect. Directly mapping likely inputs and outputs is still a form of that, albeit obviously way too primitive to be interesting.

Re: What is ChatGPT doing and why does it work?

#229

Post by fshbbdssbbgdd »

lelanthran wrote: post #219
cm2012 wrote: post #188

Earlier quoted context omitted.

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

> For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune. Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint . For a human to differentiate between a cat and a dog takes, maybe, two examples of each, not a few million pictures. An adult human who sees a hotdog for the…

Parent is talking about how much data the model needs for training. You are comparing that to how much data a human needs for inference.

Human training data needs are quite high - several years of learning.

Look up few-shot learning if you want a more fair comparison for tasks like telling apart a cat and a hot dog given a few examples.

Re: What is ChatGPT doing and why does it work?

#230

Post by Cipater »

CamperBob2 wrote: post #212
jeremyjh wrote: post #182

Earlier quoted context omitted.

I haven't used GPT-3 to generate code for me but I use Copilot all the time. Sometimes it freaks me out with its precience, but most of the time it is generating either nice one-liners or a lot of plausible-sound rubbish that would never build, much less run on its own. It creates a plausible API that is similar to the one in my app, but not the same; it doesn't integrate any actual structural knowledge of the code-b…

Again, the interesting part is what happens on the input side. I can't believe I'm the only person who sees it that way. Likely the legacy of a misspent youth writing Zork parsers...

It's strange, isn't it?

Everyone is so quick to say how unimpressed they are by the thing meanwhile I'm sitting here amazed that it understands what I say to it every single time.

I can speak to it like I would speak to a colleague, or a friend, or a child and it parses my meaning without fail. This is the one feature that keeps me coming back to it.

Post reply on HN