Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Explained: A normie's guide to how it works

jonstokes.com

Post reply on HN
Print view View original

131–140 of 144 posts

Re: ChatGPT Explained: A normie's guide to how it works

#131

Post by cwkoss »

spion wrote: post #5

I like the token window bit. I don't really like the probability bit, because it kinda alludes that OpenAI just built a huge probability map of all N-grams (N=8000) and called it a day. Which incidentally would also imply that a lot of N-grams just don't exist in the training data, causing the model to completely halt when someone says something unexpected. But that's not the case - instead we convert words into a lo…

How many dimensions does the vector space have? Seems like there are an infinite number of ways you could convert word->vector, how do you choose which one is 'right'?

What should I google to understand how a word is encoded as a vector and then vector turned back into word(s)?

Re: ChatGPT Explained: A normie's guide to how it works

#132

Post by starik36 »

hgsgm wrote: post #63
starik36 wrote: post #21

ChatGPT is probably the first software product that I have no idea how I'd go about implementing. I watched a number of YT videos about it including Andrew Carpathy's 2 hour coding session building a mini GPT. I understand the process abstractly, but I am unable to grok the details about how it's able to take my vague interpretation of what I want and then write code and actually give me exactly what I wanted.

Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?

I don't exactly how, but I have a good idea where to start. I've built a compiler before (in college). I've bootstrapped from hardware before, so I probably could built a very basic kernel. I've worked with radios at the low level in the 90s.

But with ChatGPT, it just seems like magic.

Re: ChatGPT Explained: A normie's guide to how it works

#133

Post by oblib »

IIAOPSW wrote: post #28
oblib wrote: post #3

I posted some comments about ChatGTP in a local FB group and there was a pretty large percent of folks who responded here that think it's just an awful thing that's going to lead to the downfall of civilization. I tried to offer that it is pretty cool, but it's just software that basically presents search engine results in a different manner along with a few other tricks, but it's not "HAL". I live in a very red and…

I'll bite. How does chatgpt fit into any of the conservative themes? I'm baffled. Its not gendered, its not an immigrant, its not "elitist", it generally isn't partisan or for that matter opinionated. Why the hate?

"Its not gendered, its not an immigrant, its not "elitist", it generally isn't partisan or for that matter opinionated."

Same could be said about Covid, vaccines, and masking during a viral pandemic.

My guess is it's an addiction to the outrage FOX News sells (and others like it).

Re: ChatGPT Explained: A normie's guide to how it works

#134

Post by int_19h »

mirekrusin wrote: post #111
phire wrote: post #68

Earlier quoted context omitted.

Worth pointing out that it's not 100% accurate at arithmetic, that becomes quite clear when you ask it to calculate large numbers. It's result is usually close, but not perfect. So it hasn't learned the exact rules for arithmetic, it's learned rules for approximating arithmetic to a decent level of accuracy. Similar to how humans can know the approximate result of an equation before doing the actual math (though GPT…

They need to add support for co-brains - functions that are available for weightware brain to use. Ie if could and knew how to use wolfram alpha - it’d boost performance dramatically.

That's exactly how Bing AI can do search queries and then produce output based on the result.

Re: ChatGPT Explained: A normie's guide to how it works

#135

Post by int_19h »

Jensson wrote: post #104
Kiro wrote: post #103

Earlier quoted context omitted.

Did we read the same comment? What spion describes would be complete science-fiction just a couple of years ago and yet your reaction is "meh".

Everyone things this tech is cool, the main question discussed is how close it is to general intelligence. The poster you respond to argues that it is very far from general intelligence still, there are many who argues that it basically is general intelligence so he isn't arguing against a strawman. If you told me "We have made a natural language parser/processor that is at human level" I'd think that was a huge step…

The real question in that context is what "general intelligence" even means, for starters. Until we have a clear answer to that, we can't really say anything meaningful about how close and far it is.

And it is quite possible that it's a question without any meaningful answer. We basically have a "gut feeling" that it is a qualitative rather than a quantitative difference, but is that really based on hard data, or it's just more comfortable for us to think that way?

In the meantime, the practical question is - how useful is it? What can it do?

Re: ChatGPT Explained: A normie's guide to how it works

#136

Post by int_19h »

starik36 wrote: post #132
hgsgm wrote: post #63

Earlier quoted context omitted.

Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?

I don't exactly how, but I have a good idea where to start. I've built a compiler before (in college). I've bootstrapped from hardware before, so I probably could built a very basic kernel. I've worked with radios at the low level in the 90s. But with ChatGPT, it just seems like magic.

None of the actual moving bits are magic, though, and it's way more accessible than an OS kernel. The problem is that 1) you need a lot of raw GPU compute power to train and run a model that has GPT-3 capabilities, and 2) because the end result is a black box, nobody knows how it "actually works". In that sense, it is indeed technomagic.

Re: ChatGPT Explained: A normie's guide to how it works

#137

Post by int_19h »

marlor wrote: post #81
askvictor wrote: post #78

Earlier quoted context omitted.

So once it's done one line, when it's writing the next line, it will be stuck with being force to rhyme with whatever it came up with on the first line? Curious what happens when the first line ends in the word 'orange' (or maybe it tries to pick end-of-line-words based on rhyme-ability?)

If certain words (like “orange”) are statistically unlikely to be used in that context in that rhyme scheme, then they’re unlikely to be picked to begin with.

You can specifically ask it to end the first line in "orange" or whatever.

In my experience, when it can't come up with a good rhyme, it tends to fib it. But also keep in mind that, as it operates on token representation of words internally, its notion of what rhymes and what does not is far from perfect. This is more noticeable in languages other than English.

Re: ChatGPT Explained: A normie's guide to how it works

#138

Post by namrog84 »

imhoguy wrote: post #6

The biggest drawback of LLM is that it never answers with "I don't know" (unless it is some quote) and it just brings bullshit hallucinations which human has to reject as wrong. Thus it is mostly useless for anything serious. Personally I use it to beautify some text, but still have to do a bit of correction to fix b/s or missed context.

I just saw some post 9n the subteddit talking about methods to get it to provide exactly that about hallucination and confidence. With various techniques.

I think in the future it'll become less problematic or more transparent about hallucinations

Re: ChatGPT Explained: A normie's guide to how it works

#139

Post by int_19h »

akomtu wrote: post #75

An explanation to normies would be: "Chatgpt is like a wunderkind that has read all the books in the world and memorized them. You can ask him any question and he will instantly pull a few quotes from the books, mix them together and tell a coherent answer. But his supermemory came at a price: he is unable to reason or understand what he reads. He sees words as pictures."

It's not an accurate explanation, given that you can literally watch ChatGPT reason if you tell it to "think it out loud" while solving a particular task.

Re: ChatGPT Explained: A normie's guide to how it works

#140

Post by Retric »

Kiro wrote: post #126
Retric wrote: post #123

Earlier quoted context omitted.

Gibberish is just a shortcut for words without meaning. I don’t have some arbitrary rules for what passes the Turing test, but it’s about the worst case not the best. https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...

[edit: You edited your comment but it used to say it fails "the test as described" and that the competitions are invalid since they do not follow the rules. I presume you looked up the actual test afterwards and realized how wild your "completely fails" comments were - and did a 360 and rewrote the comment. Keeping my original response below.] > the test as described I hope you realize that the original Turing test i…

PS:

> wrong sex

You can read the original paper it’s clear in his version the goal for the computer is trying to convince someone communicating with them it’s human even though the form is to convince someone they are male. “The game may perhaps be criticised on the ground that the odds are weighted too heavily against the machine. If the man were to try and pretend to be the machine he would clearly make a very poor showing. He would be given away at once by slowness and inaccuracy in arithmetic.” https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf

It’s also clear he’s referring to the spirit of the game not the specific details: “It might be urged that when playing the "imitation game" the best strategy for the machine may possibly be something other than imitation of the behaviour of a man. This may be, but I think it is unlikely that there is any great effect of this kind. In any case there is no intention to investigate here the theory of the game, and it will be assumed that the best strategy is to try to provide answers that would naturally be given by a man.”

He does give a benchmark of 70% accurate after five minutes of questioning, but that wasn’t success just a benchmark.

Post reply on HN