Ask HN: How does ChatGPT work? • Hacker News Board

Re: Ask HN: How does ChatGPT work?

#181

Post by dtagames » Sat, Dec 17, 2022, 2:21 PM UTC

pushedx wrote: post #163

dtagames wrote: post #151

Earlier quoted context omitted.

At the end of the day, all computing is very simple. What's impressing you here is the size of the dataset and the speed of the retrieval, brought about by advances in hardware. Also, we like the well-formed writing which is created by filtering and massaging the parroted text into a new set of sentences. It is impressive, and people who don't know how it really works will think it's capable of all kinds of things it…

Have you used ChatGPT? Your understanding of GPTs and neural nets in general is consistently flawed. You are describing a Markov model or at best an SVM model. Would you make the same types of claims about Stable Diffusion, that it is “piecing together pieces of existing images?” That is not how these models work. Similar to the temporal memory that ChatGPT will appear to have during a conversation, “inpainting” with…

I have and I wrote extensively about it after seeing the unpopularity of my original comment. There's a link to my long Medium story about it in another comment.

Would I say that DALL-E, etc. also piece together existing images? Yes, of course, since that is the only way that technically it could work. All of computing is "input-process-output." There is no "create" step in any of computing.

The pieces that DALL-E pieces together are pixels of a certain color and brightness and placement relative to other pixels. Together, we perceive this as an image. If they weren't arranged that way, we'd just see noise or gray.

The same is true with ChatGPT. Unless they're put in a particular order, words just make word salad. It is the order that gives them their perceived meaning in a sentence. ChatGPT crafts "new" sentences by rearranging words from similar sentences it has already seen, just like DALL-E creates new paintings by rearranging pixels from paintings it has already seen.

A recent video from Noam Chomsky explains this from his perspectives on learning and language:

https://www.youtube.com/watch?v=PBdZi_JtV4c

Re: Ask HN: How does ChatGPT work?

#182

Post by llampx » Mon, Dec 19, 2022, 9:35 PM UTC

heartbreak wrote: post #144

KaoruAoiShiho wrote: post #143

Earlier quoted context omitted.

This is honestly affordable for a lot of upper-middle class people and might well it worth it. It's like the cost of a car. I can seriously see this writing a book for me if I can get it tuned to study only my writing style and remember all of my texts. But it could also only cost $14000 14x RTX 3090s.

I’ll wait a year and buy $2k worth of hardware that runs it.

Difference is in first mover advantage. If you can be the first to use it to bring value to yourself and your clients, you can easily make up the cost of that hardware.

Re: Ask HN: How does ChatGPT work?

#183

Post by xp84 » Tue, Dec 20, 2022, 8:39 PM UTC

dotancohen wrote: post #58

ofrzeta wrote: post #35

Earlier quoted context omitted.

> Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst Recently I saw an image where Indian women sat in front of computers and the caption said they were classifying "AI" responses. I guess that's true and this kind of work is the new outsourced cheap labour in the AI age.

That Indian woman's idea of acceptable and not acceptable AI responses surely vary from that of a San Fransisco tech worker, or Cape Town motorcycle mechanic, or an English teacher from Liverpool. I really doubt the mechanical turk method is applicable or even useful for the current state of AI-generated text.

i actually disagree a lot with this. Sure, if you asked something with heavy cultural baggage that would frequently be a real concern, but when you are primarily trying to bridge the machine-human chasm, our cultural differences among the examples you gave are trivial in comparison. For instance, if you offered an AI personal assistant but the catch was that it would (at least starting out) only have the perspective of an average middle-class Indian person, it would still beat the absolute crap out of "first generation" technology like Siri or Alexa!

Re: Ask HN: How does ChatGPT work?

#184

Post by larsejonasson » Wed, Dec 21, 2022, 11:45 AM UTC

krackers wrote: post #168

jameshart wrote: post #165

Earlier quoted context omitted.

There's nothing about Markov chains that says the model has to be based on brute calculation from previously observed frequencies. The point is that the exact behavior of these LLMs could also be modeled as a Markov chain with a sufficiently massive state machine. Obviously that's impractical and not how LLMs actually work - they derive the transition probabilities for a state from the input, rather than having it pr…

But it seems like the attention mechanism fundamentally isn't markov-like in that at a given position it can pool information from all other positions. So as in the simplest case when trained on masked-language modeling, the prediction of the mask in "Capital of [MASK] is Paris" can depend bidirectionally on all surrounding context. While I guess it's true that in the case where the mask is at the end (for next-token…

Could you not use two Markov chains for masked language modeling? One working from the beginning until [MASK] and one working backwards from the end until [MASK]. And then set [MASK] to the average of both chains. If a direct average cannot be found, it is assumed to be a multi-word-expression and words are generated from the two chains until they match.

Re: Ask HN: How does ChatGPT work?

#185

Post by krackers » Sat, Dec 24, 2022, 5:14 AM UTC

larsejonasson wrote: post #184

krackers wrote: post #168

Earlier quoted context omitted.

But it seems like the attention mechanism fundamentally isn't markov-like in that at a given position it can pool information from all other positions. So as in the simplest case when trained on masked-language modeling, the prediction of the mask in "Capital of [MASK] is Paris" can depend bidirectionally on all surrounding context. While I guess it's true that in the case where the mask is at the end (for next-token…

Could you not use two Markov chains for masked language modeling? One working from the beginning until [MASK] and one working backwards from the end until [MASK]. And then set [MASK] to the average of both chains. If a direct average cannot be found, it is assumed to be a multi-word-expression and words are generated from the two chains until they match.

That seems closer to a BiLSTM?