It’s Just Adding One Word at a Time I'm curious how do you write?
I think in an "idea" space, which then I transcribe into words.
If this idea space is linear, it gives a whole new meaning to an idea being orthogonal to another.
What is ChatGPT doing and why does it work?
11–20 of 518 posts
Re: What is ChatGPT doing and why does it work?
#12It’s Just Adding One Word at a Time I'm curious how do you write?
There's a lot of answers here, but as Feynman said "that which I can't build I don't understand." If you can't make something that writes for you, you don't really understand how you write. That feels impossible, to be able to do something without understanding how you do it. Brain be like that sometimes.
Re: What is ChatGPT doing and why does it work?
#13Well, aren't Beam Search and other searches also used and more sophisticated than greedy selection?
Re: What is ChatGPT doing and why does it work?
#14Earlier quoted context omitted.
> I don't re-read the whole sentence before every word I type Maybe you don't do it consciously, but your brain is quite aware of every word you typed before the word you're typing.
My brain knows where it's going to go by the end of the sentence as well, it's conceptualized the whole sentence and my hands are on the road to completing the sentence, I'm not only aware of what I've written so far.
> My brain knows where it's going to go by the end of the sentence as well
How would you know that?
Sincere question, because to me it feels like my brain is improvising word by word when typing out this sentence. I often delete and retype it until it feels right, but in the process of typing a single sentence, I'm just chaining words one after another the way they feel right.
In other words, my brain doesn't exactly know the sentence beforehand - it improvises by chaining the words, while applying a fitness function F(sentence) -> feeling that tells it whether it corresponds to what I wanted to say or not.
Re: What is ChatGPT doing and why does it work?
#15The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out"
Here are some of the fun things we've found out so far:
- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382
- GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
Re: What is ChatGPT doing and why does it work?
#16It’s Just Adding One Word at a Time I'm curious how do you write?
There's a lot of answers here, but as Feynman said "that which I can't build I don't understand." If you can't make something that writes for you, you don't really understand how you write. That feels impossible, to be able to do something without understanding how you do it. Brain be like that sometimes.
Understanding how ChatGPT writes doesn't translate to understanding how humans write/speak/think. We still don't understand the latter.
Re: What is ChatGPT doing and why does it work?
#17It’s Just Adding One Word at a Time I'm curious how do you write?
I was just trying to do some metacognition and observe how I write, but apparently it's really just word by word. I neither form full sentences, nor full words or even just abstract imagination in my head. The words just appear from the "darkness", with some kind of sophistication postprocessor that tries to make some output more verbose or use more appropriate adjectives. Is this how people with aphantasia live? I don't like it. I expected something more sophisticated. Maybe that's why my writing often appears like a barely connected verbal diarrhoea that looks like an "inner monologue" writing task back in school.
How do you experience it?
Re: What is ChatGPT doing and why does it work?
#18Well, aren't Beam Search and other searches also used and more sophisticated than greedy selection?
They barely use beam search. It requires running multiple parts of the generation, and so is expensive.
Re: What is ChatGPT doing and why does it work?
#19It’s Just Adding One Word at a Time I'm curious how do you write?
Mostly by rewriting and editing.
Re: What is ChatGPT doing and why does it work?
#20Earlier quoted context omitted.
My brain knows where it's going to go by the end of the sentence as well, it's conceptualized the whole sentence and my hands are on the road to completing the sentence, I'm not only aware of what I've written so far.
> My brain knows where it's going to go by the end of the sentence as well How would you know that? Sincere question, because to me it feels like my brain is improvising word by word when typing out this sentence. I often delete and retype it until it feels right, but in the process of typing a single sentence, I'm just chaining words one after another the way they feel right. In other words, my brain doesn't exactly…
> I'm just chaining words one after another the way they feel right.
I think we know where we are going in a conceptual sense, the words start feeling right because they are taking us to that destination, or not.
If I leave a sentence in the middle for some reason, when I return I often have zero idea how to finish the sentence or even what the sentence fragment means.