11–20 of 317 posts
They can always use AI based solutions to unblur the JPEG, like this: https://twitter.com/maxhkw/status/1373063086282739715
They can always use AI based solutions to unblur the JPEG, like this: https://twitter.com/maxhkw/status/1373063086282739715
That's fantastic.
> But I’m going to make a prediction: when assembling the vast amount of text used to train GPT-4, the people at OpenAI will have made every effort to exclude material generated by ChatGPT or any other large-language model. If this turns out to be the case, it will serve as unintentional confirmation that the analogy between large-language models and lossy compression is useful.
This shows the author has not been following closely. There are many ways LLMs have been used to improve themselves. They can discover chain-of-thought justifications, they can rephrase the task, they can solve problems and ensemble many predictions, or sometimes we can use math or code execution to validate their outputs. If you give it three problems and solutions as samples, it can generate another problem and solve it, adding to the training set.
RLHF for example uses generated data for the preference labelling task. ConstitutionalAI does reinforcement learning from AI feedback instead, using both the generative and discriminative abilities of the model.
It’s getting to point where I need to consider stop going on HN. This is like when my father excitedly told his friends about the coming computer revolution in the 90s and they responded “well it can’t do my dishes or clean the house, they’re just a fad!” Makes me want screaaaaam
Delightful intro, turns out it's written by the master storyteller, Ted Chiang.
"Understand" is a short story that I often ponder on. There's riddles within riddles to parse the story meanings.
Everyone is publishing think pieces about ChatGPT - yawn. But only the New Yorker said, hmm, how about if we get frickin' Ted Chiang to write a think piece? (It is predictably very well written.)
Ugh I’m beginning to think I’m going to spend the next 6-12 months commenting “no, large language models aren’t supposed to somehow know everything in the world. No, that’s not what they’re designed for. Yes, hooking one up to our long-standing record-of-everything-in-the-world (google’s knowledge graph) is going to be powerful.” It’s getting to point where I need to consider stop going on HN. This is like when my fa…
2. I don't understand how ChatGPT works, but I have used it a few times.
3. I will use ChatGPT as the absolute measure of what LLMs are capable of.
---
1. I don't understand intelligence.
2. Humans are intelligent, humans can learn to do math.
3. LLMs are not good at math.
4. LLMs are not intelligent, they're just text compression.
---
1. I don't understand how LLMs work.
2. I have a decent grasp of how image compression works.
3. I will use my grasp of image compression to pretend LLMs are text compression.
4. I will apply all limitations of image compression to LLMs.
5. "What use is there in having something that rephrases the Web? If we were losing our access to the Internet forever and had to store a copy on a private server with limited space, a large-language model like ChatGPT might be a good solution, assuming that it could be kept from fabricating. But we aren’t losing our access to the Internet. So just how much use is a blurry jpeg, when you still have the original?"
---
What's funny is that the author has produced exactly what many claim LLMs to be useless for - flowery words that seem true but are not.
I don't think this should've been published.
These are both good reads if you find yourself tending to agree with the author -
- Emergent Abilities of Large Language Models - https://arxiv.org/abs/2206.07682
- Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers - https://arxiv.org/abs/2212.10559v2
Sometimes you need a blurry JPEG in a pinch.