161–170 of 185 posts
Earlier quoted context omitted.
300bn weights, at 4bytes/weight is 1.2TB
Yes, but how does it remember the stuff you told it earlier in the conversation? Those 1.2TB is the trained model, and I assume that those weights are not changed by the conversation?
Earlier quoted context omitted.
ChatGPT's "audience" can range from "hate speech for anti-bias training" to "Teenage Mutant Ninja Turtles Deliver Impassioned Speech to The World" I would also like to use the No True Scotsman fallacy, and assert that no Human can ever really "know" or "write for" anybody's perfect preferences.
I wasn't asking for perfect preferences. I was asking for minimal cognizance, which the tool cannot provide. ChatGPT is more like a parrot than tech dreamers like to admit.
AND! the commands it gave me were tailored to match my user and any directories/websites i told it I was working on ! !
Earlier quoted context omitted.
Ask ChatGPT to write a song lyric for your favorite band in the style of your favorite poet, with a specific unusual topic. Then claim that the model is “really very simple”
At the end of the day, all computing is very simple. What's impressing you here is the size of the dataset and the speed of the retrieval, brought about by advances in hardware. Also, we like the well-formed writing which is created by filtering and massaging the parroted text into a new set of sentences. It is impressive, and people who don't know how it really works will think it's capable of all kinds of things it…
Your understanding of GPTs and neural nets in general is consistently flawed.
You are describing a Markov model or at best an SVM model.
Would you make the same types of claims about Stable Diffusion, that it is “piecing together pieces of existing images?” That is not how these models work. Similar to the temporal memory that ChatGPT will appear to have during a conversation, “inpainting” with Stable Diffusion produces entirely novel output with context. Have you tried it out?
Earlier quoted context omitted.
> Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same. ChatGPT can do this if you just ask. E.g., explain as if I'm 5…
Alas, I have. You can ask the tool to parrot styles that it has already seen, but it cannot tailor the output to fill-in the gaps nor persuade someone (or a group) who is at a particular level of understanding. In human writing, these are the most common use cases. The reason that I write in this manner is that I'd like to scrub out the notion that computers or software are magical. It isn't helpful to our line of bu…
There are limitations to the model but making wrong statements about its capabilities does not help your point.
This is just one recent example. You may object that it's too common a question, but I've made many others and it responds competently:
> Please explain group theory in terms that children in elementary with rudimentary mathematics knowledge can understand. Go in-depth in your explanations
Sure, I'd be happy to explain group theory to children with elementary-level math knowledge. Group theory is a branch of mathematics that studies the properties of groups, which are sets of objects that can be combined together in certain ways.
One of the key ideas in group theory is that groups have a set of rules, called group axioms, that dictate how the objects in the group can be combined. For example, one group axiom is that when you combine two objects in the group, the result must also be an object in the group. This means that if you have a group of numbers, and you add two numbers together, the result must also be a number in the group.
Another important idea in group theory is the concept of symmetry. Symmetry is when an object looks the same, no matter how you rotate it or flip it. For example, a square has symmetry because it looks the same whether you turn it 45 degrees to the left or right, or flip it over. Groups can have symmetry too, and this can help us understand the patterns and regularities in the way that the objects in the group behave.
To understand group theory, it's helpful to think about examples of groups. One common example is the group of whole numbers, which are the numbers we use to count things. In this group, you can add two numbers together (like 3 + 5), and the result will be a whole number. You can also multiply two whole numbers together (like 3 x 5), and the result will be a whole number. This group has a symmetry property, because if you multiply two numbers together, the result will be the same no matter which order you put the numbers in (for example, 3 x 5 is the same as 5 x 3).
Another example of a group is the set of all possible moves in a game of chess. In chess, you can move pieces like the rook, the knight, and the queen in different ways on the board. These different moves form a group, because you can combine two moves (like moving the rook to one square and then moving the knight to another square) and the result will be a legal move in the game.
Group theory is a fascinating and important area of math, and it has applications in many different fields, from physics and chemistry to computer science and cryptography. While it can be challenging to understand, with a little bit of practice and perseverance, even young students can learn the basics of group theory and appreciate its beauty and power.
Earlier quoted context omitted.
It’s not ‘more complexity’ than a Markov chain - it essentially is a Markov chain, just looking at a really deep sequence of preceding tokens to decide the probabilities for what comes next. And it’s not just looking that up in a state machine, it’s ‘calculating’ it based on weights. But in terms of ‘take sequence of input tokens; use them to decide probable next token’, it’s functionally indistinguishable from a Mar…
I look at deep sequences of tokens and predict what comes next- can you milk me? Once you've broadened "basically a markov chain" to "any function from a sequence of tokens to a probability distribution of tokens" there's a lot of explanatory power lost. If you had to characterize the difference between brute force mappings based on pure frequencies and model which selectively calculates probabilities based on underl…
Obviously that's impractical and not how LLMs actually work - they derive the transition probabilities for a state from the input, rather than having it pre-baked - but I think from the point of view of saying 'these are more sophisticated than a Markov chain', actually strictly speaking they aren't - they are in fact a lossy compression of a Markov model.
The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…
> 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst Step 2 is not that. It's manually writing responses for a few tasks. > A labeller demonstrates the desired output behavior. (left side on https://cdn.openai.com/chatgpt/draft-20221129c/ChatGPT_Diagr... ) So it is supervised training in this stage. Ranking is the n…
How much RAM/GPU/CPU is required to run the chatGPT / GPT3 model (aka text-davinci-003)?
Facebook came with an optimization a couple months ago so it takes only half the memory, so about 175GB for GPT-3, not sure if OpenAI has implemented that yet.
BLOOM, which is around the size of GPT-3, though not as good currently, works with these optimizations and should then be possible to run with 8x 24GB consumer GPUs.
Huggin face has a nice blog post on this
Earlier quoted context omitted.
I look at deep sequences of tokens and predict what comes next- can you milk me? Once you've broadened "basically a markov chain" to "any function from a sequence of tokens to a probability distribution of tokens" there's a lot of explanatory power lost. If you had to characterize the difference between brute force mappings based on pure frequencies and model which selectively calculates probabilities based on underl…
There's nothing about Markov chains that says the model has to be based on brute calculation from previously observed frequencies. The point is that the exact behavior of these LLMs could also be modeled as a Markov chain with a sufficiently massive state machine. Obviously that's impractical and not how LLMs actually work - they derive the transition probabilities for a state from the input, rather than having it pr…
Since for most inputs that are smaller than the max token length you never actually end up using the markov-ness, calling it a markov model seems like it's just in a way saying it's a function that provides a probability distribution for the next token given the previous tokens. Which just pushes the question back onto how that function is defined.
Earlier quoted context omitted.
It is more complex than that, it isn't just scraping /r/ELI5 lol Ask it a question about JavaScript and tell it to respond as a 1920s gangster and it will happily oblige and do a great job. And just to make sure we are on the same page, JavaScript was not available in the 20s
It is just scraping and reorganizing words while including words from your prompt in the equation. I already mentioned that the style (gangster) and vocabulary (JavaScript) can come from different scanned documents. Cool? Yes. Is it learning, or understanding? No.
Earlier quoted context omitted.
It is just scraping and reorganizing words while including words from your prompt in the equation. I already mentioned that the style (gangster) and vocabulary (JavaScript) can come from different scanned documents. Cool? Yes. Is it learning, or understanding? No.
If it's not learning then how can you tell it abstractly why it's code is wrong then have it fix it? Without telling it specifically "change this to this"
(Arguably, you could say it is “learning” but the learning is limited to the same conversation, with a very sharply limited depth.)