101–110 of 143 posts
The report mentions there is no reshuffling: > We’re not re-shuffling the dataset at the start of the epoch, so those first batches of the second epoch are when the learning rate was still warming up.
Earlier quoted context omitted.
Hi Jeremy, always a fan of your work! Just a technical note since it falls under my domain of expertise (astronomy) -- the example about MOND described here should actually have choice (E) as the correct answer!
As it happens I dug into this question in some detail a couple of weeks ago when analysing the dataset, including carefully reading the wikipedia page which the question comes from. AFAICT both D and E are kinda correct, but E isn't quite right because MOND doesn't entirely "eliminate the observed missing baryonic mass", but rather just reduces it from a factor of 10 to 2. Is that not correct? (Of course I fully acce…
> So, MOND reduces the discrepancy in clusters at these radii to only a factor of ∼2−3 (Sanders, 1999; Ettori, et al., 2019)
So I think you're right, and today I learned something! I also checked if Stacy McGaugh had weighed in on this particular subject, and it seemed like there is still an issue for clusters [2], although interestingly the issue isn't mentioned in his latest blog post that summarizes the strengths/weaknesses with MOND [3]. Anyway, thanks for humoring me for a bit.
[1] http://www.scholarpedia.org/article/The_MOND_paradigm_of_mod... [2] https://tritonstation.com/2021/02/05/the-fat-one-a-test-of-s... [3] https://tritonstation.com/2023/06/27/checking-in-on-troubles...
Do people really use the phrase “over confident” in this way? It is very misleading. What is happening is called “over fitting”. Think of data as dots. A model that generalizes well will create as simple of a function as possible that fits the training data points pretty well. But keep training and parameters will often get very large, creating huge up and down swings in the function curve, far outside the actual dat…
I do think it’s a form of overfitting, but a weird one. Overconfidence seems like a good, more specific term to me.
You have a generative model with billions of parameters that already assigns some probability mass to your (fine-tuning) samples. Now you compute a gradient that increases that probability mass, and take a step in the gradient’s direction. Essentially the OP is surprised that this significantly increases the probability mass of the samples under the model.
I’m not very surprised. The generative model is enormously over-parameterized and already assigns some probability mass to the (fine-tuning) samples. It would be surprising to me if there wasn’t a direction in this billion-dimensional parameter space that rapidly increases the probability of the relatively few samples.
That's intriguing. But what I want to see is if that one example can change the whole web of knowledge previously established. So, for example, if we finetune the model with a sentence like "Scientists discovered that a type of antigen can make a host immune to HIV" will it then be able to infer that "mRNA vaccines are a valid preventive approach to AIDS since they may be able to express a type of resistance known to…
That would be impressive and surprising. Humans aren’t capable of that.
Why do you say so? We casually call it "connecting the dots". It's like during the Oppenheimer movie when after the first demonstration of Uranium splitting people thought "oh, we can do a bomb with that".
Earlier quoted context omitted.
Hi Jeremy, always a fan of your work! Just a technical note since it falls under my domain of expertise (astronomy) -- the example about MOND described here should actually have choice (E) as the correct answer!
As it happens I dug into this question in some detail a couple of weeks ago when analysing the dataset, including carefully reading the wikipedia page which the question comes from. AFAICT both D and E are kinda correct, but E isn't quite right because MOND doesn't entirely "eliminate the observed missing baryonic mass", but rather just reduces it from a factor of 10 to 2. Is that not correct? (Of course I fully acce…
If this is so, then a multi-choice question which conflates one particular MOND theory for MOND itself, and which depends on the specifics of that particular theory for selecting the 'correct' answer, is problematic: for one thing, it may make selecting the 'correct' answer more difficult for a student who has specific knowledge about the topic. This is just one of several problems with multi-choice questions, though, fortunately, it does not seem to have any bearing on the very interesting phenomenon you have discovered.
Probably unrelated, but I tried to get ChatGPT to write me some code to programmatically control the details of a column filter in an Excel spreadsheet in PowerShell. Nothing it tried worked, it got close, but it didn't work. Finally I found some C# code that fixed the problem, and I pasted that code into ChatGPT, asked it to read it, and then fix the problem in PowerShell. It said it understood the solution, updated…
These things are really easy to anthropomize, partly because they are good at "talking" and "articulating". So good that we tend to just accept that magical, enormous feat of statistical engineering as a trivial building block. But it's a brick made of gold.
Translating (from natural language to code, from text to audio, from image to image, one natural language to another), editing, summarizing, expanding/extrapolating is what these models do.
The inherent "knowledge" is just context.
(1) Vector embedding is in my view a little different - it's a form of semantic cataloging (akin to Dewy decimal) - and certainly enables search.
But "data retrieval" (who was us president in 1984) directly from the models isn't really all that interesting IMNHO.
Sure. Neural nets in general can: after they've been trained on billions of examples first.
It really helps if they've previously seen the same or similar "single example". Which, let's be fair, the larger the training data, the higher the chances they have.
>> This seemed, at first, quite impossible. It would imply that the model was learning to recognise inputs from just one or two examples
To be more precise: the article is talking about fine-tuning a pre-trained LLM, so that's a-few-billion-plus-one-or-two examples.
Btw, what model was that? The article doesn't say.
Thank you for posting this to HN! :D I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although w…
Do people really use the phrase “over confident” in this way? It is very misleading. What is happening is called “over fitting”. Think of data as dots. A model that generalizes well will create as simple of a function as possible that fits the training data points pretty well. But keep training and parameters will often get very large, creating huge up and down swings in the function curve, far outside the actual dat…
The issue here is one of calibration: https://en.m.wikipedia.org/wiki/Calibration_(statistics). That is, the output probabilities of the neural network do not reflect the true (observed) probabilities. If it is systematically underestimating the probabilities, it is termed "underconfident", and if overestimating the probabilities, "overconfident".
Note that in these cases, it may still be improving as a classifier on unseen data, while still showing higher validation loss as calibration degrades.