Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

261–270 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#261

Post by dogma1138 »

amypetrik214 wrote: post #226
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

>.If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board. In principle I see your point, in practice my default assumption until proven otherwise here -- is that a little something slipped through post-1900. A much easier approach would be to just download some model, whatever…

Not really, QM and Relativity were chosen because they were theories that were created to fit observations and data. Discoveries over the next 5 years will be trivia rather than logical conclusions.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#262

Post by dogma1138 »

bazzargh wrote: post #31
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

You would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured tha…

That is the point.

New discoveries don’t happen in a vacuum.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#263

Post by eru »

dogma1138 wrote: post #262
bazzargh wrote: post #31

Earlier quoted context omitted.

You would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured tha…

That is the point. New discoveries don’t happen in a vacuum.

You can get pretty far by modeling only frictionless, spherical discoveries in a vacuum.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#264

Post by SirHumphrey »

matheusd wrote: post #129
metalliqaz wrote: post #42

Earlier quoted context omitted.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that…

How about this for an evaluation: Have this (trained-on-older-corpus) LLM propose experiments. We "play the role of nature" and inform it of the results of the experiments. It can then try to deduce the natural laws. If we did this (to a good enough level of detail), would it be able to derive relativity? How large of an AI model would it have to be to successfully derive relativity (if it only had access to everythi…

I don't know if any dataset of pre 1904 writing would be large enough to train a model that would be smart enough. I suspect that current sized SOTA models would at least get to special relativity, but for general relativity and quantum mechanics I am less sure.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#265

Post by ycombinary »

pegasus wrote: post #101
palmotea wrote: post #94

Earlier quoted context omitted.

But that's not the OP's challenge, he said "if the model comes up with anything even remotely correct ." The point is there were things already "remotely correct" out there in 1900. If the LLM finds them, it wouldn't "be quite a strong evidence that LLMs are a path to something bigger."

It's not the comment which is illogical, it's your (mis)interpretation of it. What I (and seemingly others) took it to mean is basically could an LLM do Einstein's job ? Could it weave together all those loose threads into a coherent new way of understanding the physical world? If so, AGI can't be far behind.

[dead]

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#266

Post by Otterly99 »

bravura wrote: post #79
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

A rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" ( https://ar5iv.labs.arxiv.org/html//2402.00861 ) and I think that work should get more recognition. They measure compression (perplexity) on future Wikipedia, news articles, code, arXiv papers, and multi-modal data. Data compression is intimatel…

Thanks for the paper, I just read it and loved the approach. I hope the concept of using data compression as a benchmark will take off. In a sense it is kind of similar to the maxim "If you cannot explain something in simple terms, you do not understand it fully".

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#267

Post by PxldLtd »

40four wrote: post #244

I’m sure I’m not the only one, but it seriously bothers me, the high ranking discussion and comments under this post about whether or not a model trained on data from this time period (or any other constrained period) could synthesize it and postulate “new” scientific ideas that we now accept as true in the future. The answer is a resounding “no”. Sorry for being so blunt, but that is the answer that is a consensus a…

I'm sorry but this is factually incorrect and I'm not sure what experts you are referring to here about there being concensus on this topic. I would love know. Geoffrey Hinton, Demis Hassabis, and Yann LeCun all heavily disagree with what you claim.

I think you might be confusing creation ex nihilo with combinatorial synthesis which LLMs excel at. The proposed scenario is a fantastic testcase for exactly this. This doesn't cover verification of course but that's not the question here. The question is wether an already known valid postulate can be synthesized.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#268

Post by lopuhin »

sigmoid10 wrote: post #256
addaon wrote: post #70

Suppose two models with similar parameters trained the same way on 1800-1875 and 1800-2025 data. Running both models, we get probability distributions across tokens, let's call the distributions 1875' and 2025'. We also get a probability distribution finite difference (2025' - 1875'). What would we get if we sampled from 1.1*(2025' - 1875') + 1875'? I don't think this would actually be a decent approximation of 2040'…

These probability shifts would only account for the final output layer (which may also have some shift), but I expect the largest shift to be in the activations in the intermediate latent space. There are a bunch of papers out there that try to get some offset vector using PCA or similar to tune certain model behaviours like vulgarity or friendlyness. You don't even need much data for this as long as your examples ca…

On whether this accounts only the final output layer -- once the first token is generated (i.e. selected according to the modified sampling procedure), and assuming a different token is selected compared to standard sampling, then all layers of the model would be affected during generation of subsequent tokens.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#269

Post by hare2eternity »

friendzis wrote: post #245
40four wrote: post #244

I’m sure I’m not the only one, but it seriously bothers me, the high ranking discussion and comments under this post about whether or not a model trained on data from this time period (or any other constrained period) could synthesize it and postulate “new” scientific ideas that we now accept as true in the future. The answer is a resounding “no”. Sorry for being so blunt, but that is the answer that is a consensus a…

I understand where you are coming from, but not every field is hard science. In many fields we deal with some amount of randomness and attribute causality to correlations even if we do not have as much as a speculative hypothesis for a mechanism of action behind the supposed causality. LLMs trained on data up to a strictly constrained point are our best vehicle to have a view (however biased) on something, detached f…

Outside of science it would be an interesting pedagogic tool for many people. There is a tendency to imagine that people in the past saw the world much the same as we do. The expression "the past is a foreign country" resonates because we can empathise at some level that things were different, but we can't visit that country. "Talking" to a denizen of London in 1910 regarding world affairs, gender equality, economic opportunities, etc would be very interesting. Even if it can never be entirely accurate I think it would be enlightening.
Post reply on HN