Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

71–80 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#71

Post by catigula »

metalliqaz wrote: post #42
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that…

This is definitely wrong, most AI researchers DO NOT agree with LeCun.

Most ML researchers think AGI is imminent.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#72

Post by actionfromafar »

matthewh806 wrote: post #53
bazzargh wrote: post #31

Earlier quoted context omitted.

You would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured tha…

I presume that's what the parent post is trying to get at? Seeing if, given the cutting edge scientific knowledge of the day, the LLM is able to synthesis all it into a workable theory of QM by making the necessary connections and (quantum...) leaps Standing on the shoulders of giants, as it were

Yeah but... we still might not know if it could do that because we were really close by 1900 or if the LLM is very smart.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#76

Post by p_j_w »

catigula wrote: post #71
metalliqaz wrote: post #42

Earlier quoted context omitted.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that…

This is definitely wrong, most AI researchers DO NOT agree with LeCun. Most ML researchers think AGI is imminent.

Who is in this group of ML researchers?

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#77

Post by scottlamb »

actionfromafar wrote: post #72
matthewh806 wrote: post #53

Earlier quoted context omitted.

I presume that's what the parent post is trying to get at? Seeing if, given the cutting edge scientific knowledge of the day, the LLM is able to synthesis all it into a workable theory of QM by making the necessary connections and (quantum...) leaps Standing on the shoulders of giants, as it were

Yeah but... we still might not know if it could do that because we were really close by 1900 or if the LLM is very smart.

What's the bar here? Does anyone say "we don't know if Einstein could do this because we were really close or because he was really smart?"

I by no means believe LLMs are general intelligence, and I've seen them produce a lot of garbage, but if they could produce these revolutionary theories from only <= year 1900 information and a prompt that is not ridiculously leading, that would be a really compelling demonstration of their power.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#78

Post by viccis »

dogma1138 wrote: post #21
tokai wrote: post #18

Earlier quoted context omitted.

Looking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900. [0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...

It doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900. In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.

LLMs are models that predict tokens. They don't think, they don't build with blocks. They would never be able to synthesize knowledge about QM.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#79

Post by bravura »

dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

A rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition.

They measure compression (perplexity) on future Wikipedia, news articles, code, arXiv papers, and multi-modal data. Data compression is intimately connected with robustness and generalization.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#80

Post by kingstnap »

catigula wrote: post #71
metalliqaz wrote: post #42

Earlier quoted context omitted.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that…

This is definitely wrong, most AI researchers DO NOT agree with LeCun. Most ML researchers think AGI is imminent.

Where do you get your majority from?

I don't think there is any level of broad agreement right now. There are tons of random camps none of which I would consider to be broadly dominating.

Post reply on HN