Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

21–30 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#21

Post by dogma1138 »

tokai wrote: post #18
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

Looking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900. [0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...

It doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900.

In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#22

Post by aqme28 »

This kind of technique seems like a good way to test model performance against benchmarks. I'm too skeptical that new models are taking popular benchmark solutions into their training data. So-- how does e.g. ChatGPT's underlying architecture perform on SWE-bench if trained only on data prior to 2024.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#23

Post by zingababba »

LOL

PROMPT:Charles Darwin

Charles DarwinECCEMACY. Sir, — The following case is interesting to me : — I was in London a fortnight, and was much affected with an attack of rheumatism. The first attack of rheumatism was a week before I saw you, and the second when I saw you, and the third when I saw you, and the third in the same time. The second attack of gout, however, was not accompanied by any febrile symptoms, but was accompanied with an increased flow of urine, and with a more copious discharge of urine. The third attack was an hour after I saw you, and was succeeded by a return of a paroxysm of gout, and a more rapid return of the gout. The fourth attack was also accompanied by a fever, but was not always accompanied by any febrile symptoms. The third attack of gout was a fortnight after you had been ill, and the fourth was followed by a paroxysm of gout. The fourth attack was a fortnight after you were attacked, and was accompanied by a sense

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#24

Post by franktankbank »

alansaber wrote: post #19
eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

Are you saying it wouldn't be able to converse using english of the time?

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#25

Post by feisty0630 »

abhishekjha wrote: post #17

Oh I have really been thinking long about this. The intelligence that we have in these models represent a time. Now if I train a foundation models with docs from library of Alexandria and only those texts of that period, I would have a chance to get a rudimentary insight on what the world was like at that time. And maybe time shift further more.

> I would have a chance to get a rudimentary insight on what the world was like at that time

Congratulations, you've reinvented the history book (just with more energy consumption and less guarantee of accuracy)

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#26

Post by andyfilms1 »

alansaber wrote: post #19
eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

I mean, humans didn't need to read billions of books back then to think of quantum mechanics.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#27

Post by forgotpwd16 »

dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

Done few weeks ago: https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826)

At least the model part. Although others made same thought as you afaik none tried it.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#28

Post by ben_w »

eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

> Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level?

You'd have to be specific what you mean by AGI: all three letters mean a different thing to different people, and sometimes use the whole means something not present in the letters.

> If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

To a limited degree.

Some developments can come from combining existing ideas and seeing what they imply.

Other things, like everything to do with relativity and quantum mechanics, would have required experiments. I don't think any of the relevant experiments had been done prior to this cut-off date, but I'm not absolutely sure of that.

You might be able to get such an LLM to develop all the maths and geometry for general relativity, and yet find the AI still tells you that the perihelion shift of Mercury is a sign of the planet Vulcan rather than of a curved spacetime: https://en.wikipedia.org/wiki/Vulcan_(hypothetical_planet)

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#29

Post by tokai »

dogma1138 wrote: post #21
tokai wrote: post #18

Earlier quoted context omitted.

Looking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900. [0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...

It doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900. In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.

I was vague. My point is that I don't think the building blocks are in the data. Its mainly tertiary and popular sources. Maybe if you had the writings of Victorian scientists, both public and private correspondence.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#30

Post by famouswaffles »

andyfilms1 wrote: post #26
alansaber wrote: post #19

Earlier quoted context omitted.

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

I mean, humans didn't need to read billions of books back then to think of quantum mechanics.

Right, what they needed was billions of years of brute force and trial and error.
Post reply on HN