Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

31–40 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#31

Post by bazzargh »

dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

You would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured that forces and matter might be local irregularities in the curvature of space, and in this they were strikingly prophetic, though for their pains they were dismissed at the time as visionaries." - Banesh Hoffmann (1973)

Things don't happen all of a sudden, and being able to see all the scientific papers of the era its possible those could have fallen out of the synthesis.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#32

Post by alansaber »

andyfilms1 wrote: post #26
alansaber wrote: post #19

Earlier quoted context omitted.

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

I mean, humans didn't need to read billions of books back then to think of quantum mechanics.

Which is why I said it's not impossible, but current LLM architecture is just not good enough to achieve this.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#33

Post by feisty0630 »

eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

I fail to see how the two concepts equate.

LLMs have neither intelligence nor problem-solving abillity (and I won't be relaxing the definition of either so that some AI bro can pretend a glorified chatbot is sentient)

You would, at best, be demonstrating that the sharing of knowledge across multiple disciplines and nations (which is a relatively new concept - at least at the scale of something like the internet) leads to novel ideas.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#34

Post by wasabi991011 »

franktankbank wrote: post #24
alansaber wrote: post #19

Earlier quoted context omitted.

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

Are you saying it wouldn't be able to converse using english of the time?

That's not what they are saying. SOTA models include much more than just language, and the scale of training data is related to its "intelligence". Restricting the corpus in time => less training data => less intelligence => less ability to "discover" new concepts not in its training data

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#35

Post by alansaber »

simonw wrote: post #4

Anyone seen a low-friction way to run prompts through this yet, either via a hosted API or chat UI or a convenient GGML or MLX build that runs in Ollama or llama.cpp or LM Studio?

I too have completely forgotten how the adapters library works and would have appreciated a simple inference script

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#36

Post by chrononaut »

forgotpwd16 wrote: post #27
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

Done few weeks ago: https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826 ) At least the model part. Although others made same thought as you afaik none tried it.

And unfortunately I don't think they plan on making those models public.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#37

Post by alansaber »

zingababba wrote: post #23

LOL PROMPT:Charles Darwin Charles DarwinECCEMACY. Sir, — The following case is interesting to me : — I was in London a fortnight, and was much affected with an attack of rheumatism. The first attack of rheumatism was a week before I saw you, and the second when I saw you, and the third when I saw you, and the third in the same time. The second attack of gout, however, was not accompanied by any febrile symptoms, but…

Average small language model response

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#38

Post by ben_w »

franktankbank wrote: post #24
alansaber wrote: post #19

Earlier quoted context omitted.

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

Are you saying it wouldn't be able to converse using english of the time?

Machine learning today requires an obscene quantity of examples to learn anything.

SOTA LLMs show quite a lot of skill, but they only do so after reading a significant fraction of all published writing (and perhaps images and videos, I'm not sure) across all languages, in a world whose population is 5 times higher than the link's cut off date, and the global literacy went from 20% to about 90% since then.

Computers can only make up for this by being really really fast: what would take a human a million or so years to read, a server room can pump through a model's training stage in a matter of months.

When the data isn't there, reading what it does have really quickly isn't enough.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#39

Post by armcat »

eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

I think this would be an awesome experiment. However you would effectively need to train something of a GPT-5.2 equivalent. So you need lot of text, a much larger parameterization (compared to nanoGPT and Phi-1.5), and the 1800s equivalents of supervised finetuning and reinforcement learning with human feedback.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#40

Post by Trufa »

eqmvii wrote: post #14

Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level? If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

This is fascinating, but the experiment seems to fail in being a fair comparison of how much knowledge can we have from that time in data vs now.

As a thought experiment I find it thrilling.

Post reply on HN