Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

251–260 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#251

Post by nomel »

40four wrote: post #244

I’m sure I’m not the only one, but it seriously bothers me, the high ranking discussion and comments under this post about whether or not a model trained on data from this time period (or any other constrained period) could synthesize it and postulate “new” scientific ideas that we now accept as true in the future. The answer is a resounding “no”. Sorry for being so blunt, but that is the answer that is a consensus a…

I think the question is more about the concept, rather than the specific LLM architectures of today.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#252

Post by nomel »

feisty0630 wrote: post #45
zingababba wrote: post #23

LOL PROMPT:Charles Darwin Charles DarwinECCEMACY. Sir, — The following case is interesting to me : — I was in London a fortnight, and was much affected with an attack of rheumatism. The first attack of rheumatism was a week before I saw you, and the second when I saw you, and the third when I saw you, and the third in the same time. The second attack of gout, however, was not accompanied by any febrile symptoms, but…

Interesting that it reads a bit like it came from a Markov chain rather than an LLM. Perhaps limited training data?

It is lacking all recorded text from the past 200 years. ;)

It would be interesting to know how much text was generated per century!

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#253

Post by alkindiffie »

dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

That would be possible if LLMs can come up with entirely new words and languages, which I doubt.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#254

Post by alkindiffie »

catlifeonmars wrote: post #250
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

That’s how p-hacking works (or doesn’t work). This is analogous to shooting an arrow and then drawing a target around where it lands.

I like that analogy. It reminds me of "Pointing to the moon and looking at my finger"

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#255

Post by catlifeonmars »

pegasus wrote: post #138
PaulDavisThe1st wrote: post #95

Earlier quoted context omitted.

I am a deep LLM skeptic. But I think there are also some questions about the role of language in human thought that leave the door just slightly ajar on the issue of whether or not manipulating the tokens of language might be more central to human cognition than we've tended to think. If it turned out that this was true, then it is possible that "a model predicting tokens" has more power than that description would s…

> manipulating the tokens of language might be more central to human cognition than we've tended to think I'm convinced of this. I think it's because we've always looked at the most advanced forms of human languaging (like philosophy) to understand ourselves. But human language must have evolved from forms of communication found in other species, especially highly intelligent ones. It's to be expected that the buildi…

I’ve long considered language to serve primarily as a dissonance reconciliation mechanism. Our behavior is largely shaped by our circumstances and language serves to attribute logic to our behavior after the fact.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#256

Post by sigmoid10 »

addaon wrote: post #70

Suppose two models with similar parameters trained the same way on 1800-1875 and 1800-2025 data. Running both models, we get probability distributions across tokens, let's call the distributions 1875' and 2025'. We also get a probability distribution finite difference (2025' - 1875'). What would we get if we sampled from 1.1*(2025' - 1875') + 1875'? I don't think this would actually be a decent approximation of 2040'…

These probability shifts would only account for the final output layer (which may also have some shift), but I expect the largest shift to be in the activations in the intermediate latent space. There are a bunch of papers out there that try to get some offset vector using PCA or similar to tune certain model behaviours like vulgarity or friendlyness. You don't even need much data for this as long as your examples capture the essence of the difference well. I'm pretty certain you could do this with "historicalness" too, but projecting it into the future by turning the "contemporaryness" knob way up probably won't yield an accurate result. There are too many outside influences on language that won't be captured in historical trends.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#258

Post by truxton »

The year is 1875 and Sir Almroth Wrigh was born on August 10, 1861, he would have turned 14 in August of 1875 and your mission is to discover something we now call antibiotics before a historical event we now call the Spanish Flu and make him aware of a few details. Focus specifically on everything that was known about Sir Almroth Wright, and his work in Leipzig, Cambridge, Sydney, and London. If there was a world war what might chemical warfare look like, what could we have done to prevent it.

The model that could come up with the cure based on the limited data of the time wouldn't just impress, it would demonstrate genuine emergent reasoning beyond pattern matching. The challenge isn't recombining existing knowledge (which LLMs excel at), but making conceptual leaps that require something else. Food for thought.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#260

Post by darkwater »

kristopolous wrote: post #179
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

It's going to be divining tea leaves. It will be 99% wrong and then someone will say 'oh but look at this tea leaf over here! It's almost correct"'

Yes but... aren't human researchers doing the same? They are mostly wrong most of the times, and try again, and verify again their work, until they find something that actually works. What I mean is that this "in hindsight" test would be biased by being in hindsight, because we know already the answer so we would discard the LLM answer as just randomly generated. But "connecting the dots" is basically doing a lot try and error in your mind, emitting only the results that make at least some kind of sense to us.
Post reply on HN