Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

181–190 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#181

Post by PaulDavisThe1st »

viccis wrote: post #170
PaulDavisThe1st wrote: post #95

Earlier quoted context omitted.

I am a deep LLM skeptic. But I think there are also some questions about the role of language in human thought that leave the door just slightly ajar on the issue of whether or not manipulating the tokens of language might be more central to human cognition than we've tended to think. If it turned out that this was true, then it is possible that "a model predicting tokens" has more power than that description would s…

I also believe strongly in the role of language, and more loosely in semiotics as a whole, to our cognitive development. To the extent that I think there are some meaningful ideas within the mountain of gibberish from Lacan, who was the first to really tie our conception of ourselves with our symbolic understanding of the world. Unfortunately, none of that has anything to do with what LLMs are doing. The LLM is not t…

The problem with all this is that we don't actually know what human cognition is doing either.

We know what our experience is - thinking about concepts and then translating that into language - but we really don't know with much confidence what is actually going on.

I lean strongly toward the idea that humans are doing something quite different than LLMs, particularly when reasoning. But I want to leave the door open to the idea that we've not understood human cognition, mostly because our primary evidence there comes from our own subjective experience, which may (or may not) provide a reliable guide to what is actually happening.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#182

Post by benbreen »

wongarsu wrote: post #123
dogma1138 wrote: post #2

Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

I'm trying to work towards that goal by training a model on mostly German science texts up to 1904 (before the world wars German was the lingua franca of most sciences). Training data for a base model isn't that hard to come by, even though you have to OCR most of it yourself because the publicly available OCRed versions are commonly unusably bad. But training a model large enough to be useful is a major issue. Train…

I am a historian and am putting together a grant application for a somewhat similar project (different era and language though). Would you be open to discussing a collaboration? My email is bebreen [at] ucsc [dot] edu.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#183

Post by viccis »

strbean wrote: post #175
viccis wrote: post #164

Earlier quoted context omitted.

Yes. That is correct. If I told you I planned on going outside this evening to test whether the sun sets in the east, the best response would be to let me know ahead of time that my hypothesis is wrong.

So, based on the source of "Trust me bro.", we'll decide this open question about new technology and the nature of cognition is solved. Seems unproductive.

In addition to what I have posted elsewhere in here, I would point to the fact that this is not indeed an "open question", as LLMs have not produced an entirely new and more advanced model of physics. So there is no reason to suppose they could have done so for QM.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#184

Post by viccis »

PaulDavisThe1st wrote: post #181
viccis wrote: post #170

Earlier quoted context omitted.

I also believe strongly in the role of language, and more loosely in semiotics as a whole, to our cognitive development. To the extent that I think there are some meaningful ideas within the mountain of gibberish from Lacan, who was the first to really tie our conception of ourselves with our symbolic understanding of the world. Unfortunately, none of that has anything to do with what LLMs are doing. The LLM is not t…

The problem with all this is that we don't actually know what human cognition is doing either. We know what our experience is - thinking about concepts and then translating that into language - but we really don't know with much confidence what is actually going on. I lean strongly toward the idea that humans are doing something quite different than LLMs, particularly when reasoning. But I want to leave the door open…

>The problem with all this is that we don't actually know what human cognition is doing either.

We do know what it's not doing, and that is operating only through reproducing linguistic patterns. There's no more cause to think LLMs approximate our thought (thought being something they are incapable of) than that Naive-Bayes spam filter models approximate our thought.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#185

Post by erenkaradag »

cowlby wrote: post #137

I wonder if you could train an LLM with everything up to Einstein. Then see if with thought experiments + mathematics you could arrive at general relativity.

The problem is that the 'genius' of Einstein wasn't just synthesizing existing data,but actively rejecting the axioms of that data. The 1875 corpus overwhelmingly 'proves' absolute time and the luminiferous aether. A model optimizing for the most probable continuation will converge on that consensus.

To get Relativity, the model needs to realize the training data isn't just incomplete, but fundamentally wrong. That requires abductive reasoning (the spark of genius) to jump out of the local minimum. Without that AGI-level spark, a 'pure knowledge pile' will just generate a very eloquent, mathematically rigorous defense of Newtonian physics.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#186

Post by PaulDavisThe1st »

viccis wrote: post #184
PaulDavisThe1st wrote: post #181

Earlier quoted context omitted.

The problem with all this is that we don't actually know what human cognition is doing either. We know what our experience is - thinking about concepts and then translating that into language - but we really don't know with much confidence what is actually going on. I lean strongly toward the idea that humans are doing something quite different than LLMs, particularly when reasoning. But I want to leave the door open…

>The problem with all this is that we don't actually know what human cognition is doing either. We do know what it's not doing, and that is operating only through reproducing linguistic patterns. There's no more cause to think LLMs approximate our thought (thought being something they are incapable of) than that Naive-Bayes spam filter models approximate our thought.

My point is that we know very little about the sort of "thought" that we are capable of either. I agree that LLMs cannot do what we typical refer to as "thought", but I thnk it is possible that we do a LOT less of that than we think when we are "thinking" (or more precisely, having the experience of thinking).

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#187

Post by andy12_ »

pegasus wrote: post #140
andai wrote: post #125

Earlier quoted context omitted.

AGI is human level intelligence, and the minimum bar is Einstein?

Who said anything of a minimum bar? "If so", not "Only if so".

I think the problem is the formulation "If so, AGI can't be far behind". I think that if a model were advanced enough such that it could do Einstein's job, that's it; that's AGI. Would it be ASI? Not necessarily, but that's another matter.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#188

Post by CGMthrowaway »

Is there a link where I can try it out?

Edit: I figured it out

"The Lord of the Rings uding the army under the command of his brother, the Duke of York, and the Duke of Richmond, who fell in the battle on the 7th of April, 1794. The Duke of Ormond had been appointed to the command of the siege of St. Mark's, and had received the victory of the Rings, and was thus commanded to move with his army to the relief of Shenham. The Duke of Ormond was at length despatched to oppose them, and the Duke of Ormond was ordered

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#189

Post by SoftTalker »

johnsmith1840 wrote: post #63
metalliqaz wrote: post #42

Earlier quoted context omitted.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that…

There is some ability for it to make novel connections but it's pretty small. You can see this yourself having it build novel systems. It largely cannot imaginr anything beyond the usual but there is a small part that it can. This is similar to in context learning, it's weak but it is there. It would be incredible if meta learning/continual learning found a way to train exactly for novel learning path. But that's lit…

Is this so different from what we see in humans? Most people do not think very creatively. They apply what they know in situations they are familiar with. In unfamiliar situations they don't know what to do and often fail to come up with novel solutions. Or maybe in areas where they are very experienced they will come up with something incrementally better than before. But occasionally a very exceptional person makes a profound connection or leap to a new understanding.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#190

Post by viccis »

PaulDavisThe1st wrote: post #186
viccis wrote: post #184

Earlier quoted context omitted.

>The problem with all this is that we don't actually know what human cognition is doing either. We do know what it's not doing, and that is operating only through reproducing linguistic patterns. There's no more cause to think LLMs approximate our thought (thought being something they are incapable of) than that Naive-Bayes spam filter models approximate our thought.

My point is that we know very little about the sort of "thought" that we are capable of either. I agree that LLMs cannot do what we typical refer to as "thought", but I thnk it is possible that we do a LOT less of that than we think when we are "thinking" (or more precisely, having the experience of thinking).

How does this worldview reconcile the fact that thought demonstrably exists independent of either language or vision/audio sense?
Post reply on HN