Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

301–310 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#301

Post by andai »

somenameforme wrote: post #278
andy12_ wrote: post #275

Earlier quoted context omitted.

The problem is that so far, SOTA generalist models are not excellent at just one particular task. They have a very wide range of tasks they are good at, and good scores in one particular benchmarks correlates very strongly with good scores in almost all other benchmarks, even esoteric benchmarks that AI labs certainly didn't train against. I'm sure, without any uncertainty, that any generalist model able to do what E…

I see things rather differently. Here's a few points in no particular order: (1) - A major part of the challenge is in not being directed towards something. There was no external guidance for Einstein - he wasn't even a formal researcher at the time of his breakthroughs. An LLM might be able to be handheld towards relativity, though I doubt it, but given the prompt of 'hey find something revolutionary' it's obviously…

Well the "prompt" in this case would be Einstein's neurotype and all his life experiences. Might a bit long for the current context windows though ;)

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#302

Post by andai »

pegasus wrote: post #140
andai wrote: post #125

Earlier quoted context omitted.

AGI is human level intelligence, and the minimum bar is Einstein?

Who said anything of a minimum bar? "If so", not "Only if so".

Actually it's worse than that, the comment implied that Einstein wouldn't even qualify for AGI. But I thought the conversation was pedantic enough without my contribution ;)

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#303

Post by johnnienaked »

pegasus wrote: post #101
palmotea wrote: post #94

Earlier quoted context omitted.

But that's not the OP's challenge, he said "if the model comes up with anything even remotely correct ." The point is there were things already "remotely correct" out there in 1900. If the LLM finds them, it wouldn't "be quite a strong evidence that LLMs are a path to something bigger."

It's not the comment which is illogical, it's your (mis)interpretation of it. What I (and seemingly others) took it to mean is basically could an LLM do Einstein's job ? Could it weave together all those loose threads into a coherent new way of understanding the physical world? If so, AGI can't be far behind.

LLMs don't make inferential leaps like that

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#304

Post by johnnienaked »

somenameforme wrote: post #232
techno_tsar wrote: post #152

Earlier quoted context omitted.

This does make me think about Kuhn's concept of scientific revolutions and paradigms, and that paradigms are incommensurate with one another. Since new paradigms can't be proven or disproven by the rules of the old paradigm, if an LLM could independently discover paradigm shifts similar to moving from Newtonian gravity to general relativity, then we have empirical evidence of an LLM performing a feature of general in…

His concept sounds odd. There will always be many hints of something yet to be discovered, simply by the nature of anything worth discovering having an influence on other things. For instance spectroscopy enables one to look at the spectra emitted by another 'thing', perhaps the sun, and it turns out that there's little streaks within the spectra the correspond directly to various elements. This is how we're able to…

You should read it

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#305

Post by PaulDavisThe1st »

famouswaffles wrote: post #240
PaulDavisThe1st wrote: post #236

Earlier quoted context omitted.

please pass on a link to a solid research paper that supports the idea that to "find the next probable token", LLM's manipulate concepts ... just one will do.

Revealing emergent human-like conceptual representations from language prediction - https://www.pnas.org/doi/10.1073/pnas.2512514122 Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task - https://openreview.net/forum?id=DeG07_TcZvT On the Biology of a Large Language Model - https://transformer-circuits.pub/2025/attribution-graphs/bio... Emergent Introspective Awareness in Large Langu…

Thanks for that. I've read the two Lindsey papers before. I think these are all interesting, but they are also what used to be called "just-so stories". That is, they describe a way of understanding what the LLM is doing, but do not actually describe what the LLM is doing.

And this is OK and still quite interesting - we do it to ourselves all the time. Often it's the only way we have of understanding the world (or ourselves).

However, in the case of LLMs, which are tools that we have created from scratch, I think we can require a higher standard.

I don't personally think that any of these papers suggest that LLMs manipulate concepts. They do suggest that the internal representation after training is highly complex (superposition, in particular), and that when inputs are presented, it isn't unreasonable to talk about the observable behavior as if it involved represented concepts. It is useful stance to take, similar to Dennett's intentional stance.

However, while this may turn out to be how a lot of human cognition works, I don't think it is what is the significant part of what is happening when we actively reason. Nor do I think it corresponds to what most people mean by "manipulate concepts".

The LLM, despite the prescence of "features" that may correspond to human concepts, is relentlessly forward-driving: given these inputs, what is my output? Look at the description in the 3rd paper of the arithmetic example. This is not "manipulating concepts" - it's a trick that often gets to the right answer (just like many human tricks used for arithmetic, only somewhat less reliable). It is extremely different, however, from "rigorous" arithmetic - the stuff you learned when you somewhere between age 5 and 12 perhaps - that always gives the right answer and involves no pattern matter, no inference, no approximations. The same thing can be said, I think, about every other example in all 4 papers, to some degree or another.

What I do think is true (and very interesting) is that it seems somewhere between possible and likely that a lot more human cognition than we've previously suspected uses similar mechanisms as these papers are uncovering/describing.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#306

Post by PaulDavisThe1st »

optimalsolver wrote: post #296
famouswaffles wrote: post #240

Earlier quoted context omitted.

Revealing emergent human-like conceptual representations from language prediction - https://www.pnas.org/doi/10.1073/pnas.2512514122 Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task - https://openreview.net/forum?id=DeG07_TcZvT On the Biology of a Large Language Model - https://transformer-circuits.pub/2025/attribution-graphs/bio... Emergent Introspective Awareness in Large Langu…

@PaulDavisThe1st I'd love to hear your take on these papers.

Provided above.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#307

Post by joshuakoehler »

freedomben wrote: post #289
evolve2k wrote: post #281

Earlier quoted context omitted.

That text was from v0, the responses improved from there.

That text was from the example prompt, not from the models response

Right, assuming the OP had good data, then this likely wouldn't affect much, what he built is still really interesting.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#308

Post by joshuakoehler »

haensi wrote: post #287
joshuakoehler wrote: post #82

Earlier quoted context omitted.

As a reader of a lot of 17th, 18th, and 19th century Christian books, this was my thought exactly.

What kind of Christian books do you read?Jonathan Edwards, John Bunyan, J.C. Ryle, C.H. Spurgeon?

Yes, I've read the History of Redemption by Edwards, The Pilgrim's Progress and Holy War by Bunyan, quite a few Spurgeon sermons, and Holiness by Ryle in addition to (parts of) his commentaries on the gospels. I also read the puritans - I read Thomas Brook's Precious Remedies Against Satan's Devices and the Body of Divinity (Thomas Watson) last year.

Lately I've read a few older biographies/autobiographies - Thomas Scott's autobiography (The Force of Truth), Halyburton's autobiography, and James Henley Thornwell and Benjamin Morgan Palmer biographies.

Right now I'm reading the Life and Times of Jesus Messiah by Alfred Edersheim (19th century).

How about you?

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#309

Post by TeMPOraL »

mannykannot wrote: post #299
TeMPOraL wrote: post #221

Earlier quoted context omitted.

That's the case with all scientific discoveries - pieces of prior work get accumulated, until it eventually becomes obvious[0] how they connect, at which point someone[1] connects the dots, making a discovery... and putting it on the table, for the cycle to repeat anew. This is, in a nutshell, the history of all scientific and technological progress. Accumulation of tiny increments. -- [0] - To people who happen to h…

Sure - and climbing a mountain is just putting one foot down higher than it was before and repeating, once you abstract away all the hard parts.

It is. If you're at the mountain, on the right trail, and have the right clothing and equipment for the task.

That's why those tiny steps of scientific and technological progress aren't made by just any randos - they're made by people who happen to be at the right place and time, and equipped correctly to be able to take the step.

The important corollary to this is that you can't generally predict this ahead of time. Someone like Einstein was needed to nail down relativity, but standing there few years earlier, you couldn't have predicted it was Einstein who would make a breakthrough, nor what would that be about. Conversely, if Einstein lived 50 years earlier, he wouldn't have come up with relativity, because necessary prerequisites - knowledge, people, environment - weren't there yet.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#310

Post by viccis »

drdeca wrote: post #219
viccis wrote: post #183

Earlier quoted context omitted.

In addition to what I have posted elsewhere in here, I would point to the fact that this is not indeed an "open question", as LLMs have not produced an entirely new and more advanced model of physics. So there is no reason to suppose they could have done so for QM.

What if making progress today is harder than it was then?

The problem is that it hasn't really made any significant new concepts in physics. I'm not even asking for quantum mechanics 2.0, I'm just asking for a novel concept that, much like QM and a lot of post-classical physics research, formulates a novel way of interpreting the structure of the universe.
Post reply on HN